Information processing device, information processing method, and program

Abstract

To provide an information processing apparatus which can reduce a collation frequency to reduce computational complexity by determining combinations of collation objects corresponding to a person in a video before collation.SOLUTION: The information processing apparatus includes detection means which detects persons in each of a plurality of frames, tracking means which generates tracking information including feature quantities of the persons detected from the plurality of frames in accordance with the number of persons detected by the detection means, storage means in which the tracking information and person information being information associated with at least one piece of tracking information about a same person are stored, determination means which determines combinations of tracking information and person information on the basis of the feature quantities in order to perform collation of person information corresponding to the tracking information, and collation means which collates the tracking information and the person information determined by the determination means.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, there is a technique for tracking the movement trajectory of a person in a video captured by a surveillance camera. When collating the person in the video with the information of the person existing in the database (hereinafter referred to as person information) while chasing the person in the video in real time, it is necessary to complete the collation process in a short time in order to perform the tracking process without delay for each frame. However, when the number of people in the video and the number of person information increase, the amount of calculation for the collation process increases.

[0003] Conventionally, in order to suppress the amount of calculation when the number of people in the video is large, a method of limiting the person to be collated in the video has been disclosed. Patent Document 1 describes a method of limiting the person to be collated based on the past collation times and time-series information regarding the person being tracked in the video. According to this method, for example, it is possible to preferentially confirm the correctness of the person information of the subject for which a long time has passed since the last collation. Patent Document 2 describes a method of preferentially selecting a person with a large amount of position change from the previous frame, that is, a person who is highly likely to go out of the frame immediately, as a collation target. According to this method, even when there are many people in the video, it is possible to reduce the amount of calculation while preventing omission of the collation process for the people appearing in the video.

Prior Art Documents

Patent Documents

[0004] <00……

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Even when performing person matching on a person in a video across multiple frames, it may not be possible to determine the person information corresponding to the person in the video. This can occur, for example, when there are multiple pieces of person information in the database for people with similar clothing. Patent documents 1 and 2 do not take this into consideration, and when the aforementioned case occurs, the matching process has to be repeated for the same person in the video, which increases the computational load.

[0006] Therefore, the present invention aims to provide an information processing device that can reduce the number of matching operations and thus the computational load by determining the combination of matching targets corresponding to people in the video before matching. [Means for solving the problem]

[0007] To achieve the above objective, the information processing apparatus as one aspect of the present invention includes: detection means for detecting a person in a frame for each of multiple frames; tracking means for generating tracking information, which is information including the feature quantities of the people detected from the multiple frames, in accordance with the number of people detected by the detection means; storage means for storing person information, which is information that associates the tracking information with at least one piece of tracking information relating to the same person; and for matching the person information corresponding to the tracking information, In tracking information where the person has not been identified Features Similarity between them The system is characterized by having a determination means for determining a combination of tracking information and person information as a matching target based on the above, and a matching means for matching the tracking information and person information determined by the determination means. [Effects of the Invention]

[0008] According to the present invention, by determining the combination of matching targets corresponding to people in the video before matching, it is possible to reduce the number of matching operations and thus reduce the computational load. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the information processing device of Embodiment 1. [Figure 2]This is a block diagram of the functional configuration of Embodiment 1. [Figure 3] This is a processing flowchart of Embodiment 1. [Figure 4] This is a diagram illustrating the video used in Embodiment 1. [Figure 5] This diagram illustrates the information stored in the storage unit according to Embodiment 1. [Figure 6] This diagram illustrates the processing of the combination determination unit 270 in Embodiment 1. [Figure 7] This diagram illustrates the processing of the person matching unit in Embodiment 1. [Figure 8] This is a block diagram of the functional configuration of Embodiment 2. [Figure 9] This is a processing flowchart for Embodiment 2. [Figure 10] This diagram illustrates the information generated by the filtering unit of Embodiment 2. [Figure 11] This figure illustrates the video used in a modified example of Embodiment 2. [Modes for carrying out the invention]

[0010] The embodiments for carrying out the present invention will be described in detail below with reference to the attached drawings. The embodiments described below are examples of means for realizing the present invention and should be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions, and the present invention is not limited to the embodiments described below. Furthermore, some of the embodiments described later may be combined as appropriate.

[0011] <Embodiment 1> The information processing device 100 according to Embodiment 1 also functions as a person tracking device that analyzes video footage of a person captured by a network camera or the like and acquires the movement trajectory of the person. Embodiment 1 describes an example in which the movement trajectory of the same person is acquired from a single camera image.

[0012] First, the process of tracking the movement trajectory of a person handled in Embodiment 1 will be described below. In Embodiment 1, the process of tracking the short-term movement trajectory of a person is described as "tracking". Hereinafter, the information on the movement trajectory obtained by the tracking process will be described as "tracking information". The tracking information is information obtained by detecting a person for each frame in the video and arranging the information on the detection area of the person in each frame in time series. Hereinafter, the information on the detection area arranged in the tracking information will be described as "feature quantity". The feature quantity is the coordinates of the detection area and image features.

[0013] Also, hereinafter, the process of associating the tracking information of the same person will be called "person matching". In Embodiment 1, the association is performed by associating a plurality of pieces of tracking information considered to be the movement trajectory of the same person with the ID of the person existing in the database. Also, hereinafter, the ID for associating a plurality of pieces of tracking information will be described as "person information". When a person in the video disappears from the screen for a long time during the tracking process and then reappears, the tracking information is interrupted. However, when person matching is performed, the movement trajectories of the tracking information interrupted before and after the disappearance of the figure can be associated as the movement trajectory of the same person.

[0014] The tracking information includes tracking information with undetermined person information and tracking information with determined person information (first tracking information). Note that, among the tracking information, in order to distinguish the tracking information with undetermined person information from the tracking information with determined person information, hereinafter, it will be described as "query of tracking information (second tracking information)". Also, the person information includes person information corresponding to the tracking information in the frame being processed (first person information) and person information for which there is no corresponding tracking information in the frame being processed. Note that, among the person information, hereinafter, the person information for which there is no corresponding tracking information in the frame being processed will be described as "person information for which matching has not been performed (second person information)".

[0015] Then, in Embodiment 1, a method of determining, from the query of the following information and the person information, a combination of the following information and the person information that has a high possibility of determining the person information for the query of the following information as a collation target will be described. Hereinafter, the combination of the following information and the person information to be collated may be referred to as the "combination to be collated".

[0016] In Embodiment 1, using the following information, an index called the degree of confirmation indicating the possibility of determining the person information corresponding to the query of the following information is calculated, and the combination to be collated is determined based on the degree of confirmation. Further, in Embodiment 1, when there are a plurality of queries of the following information, the feature amount in the latest frame held by each query of the following information is compared with the feature amount in the past frame, and the query of the following information with a large change in the feature amount is included in the combination to be collated.

[0017] The past frame is a frame that has been processed before the frame being processed, and in Embodiment 1, it is the frame immediately before the latest acquired frame. By using only the query of the following information with a large amount of change as the collation target, it is determined that the query of the following information corresponding to a person with a small visual change with respect to the past frame cannot determine the person information even if person collation is performed again, and the process of person collation is omitted.

[0018] As a procedure, first, a person is detected for each frame of the video, and the following information generated by tracking the person over a plurality of frames is generated. Next, based on the following information, the degree of confirmation is determined for each query of the following information as the target of person collation. Specifically, the degree of confirmation is determined for each query of the following information based on the difference (change amount) of the feature amount in the query of the following information. Then, based on the degree of confirmation, a combination of the following information with a high degree of confirmation and the person information that has not been collated is determined as the collation target from the query of the following information and the person information. Finally, person collation is performed on the combination to be collated.

[0019] Next, the configuration of the information processing apparatus 100 in Embodiment 1 will be described using FIGS. 1 and 2. FIG. 1 is a diagram showing an example of the hardware configuration of the information processing apparatus 100 according to Embodiment 1.

[0020] As shown in Figure 1, the information processing device 100 of Embodiment 1 includes a CPU 101, a ROM 102, a RAM 103, an HDD 104, a display unit 105, an operation unit 106, and a communication unit 107.

[0021] The CPU 101 is a central processing unit (Central Processing Unit) composed of at least one computer, which performs calculations and logical decisions for various processes and controls each component connected to the system bus 108. The ROM (Read-Only Memory) 102 is program memory, which stores programs for control by the CPU 101, including various processing procedures described later. The RAM (Random Access Memory) 103 is used as the main memory, work area, and other temporary storage area of ​​the CPU 101. Alternatively, program memory may be realized by loading a program into the RAM 103 from an external storage device connected to the information processing device 100.

[0022] HDD104 is a hard disk for storing electronic data and programs according to Embodiment 1. An external storage device may be used to perform a similar role. Here, the external storage device can be implemented, for example, by a media (recording medium) and an external storage drive for accessing the media. Examples of such media include flexible disks (FD), CD-ROMs, DVDs, USB memory, MOs, flash memory, etc. The external storage device may also be a server device connected via a network.

[0023] The display unit 105 is, for example, a CRT display or a liquid crystal display, and is a device that outputs an image to the display screen. The display unit 105 may also be an external device connected to the information processing device 100 by wire or wireless connection. The operation unit 106 has a keyboard and mouse and accepts various operations from the user. The communication unit 107 performs two-way communication by wire or wireless connection with other information processing devices, communication equipment, external storage devices, etc., using known communication technologies.

[0024] Figure 2 is an example of a block diagram showing the functional configuration of the information processing device 100 according to Embodiment 1. Each of these functional units is realized by the CPU 101 loading the program stored in the ROM 102 into the RAM 103 and executing processing according to the flowcharts described later. The results of each processing are then stored in the RAM 103 or HDD 104. Furthermore, for example, if hardware is configured as an alternative to software processing using the CPU 101, an arithmetic unit and circuit corresponding to the processing of each functional unit described here should be configured.

[0025] As shown in Figure 2, the information processing device 100 of Embodiment 1 includes an image acquisition unit 210, a detection unit 220, a tracking unit 230, a person matching unit 240, a display control unit 250, a storage unit 260, and a combination determination unit 270.

[0026] The image acquisition unit 210 acquires the video or a series of images to be processed from an external device in chronological order. It also acquires frames extracted from the acquired video. The image acquisition unit 210 also functions as an acquisition means to perform the above-mentioned processes. Details of the processes performed by the image acquisition unit (acquisition means) 210 will be described later.

[0027] The detection unit 220 acquires one frame from the video to be processed, which has been acquired by the image acquisition unit 210, and detects a person from the acquired frame. The detection unit 220 also transmits information about the area surrounding all detected people (detection area) to the tracking unit 230. The detection unit 220 also functions as a detection means to perform the processes described above. Details of the processes performed by the detection unit (detection means) 220 will be described later.

[0028] The person matching unit 240 obtains tracking information queries and unmatched person information from the combination determination unit 270, and performs person matching based on this information. The display control unit 250 displays at least one of the tracking information or a person for each frame of the video displayed on the screen of the display unit 105. The person matching unit 240 also functions as a matching means to perform the above-described processes. Details of the processes performed by the person matching unit (matching means) 240 will be described later.

[0029] The storage unit 260 stores multiple tracking information and person information. The storage unit also manages a database related to tracking information and person information, as shown in Figure 5, which will be described later. Furthermore, the storage unit 260 also functions as a storage means for storing multiple tracking information and person information. Details of the processing performed by the storage unit (storage means) 260 will be described later.

[0030] The combination determination unit 270 calculates a degree of certainty based on the tracking information, the queries in the tracking information, and the unmatched person information, and determines the combination with the highest degree of certainty based on that degree. The combination determination unit 270 also functions as a decision means to carry out each of the processes described above. Details of the processes performed by the combination determination unit (decision means) 270 will be described later.

[0031] The processing details of each functional unit (each means) in Embodiment 1 will be explained in detail below using Figure 3. Figure 3 is a flowchart illustrating the processing of the information processing device 100 in Embodiment 1. In the following explanation, each process (step) will be preceded by an S, and the notation of the process (step) will be omitted. Figure 3(A) is a flowchart illustrating the processing in Embodiment 1. Figure 3(B) is a flowchart detailing the processing of S305 in Figure 3(A). As mentioned above, each operation (process) shown in the flowchart of Figure 3 is controlled by the CPU 101 executing a computer program.

[0032] First, in S301, the image acquisition unit 210 acquires the video or a series of images to be processed from an external device in chronological order. The external device from which images are acquired is, for example, a camera, but is not limited to a camera; it may also be a device such as a server or a device that stores images on a storage medium such as external memory. Furthermore, the external device may have a built-in camera, or it may acquire images from a camera in a remote location via a network such as an IP network.

[0033] In Embodiment 1, the image acquisition unit 210 acquires frames (frame images) extracted from the video shown in Figure 4. In the description of Embodiment 1, the video from time t1 to time tn shown in Figure 4 is used as the target. However, it is assumed that the video, which was shot before time t1, has already been processed to track the movement trajectory of a person in the video. Furthermore, it is assumed that multiple tracking information and person information have already been stored in the storage unit 260.

[0034] Figure 4 is a diagram illustrating the video used in Embodiment 1. Image 410 is the frame at time t1. Image 420 is the frame at time tm. Image 430 is the frame at time tn. In the video shown in Figure 4, at time t1 only person A is shown facing forward, but at the following time tm, person B and person C enter the frame facing backward. At this time, it is assumed that person B is wearing similar clothing to person C. At the following time tn, person B is facing sideways.

[0035] Person A is facing forward from time t1, and their face is clearly visible, making it highly likely that the corresponding person information can be determined through person matching. In other words, it is easy to determine the corresponding person information through person matching. On the other hand, Persons B and C are facing backward from time tm until just before time tn, making it difficult to determine their person information. Therefore, it is unlikely that the person information of Persons B and C can be determined at time tm. However, at time tn, Person B turns to the side, so the face, which is unique information for each person, is visible in the frame. Therefore, it is highly likely that the person information of Person B can be determined. In Embodiment 1, using such video as an example, person matching is performed only on persons whose appearance in the detection area has changed significantly from the previous frame, among the persons corresponding to the tracking information query for whom the person information has not been determined. On the other hand, for persons whose appearance in the detection area has changed little from the previous frame, it is determined that it is unlikely that the person information will be determined even if person matching is performed again, and person matching is omitted.

[0036] Returning to Figure 3, next, at S302, the detection unit 220 acquires one frame from the video to be processed, which has been acquired by the image acquisition unit 210.

[0037] Next, in S303, the detection unit 220 performs the process of detecting a person from the frame. Specifically, the detection unit 220 acquires frames from the video acquired from the image acquisition unit 210, performs detection processing on each frame, and obtains the position information of the person in the frame. In Embodiment 1, detection is performed by a model that has been previously trained to detect people from images. For example, by training the model using a large amount of training data consisting of pairs of images containing people and ground truth images indicating the position of the person in the image, it becomes possible to output information about the position of the person in the frame when a frame is input to the model. Any machine learning algorithm can be used to train the model in Embodiment 1, for example, an algorithm such as a neural network can be used.

[0038] In Embodiment 1, the detection unit 220 acquires information about the detection area of ​​each person in the frame as positional information. Here, the detection area information is defined as the coordinates and size of the upper left corner of the detection area in the frame. The rectangular detection area 411 in image 410 of Figure 4 is an image of the person's detection area (person area). After detecting the detection area information of all people in the frame, the detection unit 220 transmits (outputs) the detected information to the tracking unit 230. For the sake of simplicity, the processing of the frame at time tn will be described below.

[0039] Next, in S304, the tracking unit 230 performs tracking processing (tracking processing) to track a person in the video. Specifically, the tracking processing in the tracking unit 230 generates or updates tracking information by assigning information about the detection area of ​​each person in each frame to different tracking information for each person. As mentioned above, tracking information is information that is a sequence of feature quantities related to the detection area of ​​the person being tracked for each frame, for the number of frames in which the person was detected.

[0040] In Embodiment 1, the tracking unit 230 receives information on each detection region from the detection unit 220 and assigns the information on each detection region of the latest frame at that time as a feature to the tracking information up to the previous frame. This information is used in the assignment process of the tracking unit 230, which will be described later. If the tracking unit 230 determines that a person in a detection region in a frame is highly likely to be a person present in the previous frame (past frame), it adds the feature of the detection region of the latest frame to the tracking information of that same person. Also, if it is determined that a new person has appeared, the tracking unit 230 generates new tracking information.

[0041] The tracking unit 230 performs the assignment process by comparing the features of the detection region of the person in the latest frame with the features of the tracking information generated up to the previous frame. Specifically, it examines the similarity between the features of the detection region of each person in the latest frame and the features of the detection region of each person in the previous frame, and adds the features of the latest frame to the tracking information that is most similar to the previous state. For example, when using location information (coordinates) of the detection region as a feature, the center position of the detection region can be used. Also, when using image features, features obtained by applying color information, texture information, convolutional neural networks (CNNs), etc., to the detection region of the frame can be used. In Embodiment 1, the tracking information generated by the tracking unit 230 is transmitted to the storage unit 260, and the transmitted tracking information is managed by the storage unit 260.

[0042] Figure 5 is a diagram illustrating the information stored in the storage unit 260 according to Embodiment 1. Figure 5(A) is a diagram showing an example of a tracking information database managed by the storage unit 260. In the example shown in Figure 5(A), the tracking ID is used to distinguish the tracking information stored in the storage unit 260.

[0043] Furthermore, as shown in Figure 5(A), the storage unit 260 manages a set of information for each tracking ID, including the time of the frame in which a person was detected, the coordinates and size of the detection area, and image features. In this embodiment 1, the tracking information for tracking IDs track1 to track6 is stored after processing on another video before processing on the video shown in Figure 4. The tracking information for tracking IDs track7, 8, and 9 is the movement trajectory of people A, B, and C in the video shown in Figure 4. For example, the coordinates of the dotted line area 511 indicate the coordinates of the detection area 411 at time t1 of person A in image 410. Image features are vectors of image features obtained from the detection area. For example, image feature f7,1 is a vector of image features obtained from the detection area 411.

[0044] From now on, for simplicity of notation, the tracking information for track7 in tracking ID will be referred to as tracking information track7, and the queries for tracking information for tracks8 and 9 in tracking ID will be referred to as tracking information queries track8 and 9. In addition, the "Person Information Confirmed" column in the database shown in Figure 5(A) will contain information indicating whether the person information corresponding to the tracking information has been confirmed. In the example database shown in Figure 5(A), True indicates that the person information for the tracking information has been confirmed. False indicates that the person information for the tracking information has not been confirmed and is pending. Tracking information marked as False is a query for tracking information that requires further person matching. Tracking information with the "Person Information Confirmed" element set to False includes tracking information that has just been generated (no person matching has been performed yet) and tracking information that has been matched in the past but the person information could not be confirmed.

[0045] Next, in S305, the combination determination unit 270 determines a combination with a high degree of certainty based on the tracking information, the queries for the tracking information, and the unmatched person information. The detailed processing of the combination determination unit 270 is explained below using the flowchart in Figure 3(B). Figure 3(B) is a flowchart detailing the processing of the combination determination unit 270 in S305.

[0046] First, in S3051, the combination determination unit 270 obtains a list of tracking information queries. That is, it obtains a list equal to the number of tracking information queries. Here, as an example of a list, the combination determination unit 270 obtains information from the storage unit 260 for tracking information queries track8 and track9.

[0047] Next, in S3052, select one tracking information query from the list obtained in S3051. Here, as an example, select the tracking information query track8. The tracking information query selected in S3052 will be referred to as the "tracking information query of interest" from now on.

[0048] Next, in S3053, the combination determination unit 270 calculates the degree of certainty based on the tracking information. In Embodiment 1, the combination determination unit 270 determines the degree of certainty based on a comparison between features in the tracking information query. Specifically, the degree of certainty is determined based on the distance in data space between the feature in the latest frame and the feature in a previous frame. Here, the degree of certainty is the difference (amount of change) between the feature in the latest frame and the feature in the previous frame in the tracking information query.

[0049] Figure 6 is a diagram illustrating the processing of the combination determination unit 270 of Embodiment 1. Figure 6(A) is an image diagram of the tracking information query track8 corresponding to person B. Figure 6(B) is an image diagram illustrating how the degree of certainty is determined. The circles in Figure 6 represent the feature quantities of the detection area of ​​one person within one frame, and the feature quantities are arranged in chronological order. Here, for each tracking information query, the difference between feature quantities 611 and 612 at the latest time tn and the previous time tn-1 is calculated.

[0050] For example, suppose the difference in the feature quantities of the tracking information query track8 is 20,000. In this case, the combination determination unit 270 determines (calculates) 20,000, which is the difference in the feature quantities of the tracking information query track8, as the degree of certainty for the tracking information query track8.

[0051] Next, in S3054, it is determined whether the degree of certainty was calculated for all the tracking information queries obtained in S3051. If the degree of certainty was not calculated from all the tracking information queries obtained, the process returns to S3051 and repeats. If the degree of certainty was calculated from all the tracking information queries obtained, the process proceeds to S3055.

[0052] In this case, upon returning to S3051, the combination determination unit 270 selects the tracking information query track9. Subsequently, the combination determination unit 270 calculates the degree of certainty based on the tracking information. At this time, if the difference in the features of the tracking information query track9 is 700, this difference in features, 700, is determined (calculated) as the degree of certainty for the tracking information query track9.

[0053] The reason why the degree of certainty in tracking information query track8 is higher than in tracking information query track9 is, as mentioned above, that at time tn-1, both person B and person C were facing backward, but at time tn, only person B changed to facing sideways. In other words, because the behavior of person B changed significantly from time tn-1 to time tn, the amount of change in tracking information query track8 corresponding to person B is greater. Note that when calculating the degree of certainty, not only the difference between features but also distances such as the Euclidean distance between features may be used. The same processing can be achieved even when using distances such as the Euclidean distance between features.

[0054] Next, in S3055, the combination determination unit 270 performs a process to determine the combination of matching targets based on the calculated degree of certainty, using the tracking information queries and the unmatched person information stored in the storage unit 260. Specifically, tracking information queries with a degree of certainty higher than the threshold are included in the combination of matching targets.

[0055] For example, if the threshold is set to 10000, the confidence level of the tracking information query track8 is 20000, which is above the threshold. Therefore, the combination determination unit 270 includes the tracking information query track8 in the matching target. On the other hand, the confidence level of the tracking information query track9 is 700, which is below the threshold. Therefore, the combination determination unit 270 determines that the difference in feature quantities from the previous frame is small, and that performing person matching again would not change the result, and does not include the tracking information query track9 in the matching target. The person information to be included in the matching target combination is determined by referring to the database of person information in the storage unit 260. Note that the threshold setting is set in advance before the start of the process shown in Figure 3 or before video acquisition shown in Figure 4. Furthermore, the value of the threshold can be set arbitrarily, and the user can input or change it by operating the operation unit 106, etc.

[0056] In Embodiment 1, all person information that has not been matched with tracking information query track8, which was above the threshold mentioned above, is included in the matching target. In addition, tracking information queries that are below the threshold mentioned above are not included in the matching target. By limiting the combination of matching targets in this way, Embodiment 1 can omit the process of matching tracking information queries that have a low probability of identifying a person with person information.

[0057] Here, we will explain the database of person information stored by the storage unit 260 at time tn using Figure 5(B). As mentioned above, in Embodiment 1, a series of processes are performed to track the movement trajectories of multiple people in the video before acquiring the video shown in Figure 4, and the tracking information track1 to track6 are assumed to have been generated before acquiring the video shown in Figure 4. As shown in the column of person IDs in the person information in Figure 5(B), the storage unit 260 stores person IDs of person information person1 to person4 as person information corresponding to people who have already appeared in the video. Furthermore, it is assumed that each piece of person information has a confirmed correspondence with one or more pieces of tracking information in the past.

[0058] Hereafter, we will say that the tracking information and the person information are related when there is tracking information that has been determined to correspond to the person information up to the latest frame. For example, the person information person1 shown in Figure 5(B) is related to tracking information track1, 4, and 7.

[0059] Furthermore, in Embodiment 1, the person information also holds information on whether the associated tracking information exists in the latest frame. In the example in Figure 5(B), the "Does it exist in the latest frame?" column for each person ID is set to True if it exists. In addition, the "Does it exist in the latest frame?" column for each person ID is set to False if it does not exist. For example, person A in the video in Figure 4 has already been identified as person information person1, which corresponds to tracking information track7. Therefore, in Figure 5(B), the "Does it exist in the latest frame?" element for person information person1 is True. On the other hand, person information person2, 3, and 4, for which the "Does it exist in the latest frame?" element is False, are person information for which there is no corresponding tracking information in the frame being processed, i.e., person information that has not been matched.

[0060] Returning to Figure 3, in S306, the person matching unit 240 performs person matching processing. The person matching unit 240 obtains the tracking information query and unmatched person information from the combination determination unit 270 and performs person matching. The tracking information query and unmatched person information are collectively referred to as the combination information of the matching target. Specifically, person matching is performed based on the similarity between the feature quantities of the tracking information query and the tracking information feature quantities associated with the person information.

[0061] For example, in Embodiment 1, the feature similarity is calculated for all combinations of all features of the tracking information query and all features of the tracking information associated with the person information. When the average similarity exceeds a predetermined threshold (predetermined value) for a person, the person information corresponding to that tracking information query is determined. Equation (1) below shows an example of an equation for calculating the average similarity for person matching between one tracking information query and one person piece of information.

number

[0062] However, the similaritymean shown in equation (1) above is the average of the similarities. The denominator is the total number of combinations of features held by the tracking information and features held by the tracking information associated with all the person information being matched. The numerator is the sum of the similarity scores calculated for each of these combinations.

[0063] The meaning of each variable is explained below. First, tn is the number of tracking information associated with the person information, and ti is a variable that changes from 1 to tn for each tracking information associated with the person information. fn(ti) is an array of the number of features held by each tracking information associated with the person information, and has a different value for each tracking information. fi is a variable that changes from 1 to fn(ti) for each feature held by the tracking information associated with the person information. qfn is the number of features held by the tracking information query, and qfi is a variable that changes from 1 to qfn for each feature held by the tracking information query. f is a vector of features. F is a function that calculates the similarity between features. An example of the formula for function F is shown in equation (2) below.

number

[0064] When person matching is performed, the similarity calculation is performed a number of times corresponding to the number of combinations of the features held by the tracking information and the features held by the tracking information associated with each person's information, as shown in equation (1) above. Therefore, the computational load becomes enormous as the number of person matches increases. For this reason, Embodiment 1 aims to reduce the computational load by limiting the queries for the tracking information to be matched based on the degree of certainty.

[0065] In Embodiment 1, as described above, the tracking information query track8 is the target of matching with person information, while the tracking information query track9 is not. Therefore, the following describes the person matching process targeting the combination of the tracking information query track8 and person information persons 2, 3, and 4.

[0066] Figure 7 illustrates an example of the processing of the person matching unit 240 in Embodiment 1. In Embodiment 1, the threshold value is set to "0.9". This threshold value is set in advance before the start of the processing shown in Figure 3 or before the acquisition of video in Figure 4. The value of this threshold can be set arbitrarily and can be input or changed by the user operating the operation unit 106, etc.

[0067] As shown in Figure 7, the similarity between the tracking information query track8 and the person information person2 is "0.90", which is above the threshold. As a result, person information person2 is confirmed for tracking information query track8. When person information is confirmed for tracking information query, the storage unit 260 updates the tracking information and person information databases. Specifically, it changes the cell in the "Person Information Confirmed" column for tracking information query track8 in the table in Figure 5(A) from False to True. In addition, it adds tracking information query track8 to the tracking information query column in the row for person information person2 in the table in Figure 5(A), and changes the element in the "Is it in the latest frame?" column to True.

[0068] On the other hand, for the tracking information query track9, which was not selected as a matching target (not included in the matching target), the element in the "Person Information Confirmed" column of the tracking information database in Figure 5(A) is not changed from False. In other words, for the tracking information query track9, the person information cannot be confirmed and remains pending.

[0069] Next, in S307, the display control unit 250 updates the display screen of the display unit 105. The display control unit 250 outputs at least one of tracking information or person information for each frame in the video. Specifically, the display control unit 250 displays a detection area frame as shown in Figure 5 around the person in the frame, and further displays the tracking ID or person ID around the frame. The display control unit 250 may also color the frame to be displayed with an arbitrary color and display it on the screen of the display unit 105. For example, taking the image 430 at time tn in Figure 4 as an example, the display control unit 250 can color each of the detection areas of each person, frames 431, 432, and 433, with the same or different colors and display them on the screen of the display unit 105.

[0070] Furthermore, the display control unit 205 may output numerical information about the displayed person, such as a tracking ID or person ID, around each frame. Alternatively, the frame color may be displayed differently depending on the difference in the tracking ID or person ID. The display control unit 250 may display only the frame of the detection area on the screen of the display unit 105, or it may display the tracking ID or person ID without displaying the frame. Furthermore, the color of the displayed frame may be unified. Furthermore, the thickness of the frame may be changed depending on the difference in the tracking ID or person ID. The display control unit 205 also functions as a display control means for performing the above-described processes.

[0071] Next, in S308, CPU101 determines whether the series of processes has been completed for all frames in the video. If the determination is made and the series of processes has been completed for all frames in the video, the process terminates. On the other hand, if the series of processes has not been completed for all frames in the video, the process returns to S302 and repeats the process from S302 to S308.

[0072] By the above method, in the process of tracking the movement trajectory of a person in a video, it is possible to determine a combination with a high degree of certainty from the tracking information and the person information and perform person matching. According to Embodiment 1, since person matching can be omitted for tracking information that has a low probability of confirming the person's identity, the number of person matching operations can be reduced, and the amount of computation can be reduced.

[0073] If the method and processing in Embodiment 1 are not performed, a computational amount will be incurred corresponding to the total number of combinations of tracking information queries and unmatched person information. However, the information processing device 100 in Embodiment 1 can suppress costly computations due to person matching. In Embodiment 1, the benefit of reducing computational amount by reducing the number of person matchings is greater than the increase in computational amount due to the processing of the combination determination unit 270.

[0074] In Embodiment 1, all unmatched person information is used for person matching. The above is a description of the database of person information stored in the storage unit 260 at time tn. Specifically, in Embodiment 1, the combination determination unit 270 targets the combination of tracking information query track8 and all unmatched person information person2, 3, and 4 for matching.

[0075] By the above method, the combination determination unit 270 can determine a combination of tracking information and person information that has a high probability of confirming the person information in response to the tracking information query, from among the tracking information query and the person information that has not been matched.

[0076] Furthermore, the processing order of the information processing device 100 in Embodiment 1 is not limited to the processing order shown in Figure 3. In the above, it was explained that the tracking unit 230 acquires image features (feature quantities) from the person detection area for each frame to generate and update tracking information, but image feature extraction does not have to be performed for all frames. Specifically, image features are extracted from the latest detection area information of the tracking information included in the combination of matching targets. On the other hand, image features are not extracted from the latest detection area information of the tracking information not included in the combination of matching targets. This method is more efficient because it reduces the number of feature extractions.

[0077] Furthermore, although the above explanation states that the storage unit 260 stores tracking information and person information, older information may be deleted. For example, when the number of features held in the tracking information exceeds a predetermined number, older features are deleted so that the number of features does not exceed the predetermined number. By deleting old data, it is possible to prevent an increase in the memory load.

[0078] Furthermore, although the above states that the past frame used by the combination determination unit 270 when calculating the degree of certainty is the most recent frame, the past frame may be a frame from several frames prior. For example, when processing video shot at a high frame rate, the changes in the video from frame to frame are small, so using a frame from several frames prior makes it easier to distinguish between the image features of people who have changed in appearance and those who have not. Moreover, the user can arbitrarily set which past frame is used when calculating the degree of certainty.

[0079] Furthermore, while the above shows that the combination determination unit 270 determines the degree of certainty based on a method based on distance in the data space, there is also a method based on the similarity between features in the tracking information query. Specifically, the similarity between the features in the latest frame of the tracking information query and the features in past frames is calculated, and the degree of certainty is determined based on the similarity.

[0080] For example, in this case, the past frame is the immediately preceding frame. In the example in Figure 6(A), the similarity between the latest feature 611 and the immediately preceding feature 612 of the tracking information query track8 is calculated. The similarity is calculated using the following equation (3).

number

[0081] Furthermore, the past frames used in the method based on the similarity between features within the tracking information query are not limited to one, but may be multiple. Specifically, the past frames can be all frames held by the tracking information query, a predetermined number of frames, or frames selected every few frames.

[0082] For example, the similarity between the features of each past frame and the frame being processed is calculated using equation (2) above. Then, for example, the maximum value is taken from among the multiple similarities calculated, and the value obtained by subtracting the maximum similarity from 1 is used as the confidence score. With this method, if the features of the tracking information that could not confirm the person's information in past frames are similar to at least one of them, person matching can be omitted.

[0083] As mentioned above, the combination determination unit 270 calculates the degree of certainty based on a comparison of features within the tracking information queries. However, it can also determine the degree of certainty based on a comparison of features between tracking information queries. Specifically, the degree of certainty is determined based on the similarity between the tracking information query under consideration and the queries of other tracking information. When the similarity is low, it is interpreted that there is a possibility that unique features for each person, such as faces, are clearly visible in the frame, and the degree of certainty for the tracking information query under consideration is increased.

[0084] For example, when people wearing similar clothing are facing away from the viewer, the similarity of image features between the people will be high, but if at least one person is facing sideways, the visual similarity to other people in the image will decrease. Figure 6(B) will be used to explain how the degree of certainty is determined in this case. 621, 622, and 623 shown in Figure 6(B) are queries for tracking information. The circles in Figure 6 represent the feature quantities of one person within one frame, and the feature quantities are arranged in chronological order.

[0085] For example, when determining the degree of certainty using 621 as a tracking information query, the similarity between the latest feature 624 of tracking information query 621 and the latest features 625 and 626 of tracking information queries 622 and 623 is calculated, and the average of the similarities is calculated as shown in equation (4) below.

number

[0086] An example of calculating the degree of certainty is explained using Figure 6(B). First, the similarity between the latest feature 624 in query 621 of the tracking information under consideration and the past feature 627 is calculated using equation (3) above. Next, the similarity between query 621 of the tracking information under consideration and the latest features of queries 622 and 623 of the other tracking information is calculated using equation (4) above. Then, for example, the degree of certainty is the value obtained by dividing the similarity between features in the query of the tracking information, similarityself, by the similarity between the features of the query of the tracking information and the features of the queries of the other tracking information, similarityother.

[0087] Thus, a method based on both comparisons between features within tracking information queries and comparisons between tracking information queries, while slightly increasing computational complexity compared to a method based solely on the latter, can significantly reduce the computational complexity of person matching while suppressing mismatches.

[0088] Furthermore, the combination determination unit 270 may forcibly lower the degree of certainty when a portion of the detection area of ​​the latest frame of the tracking information is hidden. Specifically, for queries of tracking information where the detection area in the latest frame overlaps between multiple people, the degree of certainty is lowered. This is because when multiple people are included in the image range of the detection area, the image features of multiple people are included in a single feature quantity, making it unlikely that person matching can be performed to confirm the identity. For example, the presence or absence of overlap is determined by comparing the magnitude relationship between the x and y coordinates of the endpoints of one detection area and the x and y coordinates of the endpoints of the other rectangle. This method reduces the computational load on the combination determination unit 270 compared to the similarity-based method described above.

[0089] Furthermore, although the person matching unit 240 performs person matching based on image features in the detection area as described above, the method of person matching is not limited to this method. A method using facial recognition may also be used. In this method, the person matching unit 240 performs face detection within the image range of the person linked to the tracking information, and applies a pre-prepared facial recognition model to the image range of the detected face.

[0090] The facial recognition model takes an image containing a face as input and outputs a unique ID for each person. The facial recognition model is applied to the face detection area of ​​each tracking information. When the facial recognition result obtained from the tracking information query matches the facial recognition result obtained from the tracking information associated with the person, the person information corresponding to that tracking information is determined. Because the method using facial recognition is based on information that is definitely different for each person, the accuracy of the person matching results is higher than other methods.

[0091] Furthermore, although the display control unit 250 is described above as outputting tracking information or person information for each frame, the processing of the display control unit 250 is not limited to this. For tracking information that has not confirmed the person's information for a certain period of time, the reason why the person's information cannot be confirmed may be displayed (text information may be displayed). For example, information such as "Person matching has not been performed because there is not much movement" may be displayed around that person. With this method, the user can find out why the person's information is not being confirmed for a person in the video.

[0092] <Embodiment 2> In Embodiment 1, the combination determination unit 270 determines the combination by limiting the tracking information queries based on the degree of certainty. In Embodiment 2, however, the person information to be matched is also further limited.

[0093] Figure 8 is an example of a block diagram showing the functional configuration of the information processing device 100 according to Embodiment 2. In Embodiment 2, a filtering unit 810 is added as a new component. The other components are the same as in Embodiment 1, so redundant explanations are omitted.

[0094] The filtering unit 810 performs a process to narrow down the candidate person information corresponding to the tracking information query from the person information that has not been matched. In Embodiment 2, the filtering unit 810 narrows down the candidate person information for the tracking information query based on a comparison of the feature quantities of the tracking information query and the feature quantities of the tracking information associated with the person information, and further links the tracking information query with the candidate. Hereafter, the information that links the tracking information query with the candidate person information will be referred to as "linking information". The filtering unit 810 passes the linking information to the combination determination unit 270. The combination determination unit 270 determines the tracking information query to be matched based on the degree of certainty, and further determines the person information to be matched based on the linking information.

[0095] The processing flow of Embodiment 2 will be explained using Figure 9. Figure 9 is a flowchart illustrating the processing of the information processing device 100 in Embodiment 2. Note that the processing in Embodiment 2 differs from the processing in Embodiment 1 in that processing by the narrowing unit 810 is added and the processing content of the combination determination unit 270 is changed. Note that a detailed explanation of the processing similar to that shown in Figure 3, which illustrates the processing in Embodiment 1, will be omitted.

[0096] The following describes how the filtering unit 810 filters the corresponding person information for the tracking information query, through an explanation of the processing for the frame at time tm shown in Figure 4. Next, the combination determination unit 270 determines the matching target combination using the processing results of the filtering unit 810, through an explanation of the processing for the frame at time tn.

[0097] First, let's explain the processing at time tm. In Embodiment 2, as in Embodiment 1, processing is performed on other videos before processing the video shown in Figure 4, and it is assumed that information has already been stored in the database at this point. Also, in Embodiment 2, it is assumed that at time tm, the storage unit 260 has stored the database of person information shown in Figure 5(B). The table shown in Figure 5 was explained in Embodiment 1, so a detailed explanation will be omitted, but the database contains person information persons 2, 3, and 4 that have not been matched. The frame at time tm is the frame immediately after persons B and C enter the frame.

[0098] First, in S301, the image acquisition unit 210 acquires the video shown in Figure 4. Next, in S302, the detection unit 220 acquires image 420 (acquires one frame from the video). Next, in S303, the detection unit 220 performs a process to detect a person from image 420. Next, in S304, the tracking unit 230 performs a tracking process using the information of the detection area of ​​the detected person. At this time, the tracking unit 230 generates tracking information queries track 8 and 9 corresponding to person B and person C.

[0099] Next, in S305, the combination determination unit 270 determines the combination of matching targets. The processing in S305 of Embodiment 2 is carried out through the processing flow shown in Figure 3(B), similar to Embodiment 1. However, at time tm, since person B and person C have just entered the frame, it is not possible to compare the feature quantities of the frame prior to time tm with the feature quantities of the latest frame, and therefore the degree of certainty cannot be calculated in S3053 using the method described in Embodiment 1. In this case, in cases where comparison with the previous frame is not possible, the degree of certainty is forcibly set to 1, and the query for the tracking information under consideration is included in the combination of matching targets. The combination determination unit 270 also determines that the tracking information queries track8 and 9 corresponding to person B and person C, and all person information that has not been matched, person2, 3, and 4, will be the combination of matching targets.

[0100] Next, in S306, the person matching unit 240 performs person matching. Here, similar to Embodiment 1, the similarity between the tracking information query and the tracking information feature quantities associated with the person information is calculated, and if there is a combination that exceeds a predetermined threshold, the person information for the tracking information query is determined. However, in Embodiment 2, unlike Embodiment 1, if there are multiple pieces of person information with high similarity to the tracking information query, the person information is not determined. For example, for a given tracking information query, it is checked whether there is any person information with similarity exceeding a predetermined threshold. If such person information exists and there is only one such piece of person information, the person information is determined for the tracking information query. If there are two or more pieces of person information that match, the person information is not determined.

[0101] Next, in S901, the filtering unit 810 determines candidate person information for tracking information queries in which person information could not be determined. In Embodiment 2, the filtering unit 810 determines candidate person information based on the similarity between the features held by the tracking information queries and the features held by the tracking information associated with the person information that could not be matched.

[0102] Specifically, the filtering unit 810 uses the similarity calculated in S306. For example, if there are multiple pieces of person information with similarity exceeding a predetermined threshold for a given tracking information query, the filtering unit 810 selects the corresponding person information as a candidate for that tracking information query. Subsequently, the filtering unit 810 stores the association information between the tracking information query and the candidate person information in the storage unit 260.

[0103] Figure 10 is a diagram illustrating the linking information generated by the filtering unit 810. The dotted line area 1001 in Figure 10 indicates that person information person2 and person3 are linked as candidates to the tracking information query track8.

[0104] Returning to Figure 9, in S307, the display control unit 250 updates the screen in the same manner as in Embodiment 1. Next, in S308, the CPU 101 determines whether the series of processes has been completed for all frames. If the determination shows that the series of processes has been completed for all frames in the video, the process is terminated. On the other hand, if the series of processes has not been completed for all frames in the video, the process returns to S302 and repeats the process from S302 to S308.

[0105] The above is a description of the processing of image 420 (frame) at time tm. Up to this point, we have explained that the filtering unit 810 narrows down the candidate person information corresponding to the tracking information query from the person information that has not been matched. Next, we will explain the processing of image 430 (frame) at time tn and show how the combination determination unit 270 determines the combination of matching targets based on the degree of certainty and the linking information. As mentioned above, at time tm both person B and person C were facing away, but at time tn only person B has changed to a sideways orientation.

[0106] First, the processing from S302 to S304 is performed in the same way as the processing at time tm. Next, in S305, the processing of the combination determination unit 270 will be explained in detail with reference to Figure 3(B).

[0107] First, in S3051, the combination determination unit 270 obtains a list of tracking information queries. Next, in S3052, the combination determination unit 270 selects one tracking information query. Next, in S3053, the combination determination unit 270 calculates the degree of certainty for the tracking information query selected in S3052. Next, in S3054, it is determined whether or not the degree of certainty has been calculated for all tracking information queries obtained in S3051. If the degree of certainty has not been calculated for all the tracking information queries obtained, the process returns to S3051 and repeats. If the degree of certainty has been calculated for all the tracking information queries obtained, the process proceeds to S3055. Based on the processing up to this point, in Embodiment 2, similar to Embodiment 1, a higher degree of certainty is obtained for tracking information query track 8 corresponding to person B compared to tracking information query track 9 corresponding to person C.

[0108] Next, in S3055, the combination determination unit 270 determines the combination of matching targets. In Embodiment 2, the combination determination unit 270 determines the query for the tracking information of the matching targets based on the degree of certainty, and further determines the person information to be included in the combination based on the query for the tracking information and the associated information. Here, in the same manner as in Embodiment 1, the combination determination unit 270 decides to include the tracking information query track8 in the combination of matching targets.

[0109] Next, the combination determination unit 270 refers to the linking information shown in Figure 10 and decides to include person information person2 and person3 as candidates for person information linked to tracking information query track8 in the combination of matching targets. Therefore, the combination determination unit 270 determines tracking information query track8 and person information person2 and person3 as the combination of matching targets. On the other hand, tracking information query 9 is not included in the combination of matching targets. The combination determination unit 270 passes the combination information to the person matching unit 240. Once this processing is complete, the process returns to S306 in Figure 9.

[0110] Next, in S306, the person matching unit 240 performs person matching using the tracking information query included in the combination and the person information that has not been matched. In Embodiment 2, it is assumed that the person information person2 is determined for the tracking information query track8.

[0111] Next, in S307, the display control unit 250 updates the display screen in the same manner as in Embodiment 1. Finally, in S308, the CPU 101 determines whether the series of processes has been completed for all frames. If the determination shows that the series of processes has been completed for all frames in the video, the process is terminated. On the other hand, if the series of processes has not been completed for all frames in the video, the process returns to S302 and repeats the process from S302 to S308.

[0112] The above is a description of the processing of the frame at time tn. Using the method described above, the filtering unit 810 narrows down the candidates for person information corresponding to the tracking information query, and the combination determination unit 270 can appropriately determine the combination of matching targets based on the degree of certainty and the association information.

[0113] As described above, the information processing device 100 in Embodiment 2, with its configuration including a filtering unit 810, performs the above-described processing, thereby limiting the number of person information entries to be included in the combination of matching targets, and thus further reducing the computational load for person matching.

[0114] In the above description, the filtering unit 810 determines candidate person information corresponding to a tracking information query based on the similarity between the feature quantities of the tracking information query up to the previous frame and the feature quantities of the person information. However, the method of determining candidates is not limited to this method. For example, the filtering unit 810 can also narrow down candidates based on information about the intersection of people in the video. Specifically, it detects the occurrence of an intersection of people in the video based on multiple tracking information and estimates the tracking information related to the intersection. Then, for the tracking information related to the intersection after the intersection occurs, it narrows down the candidates to person information corresponding to the tracking information query that was related before the intersection occurred.

[0115] An example of the processing in this case will be explained using Figure 11. Figure 11 is a diagram illustrating the video used in a modified example of Embodiment 2. Image 1110 in Figure 11 is the frame at time t1. Image 1120 is the frame at time tm. Image 1130 is the frame at time tn. Note that Figure 11 is a different video from Figure 4.

[0116] In the video shown in Figure 11, from time t1, person D and person E move in the direction of the solid arrows, and at time tm, person D and person E intersect. The dotted arrows represent the movement trajectories of the people being tracked by the tracking information generated by the tracking unit 230. Due to this intersection, the tracking information corresponding to person D is divided, as shown in the tracking information 1111 and 1114 corresponding to person D. Furthermore, the tracking information corresponding to person E is also divided, as shown by the dotted arrows 1112 and 1113 corresponding to person E. Note that for tracking information 1111 and 1112, it is assumed that the corresponding person information was determined before time tm.

[0117] The filtering unit 810 first detects the occurrence of an intersection. The detection of an intersection is performed, for example, based on the positional relationship of the detection regions of the feature quantities held by multiple tracking information. Specifically, in order to detect (check for) an intersection for a given tracking information, the distance between the center position of the detection region held by that tracking information and the center position of the detection region of other tracking information is calculated for each frame. Then, if there is a frame in which this distance is less than a predetermined threshold, it is determined that an intersection occurred at the time corresponding to that frame.

[0118] In the example shown in Figure 11, it is assumed that an intersection was detected in the frame at time tm based on the positional relationship of the detection areas held by tracking information 1111, 1112, 1113, and 1114. The filtering unit 810 determines candidate person information corresponding to the tracking information query based on the information that tracking information 1111 and 1112 were related to the intersection before the intersection occurred, and 1113 and 1114 were related to the intersection after the intersection occurred. Here, the filtering unit 810 determines that the candidate person information corresponding to the tracking information queries 1113 and 1114 that are related after the intersection occurred is the person information related to tracking information 1111 and 1112 that were related before the intersection occurred.

[0119] In this way, the filtering unit 810 can narrow down the candidate person information based on tracking information related to the intersection. This method allows for the appropriate narrowing down of person information even when an intersection occurs.

[0120] Furthermore, while the above explanation indicates that the combination determination unit 270 determines the combination of matching targets based on the linking information and the degree of certainty, the method of determining the combination is not limited to this. For example, after determining the combination of matching targets based on the degree of certainty and the linking information, the tracking information queries and person information not included in that combination can be determined as a separate new combination.

[0121] For example, in the above explanation, the combination determination unit 270 determined that the tracking information query track8 and the person information persons 2 and 3 were to be matched. However, the remaining tracking information query track9 and the remaining person information person 4, which has not yet been matched, could also be determined as a new combination. In this way, by determining the remaining combinations after determining the initial combination and performing person matching for each, the number of unnecessary person matches can be further reduced by dividing the person matching into combinations that have a high probability of confirming person information for the tracking information query.

[0122] Furthermore, the combination determination unit 270 assumes that there are multiple tracking information queries associated with a certain person's information in the linked information. In this case, when person information is determined for at least one tracking information query, the remaining tracking information queries associated with the same candidate may be used as the next matching target. This is because determining person information corresponding to a similar tracking information query for a certain person increases the likelihood of determining person information for other tracking information queries.

[0123] For example, in the case of the similarity calculation result during person matching shown in Figure 7, the filtering unit 810 determines that the candidate person information for both tracking information queries track8 and 9 is person information person2 and 3. Then, the combination determination unit 270 determines that tracking information queries track8 and 9 and person information person2 and 3 are the combination to be matched.

[0124] Subsequently, if the person matching unit 240 determines that the person information for tracking information query track8 is person2, the combination determination unit 270 will then use tracking information query track9 as the matching target for the next loop. This method allows the combination determination unit 270 to omit the calculation of the degree of certainty for the remaining tracking information queries that were associated with the same candidate.

[0125] This embodiment includes the following configurations, methods, and programs. (Composition 1) An information processing device comprising: detection means for detecting a person in a frame for each of multiple frames; tracking means for generating tracking information, which is information containing feature quantities of the people detected from the multiple frames, in accordance with the number of people detected by the detection means; storage means for storing person information, which is information that associates the tracking information with at least one piece of tracking information relating to the same person; determination means for determining a combination of tracking information and person information as a matching target based on feature quantities in order to perform matching of person information corresponding to the tracking information; and matching means for matching the tracking information and person information determined by the determination means.

[0126] (Configuration 2) The information processing device according to configuration 1, characterized in that the tracking information includes first tracking information in which the person's information is confirmed, and second tracking information in which the person's information is not confirmed.

[0127] (Composition 3) The information processing device according to configuration 2, characterized in that the person information includes a first person information that has been matched and a second person information that has not been matched.

[0128] (Composition 4) The information processing device according to configuration 3, characterized in that the decision means determines whether or not to use the second tracking information as a target for matching with the second person information, based on the difference between the features in the second tracking information.

[0129] (Composition 5) The information processing apparatus according to configuration 3 or 4, characterized in that the determination means calculates the difference in feature quantities between multiple second tracking information by comparing the feature quantities in each of the multiple second tracking information, and determines the combination to be matched based on the difference.

[0130] (Composition 6) The determination means is an information processing device according to any one of configurations 3 to 5, characterized in that if the difference in the feature quantities between the most recently acquired frame and the frame immediately preceding the most recently acquired frame in the second tracking information is greater than or equal to a predetermined threshold, the second tracking information is included as a target for matching with the second person information.

[0131] (Composition 7) The determination means is characterized in that, if the difference in the feature quantities between the most recently acquired frame and the frame immediately preceding the most recently acquired frame in the second tracking information is less than a predetermined threshold, the second tracking information is not included as a target for matching with the second person information, as described in configurations 3 to 6 of the information processing device.

[0132] (Composition 8) The information processing apparatus according to any one of configurations 3 to 7, characterized in that the determination means determines the combination to be matched according to the similarity between the features in the second tracking information.

[0133] (Composition 9) The matching means is characterized by performing a matching based on the similarity between the feature quantities of the second tracking information and the feature quantities of the second tracking information associated with the person information, as described in any one of configurations 3 to 8.

[0134] (Composition 10) An information processing device according to any one of configurations 3 to 9, characterized in that it has a filtering means for narrowing down candidates for second person information corresponding to second tracking information, based on a comparison of the features of second tracking information and the features of second tracking information associated with second person information, and for linking the second tracking information with the candidates.

[0135] (Composition 11) The information processing device according to configuration 10, characterized in that the determination means determines the second person information to be included in the combination based on the association with the second tracking information to be included in the combination.

[0136] (Composition 12) The information processing device according to configuration 10 is characterized in that the narrowing means detects the intersection of people in the video based on multiple tracking information, estimates multiple tracking information related to the intersection, and, for tracking information related to the intersection after the intersection occurs, selects person information corresponding to tracking information that was related to the intersection before the intersection occurred as a candidate.

[0137] (Composition 13) The information processing device according to any one of configurations 3 to 12, characterized in that the determination means determines the second tracking information and second person information that are not included in the combination as a new combination for matching.

[0138] (Composition 14) The determination means is an information processing device according to any one of configurations 3 to 13, characterized in that when there are multiple second tracking pieces of information associated with a second person information, if the person information corresponding to at least one second tracking piece of information is determined, the remaining second tracking pieces of information associated with the candidate are included in the combination.

[0139] (Composition 15) An information processing device according to any one of configurations 1 to 14, characterized in that it has a display control means for displaying at least one of tracking information or person information on a display unit for each frame in the video.

[0140] (Composition 16) The information processing device according to configuration 15, characterized in that the display control means displays at least one of the following in the area surrounding the person to be displayed on the display unit: a numerical value relating to the person, characters indicating the reason why the person has not been identified, and a frame surrounding the person.

[0141] (Composition 17) A method for controlling an information processing device, characterized by detecting a person in a frame for each of multiple frames, generating tracking information which is information containing feature quantities of the people detected from the multiple frames according to the number of people, accumulating person information which is information that associates the tracking information with at least one piece of tracking information relating to the same person, determining a combination of tracking information and person information based on feature quantities in order to match the person information corresponding to the tracking information, and matching the determined tracking information with the person information.

[0142] (Composition 18) A program for causing a computer to execute a control method for an information processing device, characterized by detecting a person in a frame for each of multiple frames, generating tracking information which is information containing feature quantities of the people detected from the multiple frames according to the number of people, accumulating person information which is information that associates the tracking information with at least one piece of tracking information relating to the same person, determining a combination of tracking information and person information based on feature quantities in order to match the person information corresponding to the tracking information, and matching the determined tracking information with the person information.

[0143] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0144] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0145] 100 Information Processing Devices 210 Image acquisition unit 220 Detection unit 230 Tracking part 240 Person Matching Unit 250 Display Control Unit 260 Storage Unit 270 Combination Determination Section

Claims

1. A detection means for detecting a person in each of the frames, Tracking means that generates tracking information, which is information containing the characteristic quantities of the person detected from the plurality of frames, in accordance with the number of people detected by the detection means, A storage means for storing person information, which is information that associates the aforementioned tracking information with at least one piece of tracking information relating to the same person, In order to perform matching of the person information corresponding to the tracking information, a determination means determines the combination of the tracking information and the person information to be matched based on the similarity between the features in the tracking information in which the person has not yet been identified, The system includes a matching means for comparing the tracking information determined by the determination means with the person information, An information processing device characterized by the following:

2. The information processing apparatus according to claim 1, characterized in that the tracking information includes first tracking information in which the person information is confirmed, and second tracking information in which the person is not confirmed.

3. The information processing apparatus according to claim 2, characterized in that the person information includes the first person information for which the matching has been performed and the second person information for which the matching has not been performed.

4. The information processing apparatus according to claim 3, characterized in that the determination means determines whether or not to use the second tracking information as a target for matching with the second person information, based on the difference between the feature quantities in the second tracking information.

5. The information processing apparatus according to claim 3, wherein the determination means calculates the difference in feature quantities between the plurality of second tracking information by comparing the feature quantities in each of the plurality of second tracking information, and determines the combination to be matched based on the difference.

6. The information processing apparatus according to claim 3, wherein the determination means includes the second tracking information as a target for matching with the second person information when the difference in the feature quantities between the most recently acquired frame and the frame immediately preceding the most recently acquired frame in the second tracking information is greater than or equal to a predetermined threshold.

7. The information processing apparatus according to claim 6, characterized in that the determination means does not include the second tracking information in the matching target with the second person information if the difference in the feature quantities between the most recently acquired frame and the frame immediately preceding the most recently acquired frame in the second tracking information is less than a predetermined threshold.

8. The information processing apparatus according to claim 2, characterized in that the matching means performs the matching based on the similarity between the feature quantities of the second tracking information and the feature quantities of the second tracking information associated with the person information.

9. The information processing device according to claim 3, further comprising a filtering means for narrowing down candidates for the second person information corresponding to the second tracking information, based on a comparison of the feature quantities of the second tracking information and the feature quantities of the second tracking information associated with the second person information, and for linking the second tracking information with the candidates.

10. The information processing apparatus according to claim 9, characterized in that the determination means determines the second person information to be included in the combination based on the second tracking information to be included in the combination and the association.

11. The information processing device according to claim 9, wherein the narrowing means detects the intersection of the person in the video including the plurality of frames based on the plurality of tracking information, estimates the plurality of tracking information related to the intersection, and sets the person information corresponding to the tracking information that was associated with the intersection before the intersection occurred as the candidate for the tracking information associated with the intersection after the intersection occurred.

12. The information processing apparatus according to claim 3, characterized in that the determination means determines the second tracking information and the second person information that are not included in the combination, as a new combination for matching.

13. The information processing apparatus according to claim 9, wherein when there are multiple second tracking pieces of information associated with the second person information, and the person information corresponding to at least one of the second tracking pieces of information is determined, the remaining second tracking pieces of information associated with the candidate are included in the combination.

14. The information processing apparatus according to claim 1, characterized in that it has a display control means for displaying at least one of the tracking information or the person information on a display unit for each frame in a video including the plurality of frames.

15. The information processing apparatus according to claim 14, characterized in that the display control means displays at least one of the following in the area surrounding the person to be displayed on the display unit: a numerical value relating to the person, characters indicating the reason why the person has not been identified, and a frame surrounding the person.

16. A method for controlling an information processing device, Detection of a person in each of the frames, Tracking information, which includes the characteristic quantities of the person detected from the multiple frames, is generated according to the number of people. The aforementioned tracking information is linked to at least one piece of tracking information relating to the same person, and person information is stored. In order to match the person information corresponding to the tracking information, the combination of the tracking information and the person information is determined based on the similarity between the features in the tracking information in which the person has not yet been identified. A control method for an information processing device, characterized by comparing the determined tracking information with the person information.

17. A program that causes a computer to execute a control method for an information processing device, Detection of a person in each of the frames, Tracking information, which includes the characteristic quantities of the person detected from the multiple frames, is generated according to the number of people. The aforementioned tracking information is linked to at least one piece of tracking information relating to the same person, and person information is stored. In order to match the person information corresponding to the tracking information, the combination of the tracking information and the person information is determined based on the similarity between the features in the tracking information in which the person has not yet been identified. A program characterized by comparing the determined tracking information with the person information.

Images