Person tracking device, person tracking system, and person tracking method

Abstract

This makes it possible to track a large number of people while keeping computational costs down. [Solution] A first frame group is selected from multiple frame groups, the frame data is traced in chronological order from the end time of the first frame group toward the future to search for other frame groups, another frame group whose start time falls within a predetermined time from the end time of the first frame group is selected as a second frame group, the feature distance between the first frame group and the second frame group is calculated, a person distance is determined based on the feature distance, if the person distance is equal to or less than a threshold, it is determined that the people are the same person, if the person distance is greater than the threshold, it is determined that the people are not the same person, and if the ratio of the distance of a certain element of the feature vector to the feature distance is greater than a predetermined threshold, the element is determined to be an important element.

Description

[Technical field]

[0001] The present invention relates to a person tracking device, a person tracking system, and a person tracking method that can be used, for example, when tracking a person using a plurality of cameras. [Background technology]

[0002] For commercial facilities, attracting customers through events and advertisements is extremely important. For this reason, there is a demand for efficient analysis of visitor movement and the effectiveness of each measure.

[0003] When tracking the same person using multiple cameras, the computational costs skyrocket, especially in places with a large number of users, such as commercial facilities.

[0004] To address this issue, a technology has been proposed that takes into account the degree to which people occlude each other when selecting a camera, making it possible to select a camera with high object discrimination capabilities and achieving highly accurate tracking (see, for example, Patent Document 1).

[0005] Alternatively, there is a technique for extracting face images of other people who appear together with a particular person, and identifying those other people (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2022-177391 A [Patent Document 2] JP 2019-028936 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, none of these techniques have been sufficient in improving the efficiency of processing large amounts of data.

[0008] The present invention has been made in view of the above-mentioned problems. An object of the present invention is to provide a person tracking device, a person tracking system, and a person tracking method that are capable of tracking a large number of people while suppressing calculation costs. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, a person tracking device according to the present invention comprises: A feature vector update means; dimensionality reduction means, Person selection means, A feature distance calculation means; and Same person determination means A person re-recognition means comprising: said feature vector update means; The person selection means, The feature distance calculation means, and Reduction dimension determination method A dimension determination means consisting of Equipped with The feature vector update means updates a frame having a person ID, time information, and a feature vector. The system receives frame data that contains multiple people, and creates a frame group by grouping the frame data that contains the same person ID. the dimensionality reduction means updates the feature vector by retaining important elements determined by the dimensionality determination means and deleting other elements from among elements of an N-dimensional (N is a natural number equal to or greater than 2) feature vector included in the frame data; the person selection means selects a first frame group from a plurality of frame groups, searches for other frame groups by tracing frame data in chronological order from an end time of the first frame group toward the future, and selects, as a second frame group, another frame group whose start time falls within a predetermined time from the end time of the first frame group; the feature distance calculation means calculates, as a feature vector distance, a feature distance between one or more frames included in a first frame group and one or more frames included in a second frame group selected by the person selection means; the same person determination means determines a person distance based on the feature amount distance, and if the person distance is equal to or less than a threshold, determines that the people are the same person and combines the first frame group and the second frame group, and if the person distance is greater than the threshold, determines that the people are not the same person; The reduction dimension determining means determines an element of a feature vector to be an important element when a ratio of the distance of the element to the feature distance calculated by the feature distance calculating means is greater than a predetermined threshold value.

[0010] Where: The same person determination means determines, as a person distance, a predetermined r-th (r is an integer of 2 or more) smallest value among the feature distances between one or more frames included in the first frame group and one or more frames included in the second frame group, which are calculated by the feature distance calculation means. It can be configured.

[0011] Also, The predetermined threshold in the reduction dimension determination means is smaller than 1 / N. It can be configured.

[0012] In addition, the person tracking system of the present invention includes: one or more video cameras; Processing equipment and Equipped with The video camera creates and transmits the frame data to the processing device; The frame data includes a camera ID that identifies a video camera, The video camera is configured to include a video capture means, a person detection means, a person tracking means, and a feature extraction means; The moving image acquisition means generates frame data representing a moving image composed of successive frames; the person detection means detects a person, creates a bounding box that is a rectangular area surrounding the person, and adds a person ID that is an identification number given to the person and bbox information that defines the bounding box to the frame data; the person tracking means assigns the same person ID to the same person appearing in successive frames of the video and updates the frame data; the feature extraction means generates a feature vector for the bounding box created for each frame and adds the feature vector to the frame data; The processing device is the people tracking device described above.

[0013] The person tracking method of the present invention further comprises: A dimension determination process is performed in advance before the actual person tracking; The person re-recognition process that is carried out during actual person tracking and Equipped with The dimension determination step includes: receiving frame data including a plurality of frames each having a person ID, time information, and a feature vector, and creating a frame group by grouping together data of frames having the same person ID included in the frame data; selecting a first group of frames from a plurality of groups of frames, searching for other groups of frames by tracing the frame data in chronological order from the end time of the first group of frames toward the future, and selecting as a second group of frames another group of frames whose start time falls within a predetermined time from the end time of the first group of frames; calculating a feature distance between a first frame group and a second frame group selected by the person selection means as a distance between the feature vectors; determining that an element of the feature vector is an important element when the ratio of the distance of the element to the feature vector distance is greater than a predetermined threshold value; Equipped with The person re-recognition process includes: receiving frame data including a plurality of frames each having a person ID, time information, and a feature vector, and creating a frame group by grouping together data of frames having the same person ID included in the frame data; updating the feature vector by retaining the important elements and deleting the other elements from among the elements of the N-dimensional (N is a natural number equal to or greater than 2) feature vector included in the frame data; selecting a first group of frames from a plurality of groups of frames, searching for other groups of frames by tracing the frame data in chronological order from the end time of the first group of frames toward the future, and selecting as a second group of frames another group of frames whose start time falls within a predetermined time from the end time of the first group of frames; calculating a feature distance between a first frame group and a second frame group selected by the person selection means as a distance between the feature vectors; determining a person distance based on the feature distance, and if the person distance is equal to or less than a threshold, determining that the first frame group and the second frame group are the same person, and if the person distance is greater than the threshold, determining that the first frame group and the second frame group are not the same person; Equipped with.

[0014] Where: Among feature distances between one or more frames included in the first frame group and one or more frames included in the second frame group, a predetermined r-th (r is an integer equal to or greater than 2) smallest value is determined as the person distance. It can be configured.

[0015] Also, The predetermined threshold is less than 1 / N It can be configured.

[0016] Also, the dimension determination process and the person re-recognition process are performed before the execution of the person re-recognition process. A step of generating frame data representing a moving image composed of successive frames; A process of detecting a person, creating a bounding box that is a rectangular area surrounding the person, and adding a person ID that is an identification number given to the person and bbox information that defines the bounding box to the frame data; a step of assigning the same person ID to the same person appearing in successive frames of the video and updating the frame data; A process of generating a feature vector for the bounding box created for each frame and adding the feature vector to the frame data; It can be provided with: Effect of the Invention

[0017] According to the human tracking device, human tracking system, and human tracking method of the present invention, it is possible to efficiently reduce the dimensions of the feature vector, thereby keeping down the calculation cost and tracking a large number of people. [Brief description of the drawings]

[0018] [Figure 1] FIG. 2 is a schematic diagram of a video camera provided in the person tracking system. [Diagram 2] FIG. 2 is a schematic diagram of a person tracking device included in the person tracking system. [Diagram 3] FIG. 4 is a schematic diagram showing an example of frame data. [Figure 4] FIG. 1 is a schematic diagram (1) for explaining the operation of a person tracking device. [Diagram 5] FIG. 2 is a schematic diagram for explaining the operation of the person tracking device. [Figure 6] FIG. 13 is a diagram showing the results of an investigation into the effect of dimensionality reduction on feature distance. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings, but the shape, size, and positional relationship of each component are merely shown in a schematic manner to allow the present invention to be understood. In addition, the following describes a preferred configuration example of the present invention, but the numerical conditions, etc. are merely preferred examples. Therefore, the present invention is not limited to the following embodiment, and many changes or modifications can be made that can achieve the effects of the present invention without departing from the scope of the configuration of the present invention.

[0020] The person tracking system is placed in an area of ​​a certain size (hereinafter also referred to as a tracking area) and tracks the behavior of people within the tracking area. The person tracking system uses the tracking results to perform behavior analysis of people. For example, if the tracking area is a commercial facility, behavior analysis includes analysis of the movement of customers and analysis of the movement of employees.

[0021] The person tracking system will be described with reference to Figures 1 to 3. Figure 1 is a schematic diagram of a video camera provided in the person tracking system. Figure 2 is a schematic diagram of a person tracking device as a processing device provided in the person tracking system. Figure 3 is a schematic diagram showing an example of frame data.

[0022] The person tracking system includes one or more video cameras 100 and a person tracking device 200 .

[0023] The video camera 100 is configured to include a video capture means 110, a person detection means 120, a person tracking means 130, and a feature extraction means 140. The video camera 100 used here may be any suitable conventionally known edge AI camera, also known as an edge device (edge ​​terminal), that is equipped with the person detection means 120, the person tracking means 130, and the feature extraction means 140 in addition to the video capture means 110 included in a general video camera and is capable of AI (Artificial Intelligence) processing.

[0024] The video camera 100 transmits and receives information to and from the person tracking device 200 by wired or wireless communication. By using an edge AI camera as the video camera 100, data processing is performed in the video camera 100, and the processed data is sent to the person tracking device 200. As described above, the video camera configured in this manner may be any suitable conventional publicly available Since known techniques can be used, detailed explanations may be omitted in the following description.

[0025] For example, when a video camera other than an edge AI camera is used as the video camera, the video camera can be configured to include a video acquisition means 110, and the person tracking device can be configured to include a person detection means 120, a person tracking means 130, and a feature extraction means 140. In this case, all video data acquired by the video acquisition means 110 is sent to the person tracking device. Therefore, when a large number of video cameras are used, the traffic between each video camera and the person tracking device and the processing load in the person tracking device increase. As a result, processing in the person tracking system may take a long time.

[0026] In contrast, when an edge AI camera is used as a video camera, even when multiple video cameras are used, it is possible to suppress an increase in traffic between the video camera 100 and the person tracking device 200 and an increase in the processing load in the person tracking device 200. As a result, the processing time in the person tracking system can be shortened.

[0027] The video capture means 110 captures a video image made up of successive frames. Digital data representing the video image captured by the video capture means 110 is sent to the person detection means 120. The video capture means 110 can be configured in the same way as a general digital video camera.

[0028] The person detection means 120, the person tracking means 130, and the feature extraction means 140 are configured with any suitable electronic circuitry or the like that performs digital signal processing. In the following description, the digital data acquired by the video acquisition means 110 is also referred to as frame data. The frame data is updated in the subsequent digital signal processing.

[0029] The frame data acquired by the video acquisition means 110 includes, for example, a camera ID which is an identification number assigned to each video camera 100, time information of each frame, and a frame ID which is an identification number of each frame.

[0030] The person detection means 120 detects people from each frame of a moving image included in the frame data received from the moving image acquisition means 110. Furthermore, the person detection means 120 creates a bounding box (bbox) for the detected person. The bbox is a rectangular area that surrounds the person. A person ID, which is an identification number that identifies the person, is assigned to the bbox.

[0031] In the frame data, the minimum x-coordinate value xmin, the maximum x-coordinate value xmax, the minimum y-coordinate value ymin, and the maximum y-coordinate value ymax of the bbox are recorded as bbox information that defines the bbox. In addition, the person ID is recorded in the frame data. Note that the bbox information may be any information that can define the bbox, and is not limited to the example described here.

[0032] The process of detecting a person for each frame and the process of creating a bbox performed by the person detection means 120 can be performed using any suitable conventionally known technology, and therefore a description thereof will be omitted here.

[0033] The frame data updated by adding the bbox information and the person ID in the person detection means 120 is sent to the person tracking means 130.

[0034] The person tracking means 130 detects the same person I for the same person appearing in successive frames of the video. D is assigned to each person. Whether or not a person appears in consecutive frames is determined by comparing the bboxes of the consecutive frames.

[0035] The determination of whether the people in the consecutive frames are the same person can be performed using any suitable conventional technology. For example, if the difference in bbox between consecutive frames is equal to or smaller than a predetermined threshold, it can be determined that they are the same person, and if it is greater than the threshold, it can be determined that they are not the same person.

[0036] In this case, the area of ​​the bbox in the kth frame is S k , the area of ​​bbox in the k+1th frame is S k+1 Let the area of ​​the overlapping part between the bbox in the kth frame and the bbox in the k+1th frame be S com Then, S k and S k+1 are respectively, S k = ΔS k +S com and S k+1 = ΔS k+1 +S com In this case, the difference ΔS in bbox between consecutive frames is ΔS = ΔS k +ΔS k+1 is given by:

[0037] Or, ΔS = S com / (S k +S k+1 ) the area of ​​the bbox in the kth frame, S k and the area S of the bbox in the k+1th frame k+1 The area of ​​the overlapping parts S com The ratio may be set as the bbox difference ΔS.

[0038] If it is determined that the person appearing in consecutive frames is the same person, the person tracking means 130 applies the person ID assigned to the bbox of the previous frame (e.g., the kth frame) to the bbox of the subsequent frame (e.g., the k+1th frame), making the latter person ID the former person ID, and making both person IDs the same person ID.

[0039] On the other hand, if it is determined that the people appearing in the consecutive frames are not the same person, the person tracking means 130 does not change the person IDs of both of them, but leaves them as different person IDs.

[0040] The updated frame data in which the same person ID is assigned to the same person and different person IDs are assigned to different people is sent to the feature extraction means 140.

[0041] The feature extraction means 140 creates a person's feature vector for the bbox created for each frame. Any suitable conventionally known technology can be used as a technique for creating a person's feature vector from this bbox. Here, the person's feature vector is an N-dimensional (N is a natural number equal to or greater than 2) vector, for example, 512-dimensional. The dimension of the person's feature vector corresponds to the number of elements the feature vector has. The elements of the feature vector are, for example, numerical values ​​indicating the color of each cell when the bbox is divided into N cells.

[0042] The feature extraction means 140 adds a feature vector to the frame data. The frame data updated by the feature extraction means 140 with the addition of the feature vector (see FIG. 3) is sent from the video camera 100 to the person tracking device 200.

[0043] It should be noted that the frame ID does not have to be included in the frame data sent to the person tracking device 200. Moreover, the bbox information does not have to be included in the frame data.

[0044] When a commercially available edge AI camera is used as the video camera 100, the frame data is provided according to the specifications. In this case, the frame data only needs to include the above-mentioned camera ID, frame ID, time information, bbox information, person ID, and feature vector, and may also include other data.

[0045] The person tracking device 200 is, for example, a computer such as a personal computer. By executing a predetermined program, each of the functional means described below is realized.

[0046] The person tracking means 130 included in the video camera 100 can track the same person in consecutive frames. However, if the person being tracked once leaves the imaging range and is detected again, the person tracking means 130 will assign a different person ID to the person, assuming that the person is not the same person. Alternatively, if the tracking area is covered by multiple video cameras 100, even if the person being tracked leaves the imaging range of one video camera 100 and enters the imaging range of another video camera 100, the person will not be treated as the same person. For this reason, this is insufficient for analyzing the behavior of a person. Therefore, the person tracking device 200 determines whether the people to which different person IDs are assigned are the same person, and if they are the same person, integrates the data of the feature vectors and performs behavior analysis.

[0047] The person tracking device is configured to include, as functional means, a feature vector updating means 210, a dimension reduction means 220, a person selecting means 230, a feature distance calculation means 240, a same person determination means 250, a reduced dimension determination means 260, and an analysis means 330. The feature vector updating means 210, the dimension reduction means 220, the person selecting means 230, the feature distance calculation means 240, and the same person determination means 250 configure person re-recognition means 310, and the feature vector updating means 210, the person selecting means 230, the feature distance calculation means 240, and the reduced dimension determination means 260 configure dimension determination means 320.

[0048] The processing in the person re-recognition means 310 is performed when an actual person is tracked. After the processing in the person re-recognition means 310 is completed, behavior analysis is performed in the analysis means 330. The processing in the dimension determination means 320 is performed in advance, before the actual person is tracked. The processing in the person re-recognition means 310 is performed based on the processing result of the dimension determination means 320.

[0049] First, the process performed by the person re-recognition means 310 will be described.

[0050] The feature vector update means 210 creates a frame group by grouping together data of frames with the same person ID. Here, for example, if the number of frames with the same person ID is greater than or equal to the person selection threshold n th Data related to this person ID is discarded if: Person selection threshold n th can be set arbitrarily and suitably. Here, the person selection threshold n th We will explain what happens when is set to 5.

[0051] For example, the number of frames is the person selection threshold n th (Here, 5 or less) corresponds to a time of about 0.2 seconds or less in a 25 fps (frames per second) camera. In this case, it is possible that the bbox cannot be obtained accurately from the whole image of the person due to problems such as the accuracy of person detection.

[0052] Among the time information of the frames included in the frame group, the oldest time is referred to as the start time, and the newest time is referred to as the end time.

[0053] The dimension reduction means 220 updates the feature vector by leaving predetermined important elements out of the elements of the feature vector and deleting the other elements. The elements to be deleted are determined by the dimension determination means 320. The processing in the dimension determination means 320 will be described later. In the example described here, when the dimension N of the feature vector extracted by the feature extraction means 140 is 512, the dimension d of the updated feature vector will be, for example, about 300 to 350.

[0054] The processing in the person selection means 230 to the same person determination means 250 will be described with further reference to Fig. 4. Fig. 4 is a schematic diagram for explaining the operation of the person tracking device, in particular, the processing in the person selection means 230 to the same person determination means 250.

[0055] The person selection means 230 selects, as a first frame group, frame groups of the person IDs to be selected (for example, the person ID of person A) in order from the oldest start time included in the time information.

[0056] Next, the person selection means 230 chronologically traces a predetermined time from the end time of the frame group of the person ID of person A to be selected toward the future (newer time) to search for frame groups having other person IDs. Here, it is assumed that 10 minutes are traced in chronological order. If a frame group of another person ID whose start time is included is found within 10 minutes from the end time of the frame group of one person ID (for example, if the person ID of person B is found), this frame group is selected as the second frame group. Here, since a predetermined time is traced in chronological order from the end time of the frame group of the person ID of the person to be selected toward the future, people who exist at the same time are not selected.

[0057] The time to trace into the future can be arbitrarily and suitably set depending on the installation environment of the camera, the average movement of people in the tracking target facility where the camera is installed, the data processing time, etc. Here, tracing is performed for 10 minutes, and if a person is not captured by the camera for 10 minutes or more, it is assumed that the person has left the tracking target facility where the camera is installed, and the person re-recognition is terminated thereafter.

[0058] The feature distance calculation means 240 calculates the feature distance between two people (here, person A and person B) selected by the person selection means 230. i (i is an integer between 1 and N), and the feature vector of person B is b i Then, the feature distance can be calculated using the following formula (1).

[0059]

number

[0060] Here, f i Let f be a flag indicating whether the i-th element is an important element or not. i takes the value 0 or 1, and is 1 if the i-th element of the feature vector is important, and 0 if it is not. i is determined by the dimensionality reduction means 320 described below.

[0061] This feature distance calculation is performed for all combinations of the feature vectors of person A and person B. For example, if the number of feature vectors of person A (number of frames) is n A , the number of feature vectors of person B is n B When n A ×n B n calculations are performed, where A , n B is n as mentioned above. th is a larger integer.

[0062] The same person determination means 250 determines that the person is the same person when the person distance is equal to or smaller than the threshold value, and determines that the person is not the same person when the person distance is greater than the threshold value.

[0063] The same person determination means 250 determines the person distance between the two persons A and B selected by the person selection means 230. The person distance is determined based on the feature amount distance calculated by the feature amount distance calculation means 240.

[0064] In general, if the person is the same, the feature distance will be small, and if the person is not the same, the feature distance will be large. The distance becomes larger. However, n A ×n BAmong the feature distances, there may be irregular values ​​that are far from the average distribution. For this reason, if the smallest value is used as the person distance, there is a possibility that two people who are not the same person may be mistakenly determined to be the same person. Therefore, in this example, the smallest value of the feature distance is not used, and n A ×n B Among the feature distances, the third smallest value is set as the person distance. In this example, the third smallest value among the feature distances is set as the person distance, but this is not limited to this and can be set arbitrarily and suitably. For example, A ×n B A value other than the third smallest feature distance may be used as the person distance, or A ×n B The average value of the feature amount distances may be used as the person distance. However, when using the average value, it is preferable to take into consideration values ​​that are irregular and far from the average distribution.

[0065] The same person determination means 250 determines that person A and person B are the same person when the person distance is equal to or smaller than the threshold value, and determines that they are not the same person when the person distance is greater than the threshold value.

[0066] When the same person determination means 250 determines that person A and person B are the same person, it combines the frame group of person A with the frame group of person B to update the frame group. For example, it rewrites the person ID of person B to the person ID of person A, rewrites the end time of person A to the end time of person B, and ends the processing for person B (see FIG. 4). Thereafter, the person selection means 230, feature distance calculation means 240, and same person determination means 250 are performed with the updated frame group of person A as the first frame group.

[0067] This process is repeated until no frame group of another person ID whose start time falls within a predetermined time (here, 10 minutes) from the end time of the first frame group is found.

[0068] If there is no other person to be selected within 10 minutes from the end time of the first frame group, the processing for person A ends.

[0069] Thereafter, the person selection means 230 selects a frame group of the person ID whose start time included in the time information is the next oldest, and repeats the processing by the person selection means 230, feature distance calculation means 240, and same person determination means 250. When the processing by the person selection means 230, feature distance calculation means 240, and same person determination means 250 is completed for frame groups of all person IDs, the processing by the person re-recognition means 310 ends.

[0070] With reference to Fig. 5, as an example, processing for seven people with different person IDs will be described. Fig. 5, like Fig. 4, is a schematic diagram for explaining the operation of the person tracking device, in particular the processing by the person selection means 230 to the same person determination means 250. Fig. 5(A) shows the state before the processing by the person selection means 230 to the same person determination means 250 is performed, and Fig. 5(B) shows the state after the processing by the person selection means 230 to the same person determination means 250 is performed.

[0071] Before the processing is performed by the person selection means 230 through the same person determination means 250, a frame group of seven people, person A through person G, shown in FIG. 5(A) is obtained.

[0072] The frame group of person A, who is the person with the oldest start time, is selected as the first frame group. Next, the frame group of person B, who is the person whose start time falls within a predetermined time from the end time of the frame group of person A, is selected as the second frame group. The frame group of person A and the frame group of person B are subjected to a process of determining whether they are the same person, and if they are determined to be the same person, the frame group of person A and the frame group of person B are combined.

[0073] The combined frame group of person A is newly selected as a first frame group, and the frame group of person C is selected as a person whose start time is within a predetermined time from the end time of the frame group of person A. The frame group of person A and the frame group of person C are selected as the second frame group. If the frame group of person A and the frame group of person C are not determined to be the same person as a result of performing the same person determination process, the frame group of person A and the frame group of person C are not combined. If a frame group of another person is not found whose start time falls within a predetermined time from the end time of the frame group of person A, the processing for person A ends.

[0074] Next, the frame group of person D is selected as the first frame group as the person whose start time is the second oldest after person A. Next, the frame group of person G is selected as the second frame group as the person whose start time falls within a predetermined time from the end time of the frame group of person D. If the frame groups of person D and person G are not determined to be the same as a result of performing the same person determination process on the frame groups of person D and person G, the frame groups of person D and person G are not combined. Next, the frame group of person E is selected as the second frame group as the person whose start time falls within a predetermined time from the end time of the frame group of person D. The frame groups of person D and person E are processed to determine whether they are the same person, and if they are determined to be the same, the frame groups of person D and person E are combined.

[0075] This combined frame group of person D is then newly selected as a first frame group, and the frame group of person C is selected as a second frame group as a person whose start time falls within a predetermined time from the end time of the frame group of person D. If the frame groups of person D and person C are not determined to be the same person as a result of performing the same person determination process on the frame groups of person D and person C, the frame groups of person D and person C are not combined. If a frame group of another person is not found as a person whose start time falls within a predetermined time from the end time of the frame group of person D, the processing for person D ends.

[0076] Next, the frame group of person F is selected as the first frame group as the person whose start time is the second oldest after person D. Next, the frame group of person G is selected as the second frame group as the person whose start time falls within a predetermined time from the end time of the frame group of person F. If the frame groups of person F and person G are not determined to be the same person as a result of performing the same person determination process on the frame groups of person F and person G, the frame groups of person F and person G are not combined. If a frame group of another person is not found as a person whose start time falls within a predetermined time from the end time of the frame group of person F, the processing for person F ends.

[0077] Next, the frame group of person G is selected as the first frame group as the person whose start time is the second oldest after person F. Next, the frame group of person C is selected as the second frame group as the person whose start time falls within a predetermined time from the end time of the frame group of person G. If the frame groups of person G and person C are not determined to be the same person as a result of performing the same person determination process on the frame groups of person G and person C, the frame groups of person G and person C are not combined. If no frame group of another person is found as a person whose start time falls within a predetermined time from the end time of the frame group of person G, the processing for person F ends.

[0078] Next, the frame group of person C is selected as the first frame group because the person whose start time is the second oldest after person G. If a frame group of another person is not found whose start time falls within a predetermined time from the end time of the frame group of person C, the processing for person C is terminated.

[0079] As a result, the state shown in FIG. 5(B) is obtained.

[0080] After the above-mentioned processing by the person re-recognition means 310 is completed, the analysis means 330 performs the analysis processing. The analysis processing by the analysis means is performed according to the design, and is realized by any suitable conventionally known technology. For example, when tracking the behavior of a person in a certain facility, the analysis means 330 performs the analysis processing by detecting whether the person is in the facility or not. The system obtains information such as the route taken within the area and the amount of time spent at each location.

[0081] The analysis results from the analysis means 330 are sent to and output from an output means configured with a display means such as a liquid crystal display (LCD) or a printing means such as a printer, or may be configured to send the analysis results to another terminal via a communication line such as the Internet.

[0082] The dimension determination means 320 obtains the elements of the feature vector used in the calculation of the feature distance and the number of dimensions d corresponding to the number of the elements. The processing in the dimension determination means 320 is performed in advance before the actual tracking. Note that the processing may be performed appropriately during the actual tracking, or data obtained during the actual tracking may be used.

[0083] In the dimension determination means 320, similarly to the person re-recognition means 310, the feature vector update means 210 groups together data having the same person ID to create a frame.

[0084] The dimension determination means 320 does not perform the processing in the dimension reduction means 220. In other words, elements of the feature vector are not deleted.

[0085] As in the person re-recognition means 310, the person selection means 230 selects two people (e.g., person A and person B), and the feature distance calculation means 240 calculates the feature distance between the two people using the above formula (1). At this time, no elements of the feature vector are deleted, so f i are all 1.

[0086] Next, the reduction dimension determination means 260 performs a determination for each element of N dimensions, 512 dimensions in this example, using the following formula (2).

[0087]

number

[0088] As a result of the determination by the reduction dimension determination means 260, the dimension i that satisfies the above formula (2) is determined to be an important dimension and is not targeted for reduction. i and set all elements to 0. If the above formula (2) is satisfied for a certain i, then g i Let's set it to 1.

[0089] This determination is performed for all combinations of the feature vectors of person A and person B. As a result, g i The elements with 0 are subject to deletion, and g i Elements with g = 1 are important elements and are not reduced. i f i The reduction dimension is determined by substituting

[0090] Here, the threshold value in the above formula (2) can be set arbitrarily and suitably.

[0091] For example, consider a case where the threshold is set to 1 / 100. This corresponds to considering the difference in a feature of a certain dimension to be important when it is 1% or more of the feature distance. In this case, the number of dimensions after deletion in a specific feature extraction model will be 312.

[0092] In addition, if the difference in feature values ​​in many dimensions is close to 1%, for example 0.99%, the dimensions with values ​​close to 1% are not judged to be important dimensions. Taking all dimensions into account, in the case of 512 dimensions as in this example, we can expect the proportion of feature distance to be significant.

[0093] Next, consider the case where the threshold is set to 1 / 100 × 1 / 512. This corresponds to selecting important dimensions such that the sum of all the values ​​of unimportant dimensions is less than 1% of the feature distance. In this case, the number of dimensions after removal is 326.

[0094] Next, consider the case where the threshold is set to 1 / 512, which is a value between 1 / 100 and 1 / 100 x 1 / 512. This corresponds to considering the difference in a feature of a certain dimension to be important when it is equal to or greater than the average value of the feature distance per dimension. In this case, the number of dimensions after deletion will be 322.

[0095] Thus, no significant difference was observed in the number of dimensions when the threshold values ​​were 1 / 100, 1 / 512, and 1 / 100×1 / 512.

[0096] Therefore, the threshold can be set to a predetermined ratio x 1 / number of dimensions. If the threshold is set in this way, the sum of all values ​​of unimportant dimensions will be less than the predetermined ratio of the feature distance. For example, consider a case where the predetermined ratio is 1%, and the number of dimensions is 512, but the number of dimensions after reduction is 326.

[0097] In this case, even when calculation is performed using the reduced dimensions (326 dimensions in this example), the value is 99% or more of the value calculated using all the dimensions before the deletion (512 dimensions in this example), so the effect of deleting the dimensions on the feature distance is small. Figure 6 shows the results of investigating the effect of reducing the dimensions on the feature distance.

[0098] In Figures 6(A) to (C), the horizontal axis shows the feature distance, and the vertical axis shows the count number. Figure 6(A) shows the measurement results of the feature distance between the same person and different people before dimension reduction. Figure 6(B) shows the measurement results of the feature distance between the same person and different people after dimension reduction. Figure 6(C) shows the measurement results of the feature distance between the same person and different people before and after dimension reduction, superimposed.

[0099] As can be seen by comparing Figures 6(A) and (B), there is almost no difference in the feature distance calculation results before and after dimension reduction. Figure 6(C) shows the results before and after dimension reduction, but the difference is so small that it is not noticeable in the figure. In fact, the average difference in distance between the same person before and after dimension reduction was 0.455, and the average difference in distance between different people was 0.004.

[0100] On the other hand, the amount of calculation in the feature distance calculation means 240 can be reduced to about 63.7% (=326 / 512).

[0101] In this way, according to the distance tracking device of the present invention, the time required for calculating the feature distance can be reduced without affecting the result of the same person determination.

[0102] The same person determination process in the same person determination means 250 is performed without using a camera ID. That is, even if multiple video cameras are used and multiple frame data are sent to the person tracking device, the same person determination process is possible.

[0103] In addition, even if frame data of the same camera ID is sent to the person tracking device in separate time intervals, for example, every hour, it is possible to perform processing to determine whether the data is the same person. Therefore, since data can be processed at any time while it is being acquired, real-time processing is also possible in some cases. [Explanation of symbols]

[0104] 100: Video camera 110:Video acquisition means 120: Person detection means 130: Person tracking methods 140: Feature extraction means 200: Person tracking device 210: Feature vector update means 220: Dimensionality reduction means 230: Person selection method 240: Feature distance calculation means 250: Same person determination means 260: Reduction dimension determination means 310: Person re-recognition means 320: Dimension determining means 330: Analysis means

Claims

1. A feature vector update means; dimensionality reduction means, Person selection means, A feature distance calculation means; and Same person determination means A person re-recognition means comprising: said feature vector update means; The person selection means, The feature distance calculation means, and Reduction dimension determination method A dimension determination means consisting of Equipped with the feature vector update means receives frame data including a plurality of frames each having a person ID, time information, and a feature vector, and creates a frame group by grouping together data of frames having the same person ID included in the frame data; the dimensionality reduction means updates the feature vector by retaining important elements determined by the dimensionality determination means and deleting other elements from among elements of an N-dimensional (N is a natural number equal to or greater than 2) feature vector included in the frame data; the person selection means selects a first frame group from a plurality of frame groups, searches for other frame groups by tracing frame data in chronological order from an end time of the first frame group toward the future, and selects, as a second frame group, another frame group whose start time falls within a predetermined time from the end time of the first frame group; the feature distance calculation means calculates, as a feature vector distance, a feature distance between one or more frames included in a first frame group and one or more frames included in a second frame group selected by the person selection means; the same person determination means determines a person distance based on the feature distance calculated by the feature distance calculation means for the feature vector updated by the dimension reduction means, and if the person distance is equal to or less than a threshold, determines that the people are the same person and combines the first frame group and the second frame group, and if the person distance is greater than the threshold, determines that the people are not the same person; The reduction dimension determination means determines an element as an important element when a ratio of a distance of an element of a feature vector to the feature distance calculated by the feature distance calculation means is greater than a predetermined threshold value. People tracking device.

2. The same person determination means determines, as a person distance, a predetermined r-th (r is an integer of 2 or more) smallest value among the feature distances between one or more frames included in the first frame group and one or more frames included in the second frame group, which are calculated by the feature distance calculation means. The person tracking device according to claim 1 .

3. The predetermined threshold in the reduction dimension determination means is smaller than 1 / N. The person tracking device according to claim 1 .

4. one or more video cameras; Processing equipment and Equipped with The video camera creates and transmits the frame data to the processing device; The frame data includes a camera ID that identifies a video camera, The video camera includes a video capture means, a person detection means, a person tracking means, and a feature extraction means. The present invention is configured with a step. The moving image acquisition means generates frame data representing a moving image composed of successive frames; the person detection means detects a person, creates a bounding box that is a rectangular area surrounding the person, and adds a person ID that is an identification number given to the person and bbox information that defines the bounding box to the frame data; the person tracking means assigns the same person ID to the same person appearing in successive frames of the video image and updates the frame data; the feature extraction means generates a feature vector for the bounding box created for each frame and adds the feature vector to the frame data; The processing device is a person tracking device according to claims 1 to 3. People tracking system.

5. This is done using a person tracking device. A dimension determination process is performed in advance before the actual person tracking; The person re-recognition process that is carried out during actual person tracking and Equipped with The dimension determination step includes: a feature vector updating process of receiving frame data including a plurality of frames each having a person ID, time information, and a feature vector, and creating a frame group by grouping data of frames having the same person ID included in the frame data; a person selection process for selecting a first frame group from a plurality of frame groups, searching for other frame groups by tracing the frame data in chronological order from the end time of the first frame group toward the future, and selecting, as a second frame group, another frame group whose start time falls within a predetermined time from the end time of the first frame group; a feature distance calculation step of calculating a feature distance between a first frame group and a second frame group selected in the person selection step as a distance between the feature vectors; a reduction dimension determination step of determining an element of a feature vector as an important element when the ratio of the distance of the element to the feature distance is greater than a predetermined threshold value; Equipped with The person re-recognition process includes: a feature vector updating process of receiving frame data including a plurality of frames each having a person ID, time information, and a feature vector, and creating a frame group by grouping data of frames having the same person ID included in the frame data; a dimension reduction process for updating the feature vector by retaining the important elements and deleting the other elements from among the elements of the N-dimensional (N is a natural number equal to or greater than 2) feature vector included in the frame data; a person selection process for selecting a first frame group from a plurality of frame groups, searching for other frame groups by tracing the frame data in chronological order from the end time of the first frame group toward the future, and selecting, as a second frame group, another frame group whose start time falls within a predetermined time from the end time of the first frame group; a feature distance calculation step of calculating a feature distance between a first frame group and a second frame group selected in the person selection step as a distance between the feature vectors updated in the dimension reduction step; a same person determination process for determining a person distance based on the feature distance calculated in the feature distance calculation process for the feature vector updated in the dimension reduction process, determining that the person is the same person if the person distance is equal to or smaller than a threshold, and combining the first frame group and the second frame group, and determining that the person is not the same person if the person distance is greater than the threshold; Equipped Person tracking methods.

6. In determining the same person, among the feature distances between one or more frames included in the first frame group and one or more frames included in the second frame group, The rth (r is an integer equal to or greater than 2) smallest value determined in advance is determined as the person distance. The person tracking method according to claim 5 .

7. The predetermined threshold is less than 1 / N. The person tracking method according to claim 5 .

8. The dimension determination process and the person re-recognition process are performed before the process is performed. A video capture process for generating frame data representing a video image composed of successive frames; a person detection process for detecting a person, creating a bounding box that is a rectangular area surrounding the person, and adding a person ID that is an identification number given to the person and bbox information that defines the bounding box to the frame data; a person tracking process for assigning the same person ID to the same person appearing in consecutive frames of the video and updating the frame data; A feature extraction process generates a feature vector for the bounding box created for each frame and adds the feature vector to the frame data. The person tracking method according to any one of claims 5 to 7, comprising:

Images