Human tracking apparatus, human tracking system, and human tracking method

Abstract

To track many people while suppressing computational cost.SOLUTION: A human tracking apparatus is configured to: select a first frame group out of a plurality of frame groups; search other frame groups by tracing frame data in chronological order from the end time of the first frame group to the future; select, as a second frame group, another frame group of which the start time is included within a predetermined time from the end time of the first frame group; calculate feature distance between the first frame group and the second frame group; determine a person distance based on the feature distance; and determine that the individuals are the same person when the person distance is equal to or less than a threshold or determine that the individuals are not the same person when the person distance is greater than the threshold. When a ratio of distance of feature vector elements is larger than a predetermined threshold, the elements are determined to be important elements.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a person tracking device, a person tracking system, and a person tracking method that can be used, for example, when performing person tracking using a plurality of cameras.

Background Art

[0002] In commercial facilities, attracting customers through events and advertisements is very important. For this reason, there is a demand for efficiently analyzing the movement analysis of visitors and analyzing the effects of each measure.

[0003] When tracking the same person using a plurality of cameras, especially when there are many users such as in commercial facilities, the calculation cost will soar.

[0004] In response to this problem, a technique has been proposed in which by selecting a camera in consideration of the degree of occlusion between people, it is possible to select a camera with high object discriminability and achieve highly accurate tracking (see, for example, Patent Document 1).

[0005] Alternatively, there is also a technique of cutting out the face images of other people who appear with a specific person and identifying the other people (see, for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, none of these techniques are sufficient in terms of efficiently processing a large amount of data.

[0008] The present invention has been made in view of the above-described problems. An object of the present invention is to provide a person tracking device, a person tracking system, and a person tracking method that can track a large number of people while suppressing calculation costs.

Means for Solving the Problems

[0009] In order to solve the above-described problems, a person tracking device according to the present invention includes feature vector update means, dimensionality reduction means, person selection means, feature quantity distance calculation means, and same person determination means constituting person re-recognition means, and the feature vector update means, the person selection means, the feature quantity distance calculation means, and dimensionality reduction determination means constituting dimensionality determination means and includes The feature vector update means receives frame data including a plurality of frames having a person ID, time information, and a feature vector, groups data of frames having the same person ID included in the frame data, and creates a frame group. The dimensionality reduction means leaves important elements determined by the dimensionality determination means among the elements of an N-dimensional (N is a natural number of 2 or more) feature vector included in the frame data, deletes other elements, and updates the feature vector. The person selection means selects a first frame group from a plurality of frame groups, traces frame data in time series order from the end time of the first frame group toward the future to search for other frame groups, and selects other frame groups whose start times are included within a predetermined time from the end time of the first frame group as a second frame group. ​The feature amount distance calculation means calculates the feature amount distance between each of one or more frames included in the first frame group and one or more frames included in the second frame group, which are selected by the person selection means, as the distance between feature vectors. The same person determination means determines the person distance based on the feature amount distance. When the person distance is less than or equal to the threshold value, it determines that they are the same person and combines the first frame group and the second frame group. When the person distance is greater than the threshold value, it determines that they are not the same person. When the ratio of the distance of a certain element of the feature vector to the feature amount distance calculated by the feature amount distance calculation means is greater than a predetermined threshold value, the reduction dimension determination means regards the element as an important element.

[0010] Here, The same person determination means determines, as the person distance, the value that is the r-th (r is an integer of 2 or more) smallest among the feature amount distances between each of one or more frames included in the first frame group and one or more frames included in the second frame group, which are calculated by the feature amount distance calculation means. It can be configured.

[0011] Also, The predetermined threshold value in the reduction dimension determination means is smaller than 1 / N. It can be configured.

[0012] Also, the person tracking system of this invention includes one or more video cameras, a processing device and is provided with The video camera creates the frame data and sends it to the processing device. The frames included in the frame data have a camera ID that identifies the video camera. The video camera is configured to include a video acquisition means, a person detection means, a person tracking means, and a feature extraction means. The video acquisition means generates frame data indicating a moving image composed of consecutive frames. The person detection means detects a person, creates a bounding box which is a rectangular area surrounding the person, and adds the person ID which is an identification number given to the person and bbox information that determines the bounding box to the frame data. The person tracking means assigns the same person ID to the same person appearing in consecutive frames of a moving image and updates the frame data. The feature extraction means generates a feature vector for the bounding box created for each frame and adds the feature vector to the frame data. The processing device is the above-described person tracking device.

[0013] Also, the person tracking method of this invention includes a dimension determination process that is performed in advance before actual person tracking, and a person re-recognition process that is performed during actual person tracking and the dimension determination process receives frame data including a plurality of frames having a person ID, time information, and a feature vector, groups the data of the frames having the same person ID included in the frame data, and creates a frame group; selects a first frame group from a plurality of frame groups, traces the frame data in chronological order from the end time of the first frame group towards the future to search for other frame groups, and selects another frame group whose start time is included within a predetermined time from the end time of the first frame group as a second frame group; calculates the feature quantity distance between the first frame group and the second frame group selected by the person selection means as the distance between the feature vectors; when the ratio of the distance of a certain element of the feature vector to the feature quantity distance is greater than a predetermined threshold, designates the element as an important element and the person re-recognition process Receiving frame data including a plurality of frames having a person ID, time information, and a feature vector, and collecting data of frames with the same person ID included in the frame data to create a frame group; In the feature vector of N dimensions (N is a natural number of 2 or more) included in the frame data, leaving the important elements and deleting the other elements to update the feature vector; Selecting a first frame group from a plurality of frame groups, searching for other frame groups in chronological order of frame data from the end time of the first frame group towards the future, and selecting, as a second frame group, another frame group whose start time is included within a predetermined time from the end time of the first frame group; Calculating, as the distance between the feature vectors, the feature amount distance between the first frame group and the second frame group selected by the person selection means; Determining a person distance based on the feature amount distance, and if the person distance is less than or equal to a threshold value, determining that they are the same person and combining the first frame group and the second frame group, and if the person distance is greater than the threshold value, determining that they are not the same person; comprising.

[0014] Here, Among the feature amount distances between each of the one or more frames included in the first frame group and the one or more frames included in the second frame group, determining, as the person distance, the value that is the r-th (r is an integer of 2 or more) smallest value determined in advance; It can be configured.

[0015] Also, The predetermined threshold value is smaller than 1 / N It can be configured.

[0016] Also, before executing the dimension determination process and the person re - recognition process, A process of generating frame data showing a moving image composed of consecutive frames; The process of detecting a person, creating a bounding box which is a rectangular area surrounding the person, and adding to the frame data the person ID which is an identification number given to the person and the bbox information that determines the bounding box, the process of updating the frame data by assigning the same person ID to the same person appearing in consecutive frames of a moving image, for the bounding box created for each frame, the process of generating a feature vector and adding the feature vector to the frame data can be provided with.

Advantages of the Invention

[0017] According to the person tracking device, person tracking system, and person tracking method of this invention, by efficiently reducing the dimension of the feature vector, while suppressing the calculation cost, it becomes possible to track a large number of people.

Brief Description of the Drawings

[0018]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, regarding the shape, size, and arrangement relationship of each component, only a schematic illustration is provided to the extent that the present invention can be understood. Further, hereinafter, preferred configuration examples of the present invention will be described, but numerical conditions and the like are merely preferred examples. Therefore, the present invention is not limited to the following embodiments, and many changes or modifications can be made without departing from the scope of the configuration of the present invention and still achieving the effects of the present invention.

[0020] The person tracking system is arranged, for example, in an area of a predetermined width (hereinafter also referred to as a tracking area), and tracks the actions of people within the tracking area. The person tracking system uses this tracking result to perform action analysis of people. For example, when the tracking area is a commercial facility, as action analysis, a roaming analysis of customers and a traffic flow analysis of employees are performed.

[0021] With reference to FIGS. 1 to 3, the person tracking system will be described. FIG. 1 is a schematic diagram of a video camera included in the person tracking system. Further, FIG. 2 is a schematic diagram of a person tracking device as a processing device included in the person tracking system. FIG. 3 is a schematic diagram showing an example of frame data.

[0022] The person tracking system is configured to include one or two or more video cameras 100 and a person tracking device 200.

[0023] The video camera 100 is configured to include a video acquisition means 110, a person detection means 120, a person tracking means 130, and a feature extraction means 140. As the video camera 100 used here, in addition to the video acquisition means 110 included in a general video camera, a person detection means 120, a person tracking means 130, a feature extraction means 140, etc. are provided, and any suitable conventionally known edge AI camera, also known as an edge device (edge terminal) capable of artificial intelligence (AI: Artificial Intelligence) processing, can be used.

[0024] The video camera 100 transmits and receives information to and from the person tracking device 200 through wired or wireless communication. By using an edge AI camera as the video camera 100, data processing is performed in the video camera 100, and the processed data is sent to the person tracking device 200. As described above, since a video camera configured in this way can use any suitable conventionally publicly known one, detailed description may be omitted in the following explanation.

[0025] For example, when using a video camera that is not an edge AI camera as the video camera, the video camera can be configured to include a video acquisition means 110, and the person tracking device can be configured to include a person detection means 120, a person tracking means 130, and a feature extraction means 140. In this case, all the moving image data acquired by the video acquisition means 110 is sent to the person tracking device. Therefore, especially when using a large number of video cameras, the traffic between each video camera and the person tracking device and the processing load on the person tracking device increase. As a result, the processing in the person tracking system may take time.

[0026] On the other hand, when using an edge AI camera as the video camera, even when using a large number of video cameras, it is possible to suppress the increase in traffic between the video camera 100 and the person tracking device 200 and the processing load on the person tracking device 200. As a result, the processing time in the person tracking system can be shortened.

[0027] The video acquisition means 110 acquires a moving image composed of consecutive frames. The digital data indicating the moving image acquired by the video acquisition means 110 is sent to the person detection means 120. The video acquisition means 110 can be configured in the same way as a general digital video camera.

[0028] The person detection means 120, the person tracking means 130, and the feature extraction means 140 are constituted by any suitable electronic circuit or the like that performs digital signal processing. In the following description, the digital data acquired by the moving image acquisition means 110 is also referred to as frame data. The frame data is updated in the subsequent digital signal processing.

[0029] The frame data acquired by the moving image acquisition means 110 includes, for example, a camera ID which is an identification number assigned to each video camera 100, time information of each frame, and a frame ID which is an identification number of each frame.

[0030] The person detection means 120 detects a person for each frame of the moving image included in the frame data received from the moving image acquisition means 110. Further, the person detection means 120 creates a bounding box (bbox) for the detected person. The bbox is a rectangular area surrounding the person. A person ID, which is an identification number for specifying the person, is assigned to the bbox.

[0031] In the frame data, the minimum value xmin of the x coordinate of the bbox, the maximum value xmax of the x coordinate, the minimum value ymin of the y coordinate, and the maximum value ymax of the y coordinate are recorded as bbox information for determining the bbox. Also, the person ID is recorded in the frame data. Note that the bbox information may be any information that can determine the bbox and is not limited to the example described here.

[0032] The process of detecting a person for each frame and the process of creating a bbox performed by the person detection means 120 can be carried out using any suitable conventionally known technique, so the description is omitted here.

[0033] In the person detection means 120, the frame data updated with the addition of the bbox information and the person ID is sent to the person tracking means 130.

[0034] The person tracking means 130 assigns the same person I to the same person appearing in consecutive frames of the moving image Assign D. The determination of whether the person shown in consecutive frames is the same person is made by comparing the bounding boxes (bbox) of consecutive frames.

[0035] The determination of whether the person shown in this consecutive frame is the same person can be made using any suitable conventionally known technique. For example, if the difference in bbox between consecutive frames is less than or equal to a predetermined threshold, it can be determined that they are the same person, and if it is greater than the threshold, it can be determined that they are not the same person.

[0036] In this case, let the area of the bbox in the k-th frame be S k , the area of the bbox in the (k + 1)-th frame be S k+1 , and the area of the overlapping part between the bbox in the k-th frame and the bbox in the (k + 1)-th frame be S com . When this is the case, S k and S k+1 can be expressed as S k = ΔS k + S com and S k+1 = ΔS k+1 + S com respectively. At this time, the difference in bbox between consecutive frames, ΔS, is given by ΔS = ΔS k + ΔS k+1 .

[0037] Alternatively, ΔS = S com / (S k + S k+1 ) can be used. That is, the ratio of the area of the overlapping part S k to the sum of the area of the bbox in the k-th frame S k+1 and the area of the bbox in the (k + 1)-th frame S com can also be used as the difference in bbox, ΔS.

[0038] When it is determined that the person shown in consecutive frames is the same person, the person tracking means 130 applies the person ID assigned to the bbox of the previous frame (for example, the k-th frame) to the bbox of the subsequent frame (for example, the (k + 1)-th frame), sets the latter person ID to the former person ID, and makes the person IDs of both the same person ID.

[0039] On the other hand, when it is determined that the person shown in consecutive frames is not the same person, the person tracking means 130 leaves the person IDs of both unchanged as different person IDs without changing them.

[0040] The frame data updated with the same person ID assigned to the same person and different person IDs assigned to different persons is sent to the feature extraction means 140.

[0041] The feature extraction means 140 creates a feature vector of a person for each bbox created for each frame. As a technique for creating a feature vector of a person from this bbox, any suitable conventionally known technique can be used. Here, the feature vector of a person is an N-dimensional (N is a natural number of 2 or more), for example, a 512-dimensional vector. The dimension of the feature vector of a person corresponds to the number of elements the feature vector has. The elements of the feature vector are, for example, numerical values indicating the color of each cell when the bbox is divided into N cells.

[0042] The feature extraction means 140 adds a feature vector to the frame data. In the feature extraction means 140, the frame data (see FIG. 3) updated with the addition of the feature vector is sent from the video camera 100 to the person tracking device 200.

[0043] Note that the frame data sent to the person tracking device 200 may not include the frame ID. Also, the bbox information may not be included in the frame data.

[0044] When using a commercially available edge AI camera as the video camera 100, the frame data is provided according to the specifications. At this time, the frame data only needs to include the above camera ID, frame ID, time information, bbox information, person ID, and feature vector, and may also include data other than these.

[0045] The person tracking device 200 is composed of an electronic computer such as a personal computer, for example. By executing a predetermined program, each functional means described later is realized.

[0046] In the person tracking means 130 provided in the video camera 100, it is possible to track the same person in consecutive frames. However, when the person being tracked once goes out of the imaging range and is detected again, the person tracking means 130 will assign different person IDs assuming they are not the same person. Or, when covering the tracking area with a plurality of video cameras 100, when the person being tracked goes out of the imaging range of one video camera 100 and enters the imaging range of another video camera 100, it is not treated as the same person. Therefore, it is insufficient for analyzing human behavior. Thus, the person tracking device 200 determines whether persons assigned different person IDs are the same person, and if they are the same person, it integrates the feature vector data and performs behavior analysis.

[0047] The person tracking device is configured to include, as functional means, a feature vector update means 210, a dimension reduction means 220, a person selection means 230, a feature quantity distance calculation means 240, a same person determination means 250, a reduced dimension determination means 260, and an analysis means 330. The feature vector update means 210, the dimension reduction means 220, the person selection means 230, the feature quantity distance calculation means 240, and the same person determination means 250 constitute the person re - recognition means 310, and the feature vector update means 210, the person selection means 230, the feature quantity distance calculation means 240, and the reduced dimension determination means 260 constitute the dimension determination means 320.

[0048] The processing in the person recognition means 310 is performed during actual person tracking. After the processing in the person recognition means 310 is completed, action analysis is performed in the analysis means 330. The processing in the dimension determination means 320 is performed in advance before actual person tracking. Based on the processing result of this dimension determination means 320, the processing in the person recognition means 310 is performed.

[0049] First, the processing in the person recognition means 310 will be described.

[0050] The feature vector update means 210 groups the data of frames with the same person ID to create a frame group. Here, for example, when the number of frames having the same person ID is the person selection threshold n th In the following cases, the data related to this person ID is discarded. The person selection threshold n th Can be set arbitrarily as appropriate. Here, the person selection threshold n th Is described as being 5.

[0051] For example, when the number of frames is less than or equal to the person selection threshold n th (Here, 5), in a 25fps (frame per second) camera, it corresponds to a time of about 0.2 seconds or less. In this case, it is considered that the bbox cannot be accurately obtained from the overall image of the person due to problems such as the accuracy of person detection.

[0052] Among the time information of the frames included in the frame group, the oldest time is referred to as the start time, and the newest time is referred to as the end time.

[0053] The dimension reduction means 220 leaves the predetermined important elements among the elements of the feature vector and deletes the other elements to update the feature vector. The determination of the elements to be deleted is performed by the dimension determination means 320. The processing in the dimension determination means 320 will be described later. In the example described here, when the dimension N of the feature vector extracted by the feature extraction means 140 is 512, the dimension d of the updated feature vector is, for example, about 300 to 350.

[0054] Referring further to FIG. 4, the processing in the person selection means 230 to the same person determination means 250 will be described. FIG. 4 is a schematic diagram for explaining the operation of the person tracking device, particularly the processing in the person selection means 230 to the same person determination means 250.

[0055] The person selection means 230 selects, in order from the oldest start time included in the time information, the frame group of the person ID to be selected (for example, the person ID of person A) as the first frame group.

[0056] Next, the person selection means 230 traces in chronological order for a predetermined time from the end time of the frame group of the person ID of person A to be selected, toward the future (newer time), and searches for a frame group having another person ID. Here, it is assumed to trace 10 minutes in chronological order. If a frame group of another person ID whose start time is included is found within 10 minutes from the end time of the frame group of one person ID (for example, if the person ID of person B is found), this frame group is selected as the second frame group. Here, since it traces in chronological order for a predetermined time from the end time of the frame group of the person ID of the person to be selected, toward the future, the persons existing at the same time will not be selected as the object.

[0057] Note that the time for tracing toward the future can be arbitrarily and preferably set according to the installation environment of the camera, the average movement of people in the tracking target facility where the camera is installed, the data processing time, etc. Here, it is assumed to trace for 10 minutes, and for a person who has not appeared in the camera for 10 minutes or more, it is assumed that the person has left the tracking target facility where the camera is installed, and subsequent person recognition is terminated.

[0058] The feature amount distance calculation means 240 calculates the feature amount distance between the two persons (here, person A and person B) selected as the selection target by the person selection means 230. Let the elements of the feature vector of person A be a i (where i is an integer from 1 to N), and the feature vector of person B be b i Then, the feature amount distance can be calculated by the following formula (1).

[0059] [Number]

[0060] Here, let f i be a flag indicating whether the i-th element is an important element. f i takes either the value 0 or 1, and is 1 when the i-th element of the feature vector is important, and 0 otherwise. The flag f i is determined by the dimension reduction means 320 described later.

[0061] This calculation of the feature quantity distance is performed for all combinations of the feature vector of person A and the feature vector of person B. For example, if the number of feature vectors of person A (the number of frames) is n A and the number of feature vectors of person B is n B when there are n A ×n B calculations are performed. Here, n A and n B are integers greater than n th as described above.

[0062] The same person determination means 250 determines that they are the same person when the person distance is less than or equal to the threshold value, and determines that they are not the same person when the person distance is greater than the threshold value.

[0063] The same person determination means 250 determines the person distance between the two persons A and B selected as the selection targets by the person selection means 230. The person distance is determined based on the feature quantity distance calculated by the feature quantity distance calculation means 240.

[0064] Generally, in the case of the same person, the feature quantity distance becomes small, and in the case of different persons, the feature quantity distance becomes large. However, n A ×n BAmong the feature distance values, there may be some that deviate significantly from the regular average distribution. Therefore, if the smallest value is used as the person distance, there is a possibility of misjudging two non-identical persons as the same person. Thus, in this example, instead of using the minimum value of the feature distance, n A ×n B Of the n A ×n B feature distance values, the third smallest value is used as the person distance. In this example, the third smallest value of the feature distance is used as the person distance, but it is not limited to this and can be set arbitrarily as appropriate. For example, instead of the third smallest value, other values from the smaller ones among the n A ×n B feature distance values can be used as the person distance, or the average value of the n

[0065] When the person distance is less than or equal to the threshold value, the same person determination means 250 determines that person A and person B are the same person. When the person distance is greater than the threshold value, it determines that they are not the same person.

[0066] When the same person determination means 250 determines that person A and person B are the same person, it combines the frame group of person A and the frame group of person B to update the frame group. For example, it rewrites the person ID of person B to the person ID of person A, rewrites the end time of person A to the end time of person B, and ends the processing for person B (see Fig. 4). Then, using the updated frame group of person A as the first frame group, the person selection means 230, the feature distance calculation means 240, and the same person determination means 250 are executed.

[0067] This process is repeated until no frame group of other person IDs whose start time is included within a predetermined time (here, 10 minutes) from the end time of the first frame group is found.

[0068] If there is no other person to be selected as a target within 10 minutes from the end time of the first frame group, the process for person A ends.

[0069] After that, in the person selection means 230, the frame group with the person ID whose start time included in the time information is the next oldest is selected, and the processes in the person selection means 230, the feature amount distance calculation means 240, and the same person determination means 250 are repeated. When the processes in the person selection means 230, the feature amount distance calculation means 240, and the same person determination means 250 are completed for all the frame groups of person IDs, the process in the person re - recognition means 310 ends.

[0070] Referring to FIG. 5, as an example, the process for seven persons with different person IDs will be described. FIG. 5 is a schematic diagram for explaining the operation of the person tracking device, particularly the processes in the person selection means 230 to the same person determination means 250, similar to FIG. 4. FIG. 5(A) shows the state before the processes in the person selection means 230 to the same person determination means 250 are performed, and FIG. 5(B) shows the state after the processes in the person selection means 230 to the same person determination means 250 are performed.

[0071] Before the processes in the person selection means 230 to the same person determination means 250 are performed, the seven frame groups of persons A to G shown in FIG. 5(A) are obtained.

[0072] The frame group of person A is selected as the first frame group as the person with the oldest start time. Next, the frame group of person B is selected as the second frame group as the person whose start time is included within a predetermined time from the end time of the frame group of person A. For the frame groups of person A and person B, the process of same - person determination is performed, and if they are determined to be the same, the frame groups of person A and person B are combined.

[0073] This combined frame group of person A is newly selected as the first frame group, and the frame of person C is regarded as the person whose start time is included within a predetermined time from the end time of the frame group of person A. Select a group of frames as the second group of frames. If, as a result of performing the same-person determination process on the group of frames of Person A and the group of frames of Person C, they are not determined to be the same, the group of frames of Person A and the group of frames of Person C are not combined. If no group of frames of other persons is found as a person whose start time is included within a predetermined time from the end time of the group of frames of Person A, end the process for Person A.

[0074] Next, select the group of frames of Person D as the first group of frames with the start time being the person second oldest after Person A. Next, select the group of frames of Person G as the second group of frames as a person whose start time is included within a predetermined time from the end time of the group of frames of Person D. If, as a result of performing the same-person determination process on the group of frames of Person D and the group of frames of Person G, they are not determined to be the same, the group of frames of Person D and the group of frames of Person G are not combined. Next, select the group of frames of Person E as the second group of frames as a person whose start time is included within a predetermined time from the end time of the group of frames of Person D. If, as a result of performing the same-person determination process on the group of frames of Person D and the group of frames of Person E, they are determined to be the same, the group of frames of Person D and the group of frames of Person E are combined.

[0075] Select this combined group of frames of Person D as the new first group of frames, and select the group of frames of Person C as the second group of frames as a person whose start time is included within a predetermined time from the end time of the group of frames of Person D. If, as a result of performing the same-person determination process on the group of frames of Person D and the group of frames of Person C, they are not determined to be the same, the group of frames of Person D and the group of frames of Person C are not combined. If no group of frames of other persons is found as a person whose start time is included within a predetermined time from the end time of the group of frames of Person D, end the process for Person D.

[0076] Next, as the person whose start time is the second oldest after person D, the frame group of person F is selected as the first frame group. Next, as the person whose start time is included within a predetermined time from the end time of the frame group of person F, the frame group of person G is selected as the second frame group. If, as a result of performing the same person determination process on the frame group of person F and the frame group of person G, they are not determined to be the same, the frame group of person F and the frame group of person G are not combined. If no frame group of another person is found as the person whose start time is included within a predetermined time from the end time of the frame group of person F, the process for person F ends.

[0077] Next, as the person whose start time is the second oldest after person F, the frame group of person G is selected as the first frame group. Next, as the person whose start time is included within a predetermined time from the end time of the frame group of person G, the frame group of person C is selected as the second frame group. If, as a result of performing the same person determination process on the frame group of person G and the frame group of person C, they are not determined to be the same, the frame group of person G and the frame group of person C are not combined. If no frame group of another person is found as the person whose start time is included within a predetermined time from the end time of the frame group of person G, the process for person F ends.

[0078] Next, as the person whose start time is the second oldest after person G, the frame group of person C is selected as the first frame group. If no frame group of another person is found as the person whose start time is included within a predetermined time from the end time of the frame group of person C, the process for person C ends.

[0079] As a result, the state shown in FIG. 5(B) is obtained.

[0080] After the processing in the above-described person re-recognition means 310 is completed, analysis processing is performed in the analysis means 330. The analysis processing in the analysis means is performed according to the design and is realized by any suitable conventionally known technique. For example, when performing action tracking in a certain facility, the analysis means 330 is such that a person Obtain how the inside passes through what route, or the stay time at each location, etc.

[0081] The analysis result in the analysis means 330 is sent to an output means composed of a display means such as a liquid crystal display (LCD) or a printing means such as a printer and output. Alternatively, it may be configured to send the analysis result to another terminal through a communication line such as the Internet.

[0082] The dimension determination means 320 obtains the elements of the feature vector used for the calculation of the feature amount distance and the dimension number d corresponding to the number of the elements. The process in the dimension determination means 320 is performed in advance before the actual tracking. Note that it may be performed appropriately during the actual tracking, or the data obtained by the actual tracking may be used.

[0083] In the dimension determination means 320, similarly to the person recognition means 310, the feature vector update means 210 groups the data with the same person ID and creates a frame.

[0084] In the dimension determination means 320, the process in the dimension reduction means 220 is not performed. That is, the deletion of the elements of the feature vector is not performed.

[0085] Similarly to the person recognition means 310, the person selection means 230 selects two persons (for example, person A and person B), and the feature amount distance calculation means 240 calculates the feature amount distance between the two persons using the above formula (1). At this time, since the deletion of the elements of the feature vector is not performed, f i is all 1.

[0086] Next, the dimension reduction determination means 260 makes a determination for each element of N dimensions, in this example, 512 dimensions, according to the following formula (2).

[0087]

Equation

[0088] As a result of the determination by the dimensionality reduction determination means 260, the dimension i that satisfies the above formula (2) is not considered as a reduction target as an important dimension. Specifically, for example, N (in this example, 512) arrays g i are prepared, and all elements are set to 0. When the above formula (2) is satisfied for a certain i, g i is set to 1.

[0089] This determination is performed for all combinations of the feature vector of person A and the feature vector of person B. As a result, the elements for which g i is 0 become deletion targets, and the elements for which g i is 1 become important elements and are not reduced. By substituting g i into f i , the reduced dimension is determined.

[0090] Here, the threshold value of the above formula (2) can be set arbitrarily as appropriate.

[0091] For example, consider the case where the threshold value is 1 / 100. This corresponds to considering a dimension important when the difference in feature amounts of a certain dimension is 1% or more of the feature amount distance. At this time, in a specific feature amount extraction model, the number of dimensions after deletion is 312 dimensions.

[0092] Note that if the difference in feature amounts is close to 1%, for example, 0.99%, in many dimensions, the dimensions with this value close to 1% are not determined to be important dimensions. However, considering all of these dimensions, it can be expected that the proportion in the feature amount distance will be significant in the case of 512 dimensions as in this example.

[0093] Next, consider the case where the threshold value is 1 / 100 × 1 / 512. This corresponds to selecting important dimensions so that even if all the values of unimportant dimensions are summed up, it is less than 1% of the feature amount distance. At this time, the number of dimensions after deletion is 326 dimensions.

[0094] Next, consider the case where the threshold is set to 1 / 512, which is a value between 1 / 100 and 1 / 100 × 1 / 512. This corresponds to the consideration that it is important when the difference in feature amounts in a certain dimension is equal to or greater than the average value per dimension of the feature amount distance. At this time, the number of dimensions after deletion is 322 dimensions.

[0095] As described above, there was no extreme difference in the number of dimensions for each case where the threshold was 1 / 100, 1 / 512, and 1 / 100 × 1 / 512.

[0096] Therefore, the threshold can be set to a predetermined ratio × 1 / dimension number. By setting the threshold in this way, even if all the values of unimportant dimensions are summed, it will be less than the predetermined ratio of the feature amount distance. For example, consider the case where the predetermined ratio is 1% and the number of dimensions is 512 dimensions, and the number of dimensions after reduction is 326 dimensions.

[0097] In this case, even when calculated with the reduced dimensions (326 dimensions in this example), it will be 99% or more of the value when calculated with all the dimensions before deletion (512 dimensions in this example). Therefore, the impact of dimension deletion on the feature amount distance is small. FIG. 6 is a diagram showing the result of examining the impact of dimension reduction on the feature amount distance.

[0098] In FIGS. 6(A) to (C), the horizontal axis shows the feature amount distance, and the vertical axis shows the count number. FIG. 6(A) shows the measurement results of the feature amount distances between the same person and another person before dimension reduction. FIG. 6(B) shows the measurement results of the feature amount distances between the same person and another person after dimension reduction. FIG. 6(C) shows the superposed measurement results of the feature amount distances between the same person and another person before and after dimension reduction.

[0099] As can be seen by comparing FIGS. 6(A) and (B), there is almost no difference in the calculation results of the feature amount distance before and after dimension reduction. FIG. 6(C) shows the results before and after dimension reduction, but the difference is not recognizable in the figure. Actually, the average of the differences in distances between the same persons before and after dimension reduction was 0.455, and the average of the differences in distances between different persons was 0.004.

[0100] On the other hand, the computational load in the feature amount distance calculation means 240 can be reduced to about 63.7% (= 326 / 512).

[0101] As described above, according to the distance tracking device of the present invention, the time for calculating the feature amount distance can be shortened without affecting the result of the same person determination.

[0102] The process of determining the same person in the same person determination means 250 is performed without using the camera ID. That is, even when a plurality of video cameras are used and a plurality of frame data are sent to the person tracking device, the process of determining the same person is possible.

[0103] Also, even when the frame data of the same camera ID is divided every time, for example, every hour, and sent to the person tracking device, the process of determining the same person is possible. Therefore, since the data can be processed at any time during data acquisition, in some cases, real-time processing is also possible.

Explanation of reference numerals

[0104] 100: Video camera 110: Video acquisition means 120: Person detection means 130: Person tracking means 140: Feature extraction means 200: Person tracking device 210: Feature vector update means 220: Dimension reduction means 230: Person selection means 240: Feature amount distance calculation means 250: Same person determination means 260: Reduced dimension determination means 310: Person re-identification means 320: Dimension determination means 330: Analysis means

Claims

1. Feature vector update means, Dimensionality reduction means, Person selection means, Feature quantity distance calculation means, and Same person determination means A person recognition means composed of; The feature vector update means, The person selection means, The feature quantity distance calculation means, and Dimensionality reduction determination means A dimensionality determination means composed of and comprising: The feature vector update means receives frame data including a plurality of frames having a person ID, time information, and a feature vector, groups the data of the frames having the same person ID included in the frame data, and creates a frame group. The dimensionality reduction means retains important elements determined by the dimensionality determination means among the elements of the N-dimensional (N is a natural number of 2 or more) feature vector included in the frame data, deletes other elements, and updates the feature vector. The person selection means selects a first frame group from a plurality of frame groups, searches for other frame groups in chronological order of frame data from the end time of the first frame group toward the future, and selects another frame group whose start time is included within a predetermined time from the end time of the first frame group as a second frame group. The feature quantity distance calculation means calculates the feature quantity distance between each of one or more frames included in the first frame group selected by the person selection means and one or more frames included in the second frame group as the distance of the feature vector. The same person determination means determines a person distance based on the feature quantity distance. If the person distance is equal to or less than a threshold value, it determines that they are the same person and combines the first frame group and the second frame group. If the person distance is greater than the threshold value, it determines that they are not the same person. The dimensionality reduction determination means designates an element as an important element when the ratio of the distance of a certain element of the feature vector to the feature quantity distance calculated by the feature quantity distance calculation means is greater than a predetermined threshold value. A person tracking device.

2. The same person determination means determines, as the person distance, the r-th (r is an integer of 2 or more) smallest value among the feature quantity distances between each of one or more frames included in the first frame group and one or more frames included in the second frame group, which are calculated by the feature quantity distance calculation means. The person tracking device according to Claim 1.

3. The predetermined threshold value in the dimensionality reduction determination means is smaller than 1 / N. The person tracking device according to Claim 1.

4. One or more video cameras, A processing device and is provided, wherein the video camera creates the frame data and sends it to the processing device, the frames included in the frame data have a camera ID for identifying the video camera, the video camera is configured to include a video acquisition means, a person detection means, a person tracking means, and a feature extraction means is provided, the video acquisition means generates frame data indicating a moving image composed of consecutive frames, the person detection means detects a person, creates a bounding box which is a rectangular area surrounding the person, and adds person ID which is an identification number given to the person and bbox information for determining the bounding box to the frame data, the person tracking means assigns the same person ID to the same person appearing in consecutive frames of the moving image and updates the frame data, the feature extraction means generates a feature vector for the bounding box created for each frame and adds the feature vector to the frame data, the processing device is a person tracking device according to claims 1 to 3 person tracking system.

5. Performed by a person tracking device, before actual person tracking, a dimension determination process performed in advance, and a person recognition process performed during actual person tracking is provided, the dimension determination process is a process of receiving frame data including a plurality of frames having a person ID, time information, and a feature vector, grouping the data of the frames having the same person ID included in the frame data to create a frame group, selecting a first frame group from a plurality of frame groups, searching for other frame groups in chronological order of frame data from the end time of the first frame group towards the future, and selecting another frame group whose start time is included within a predetermined time from the end time of the first frame group as a second frame group, a process of calculating the feature amount distance between the first frame group and the second frame group selected by the person selection means as the distance between the feature vectors, a process of setting an element as an important element when the ratio of the distance of an element of the feature vector to the feature amount distance is greater than a predetermined threshold is provided, the person recognition process is a process of receiving frame data including a plurality of frames having a person ID, time information, and a feature vector, grouping the data of the frames having the same person ID included in the frame data to create a frame group, A process of updating a feature vector by leaving the important elements among the elements of an N-dimensional (N is a natural number of 2 or more) feature vector included in frame data and deleting other elements. A process of selecting a first frame group from a plurality of frame groups, searching for other frame groups in chronological order of frame data from the end time of the first frame group toward the future, and selecting, as a second frame group, another frame group whose start time is included within a predetermined time from the end time of the first frame group. A process of calculating, as the distance between the feature vectors, the feature amount distance between the first frame group and the second frame group selected by the person selection means. A process of determining a person distance based on the feature amount distance, determining that they are the same person and combining the first frame group and the second frame group if the person distance is less than or equal to a threshold value, and determining that they are not the same person if the person distance is greater than the threshold value. Comprising A person tracking method.

6. In determining the same person, among the feature amount distances between each of one or more frames included in the first frame group and one or more frames included in the second frame group, Determine, as the person distance, a value that is the r-th (r is an integer of 2 or more) smallest value determined in advance. The person tracking method according to claim 5.

7. The predetermined threshold value is smaller than 1 / N. The person tracking method according to claim 5.

8. Before the execution of the dimension determination process and the person re-recognition process, A process of generating frame data showing a moving image composed of consecutive frames. A process of detecting a person, creating a bounding box which is a rectangular area surrounding the person, and adding, to the frame data, bbox information that determines a person ID, which is an identification number given to the person, and the bounding box. A process of updating the frame data by assigning the same person ID to the same person appearing in consecutive frames of the moving image. A process of generating a feature vector for each bounding box created for each frame and adding the feature vector to the frame data. The person tracking method according to any one of claims 5 to 7, comprising

Images