Object Tracking Processing Device, Object Tracking Processing Method, and Program

Abstract

An object tracking processing device (1) comprises an object grouping processing unit (20) that calculates at least one similar object group including at least one object similar to a tracked object on the basis of at least a feature of the tracked object, and an object tracking unit (50) that assigns, to an object belonging to the similar object group, a tracking ID for identifying the object. The present invention thereby improves the accuracy of tracking an object appearing in a video.

Description

Technical Field

[0001] The present disclosure relates to an object tracking processing apparatus, an object tracking processing method, and a non-transitory computer-readable medium.

Background Art

[0002] For example, Patent Document 1 describes a system that detects an object appearing in a video and tracks (MOT (Multi Object Tracking)) the same object across successive frames.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, since the same object is determined based on the non-spatiotemporal similarity of the object, there is a problem that a tracking result that violates the spatiotemporal constraints appears, and the tracking accuracy decreases.

[0005] An object of the present disclosure is to provide an object tracking processing apparatus, an object tracking processing method, and a non-transitory computer-readable medium that can improve the tracking accuracy of an object appearing in a video in view of the above-described problems.

Means for Solving the Problems

[0006] The object tracking processing apparatus of the present disclosure includes an object grouping processing unit that calculates at least one similar object group including at least one object similar to the object to be tracked based on at least a feature amount of the object to be tracked, and an object tracking unit that assigns a tracking ID for identifying the object to the object belonging to the similar object group.

[0007] The object tracking processing method of the present disclosure includes an object grouping processing step of calculating at least one similar object group including at least one object similar to the object to be tracked based on at least the feature amount of the object to be tracked, and an object tracking step of assigning a tracking ID for identifying the object to the object belonging to the similar object group.

[0008] Another object tracking processing method of the present disclosure includes: a step of detecting an object to be tracked and a feature amount of the object to be tracked in a frame each time the frame constituting the video is input; calculating at least one similar object group including at least one object similar to the object to be tracked based on at least the feature amount of the detected object to be tracked by referring to an object feature amount storage unit; storing, in the object feature amount storage unit, a position of the object, a detection time of the object, a feature amount of the object, and a group ID for identifying a group to which the object belongs for the detected object to be tracked; storing, in an object group information storage unit, a position of the object, a detection time of the object, and a group ID for identifying a group to which the object belongs for the detected object to be tracked; and executing a batch process of referring to the object group information storage unit at a predetermined cycle and assigning a tracking ID for identifying the object to the object belonging to the similar object group.

[0009] The non-transitory computer-readable medium of the present disclosure is a non-transitory computer-readable medium recording a program for causing a computer to execute an object grouping processing step of calculating at least one similar object group including at least one object similar to the object to be tracked based on at least the feature amount of the object to be tracked, and an object tracking step of assigning a tracking ID for identifying the object to the object belonging to the similar object group.

Effect of the Invention

[0010] According to the present disclosure, an object tracking processing device, an object tracking processing method, and a non-transitory computer-readable medium capable of improving the tracking accuracy of an object appearing in a video can be provided.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0012] (Embodiment 1) First, with reference to FIG. 1, a configuration example of the object tracking processing apparatus 1 according to Embodiment 1 will be described.

[0013] FIG. 1 is a schematic configuration diagram of the object tracking processing apparatus 1.

[0014] As shown in FIG. 1, the object tracking processing apparatus 1 includes an object grouping processing unit 20 that calculates at least one similar object group including at least one object similar to the object to be tracked based on at least the feature amount of the object to be tracked, and an object tracking unit 50 that assigns a tracking ID to the objects belonging to the similar object group.

[0015] Next, an example of the operation of the object tracking processing apparatus 1 will be described.

[0016] FIG. 2 is a flowchart of an example of the operation of the object tracking processing apparatus 1.

[0017] First, the object grouping processing unit 20 calculates at least one similar object group including at least one object similar to the object to be tracked based on at least the feature amount of the object to be tracked (step S1).

[0018] Next, the object tracking unit 50 assigns a tracking ID to the objects belonging to the similar object group (step S2).

[0019] As described above, according to Embodiment 1, the tracking accuracy of the objects appearing in the video can be improved.

[0020] This is achieved by executing a two-stage process of detecting an object to be tracked in a frame, classifying the detected object to be tracked into a group of similar objects (a process using non-spatiotemporal similarity), and for each of the classified groups of similar objects, assigning a tracking ID for identifying the object to the objects belonging to the group of similar objects (a process using spatiotemporal similarity). That is, high tracking accuracy can be realized by achieving both the matching of the same object over a wide range of frames and time and the consideration of spatiotemporal similarity.

[0021] (Embodiment 2) Hereinafter, as Embodiment 2 of the present disclosure, the object tracking processing device 1 will be described in detail. Embodiment 2 is an embodiment that concretizes Embodiment 1.

[0022] First, the outline of the object tracking processing device 1 will be described.

[0023] The object tracking processing device 1 is a device that detects all objects appearing in a single video and tracks the same object across successive frames (MOT (Multi Object Tracking)). A single video refers to a video input from one camera 70 (see FIG. 12) or one video file (not shown). A frame refers to each individual frame (hereinafter also referred to as an image) constituting the single video.

[0024] The object tracking processing device 1 executes a two-stage process.

[0025] FIG. 3A is an image diagram of the first-stage process executed by the object tracking processing device 1.

[0026] As a first-stage process, the object tracking processing device 1 executes a process (online process) of detecting an object to be tracked in a frame and classifying the detected object to be tracked into a similar object group. This process utilizes the non-spatiotemporal similarity of objects. FIG. 3A shows that as a result of executing the first-stage process for frames 1 to 3, each object to be tracked (persons U1 to U4) is classified into three similar object groups G1 to G3.

[0027] FIG. 3B is an image diagram of the second-stage process executed by the object tracking processing device 1.

[0028] As a second-stage process, the object tracking processing device 1 executes a process (batch process) of assigning a tracking ID for identifying the object to each object belonging to the similar object group for each similar object group classified in the first-stage process. At this time, the object tracking processing device 1 performs online tracking based on spatiotemporal similarity, such as the overlap between the detection position of the object (refer to the rectangular frame drawn with a solid line in FIG. 3B) and the predicted position of the tracked object (refer to the rectangular frame drawn with a dotted line in FIG. 3B), and IoU (Intersection over Union). This process utilizes spatiotemporal similarity.

[0029] By executing the two-stage process as described above, high tracking accuracy that cannot be achieved by a process using either non-spatiotemporal similarity or spatiotemporal similarity of objects can be achieved. Also, by classifying the objects to be tracked into similar object groups, the process of assigning a tracking ID for identifying the object to each object belonging to the similar object group can be executed in parallel for each similar object group. Thereby, an improvement in throughput can be realized.

[0030] Next, the details of the object tracking processing device 1 will be described.

[0031] FIG. 4 is a block diagram showing the configuration of the object tracking processing device 1 according to Embodiment 2.

[0032] As shown in FIG. 4, the object tracking processing apparatus 1 includes an object detection unit 10, an object grouping processing unit 20, an object feature amount information storage unit 30, an object group information storage unit 40, an object tracking unit 50, and an object tracking information storage unit 60.

[0033] The object detection unit 10 executes a process of detecting an object to be tracked (the position of the object to be tracked) and the feature amount of the object to be tracked in a frame constituting a single video. This process is an online process that is executed every time a frame is input. This process is realized by executing predetermined image processing on the frame. As the predetermined image processing, various existing algorithms can be used. The object detected by the object detection unit 10 is, for example, a moving object (a moving body) such as a person, a vehicle, or a motorcycle. Hereinafter, an example in which the object detected by the object detection unit 10 is a person will be described. The feature amount is an object feature amount (ReID), which refers to data that can calculate a similarity score between two objects by comparison. The position of the object detected by the object detection unit 10 is, for example, the coordinates of a rectangular frame surrounding the object detected by the object detection unit 10. The feature amount of the object detected by the object detection unit 10 is, for example, the feature amount of a person's face or the feature amount of a person's skeleton. The object detection unit 10 may be built into the camera 70 (see FIG. 12) or may be provided outside the camera 70.

[0034] The object grouping processing unit 20 executes a process of calculating at least one similar object group including at least one object similar to the object to be tracked based on at least the feature amount of the object to be tracked by referring to the object feature amount information storage unit 30. At this time, the object grouping processing unit 20 uses the non-spatial similarity of the objects (for example, the similarity of face feature data or humanoid feature data) to execute a process (clustering) of classifying the objects detected by the object detection unit 10 into similar object groups. This process is an online process that is executed every time the object detection unit 10 detects an object. As the clustering algorithm, data clustering and grouping techniques based on similarity with data at a wide time interval, for example, DBSCAN, k-means, and agglomerative clustering can be used.

[0035] Specifically, the object grouping processing unit 20 refers to the object feature amount information storage unit 30 and searches for similar objects similar to the objects detected by the object detection unit 10. At this time, all the information stored in the object feature amount information storage unit 30 (for example, the feature amounts for all frames) may be the search target, or a part of the information stored in the object feature amount information storage unit 30 (for example, the feature amounts for 500 frames stored within 30 seconds from the current time) may be the search target.

[0036] As a result of the above search, when similar objects are searched, the object grouping processing unit 20 assigns the group ID of the similar objects to the objects detected by the object detection unit 10. Specifically, the object grouping processing unit 20 stores in the object feature amount information storage unit 30 the position of the object, the detection time of the object, the feature amount of the object, and the group ID that identifies the similar object group to which the object belongs. Note that when no similar objects are searched, a newly numbered group ID is assigned.

[0037] In the object feature quantity information storage unit 30, for each object detected by the object detection unit 10, the position of the object, the detection time of the object, the feature quantity of the object, and the group ID assigned to the object are stored. Since the object feature quantity information storage unit 30 is frequently accessed by the object grouping processing unit 20, it is preferably a storage device (such as a memory) that can read and write at high speed.

[0038] In the object group information storage unit 40, information regarding objects belonging to similar object groups is stored. Specifically, in the object group information storage unit 40, for each object detected by the object detection unit 10, the position of the object, the detection time of the object, and the group ID that identifies the similar object group to which the object belongs are stored. Note that the feature quantity of the object may further be stored in the object group information storage unit 40. Since the object group information storage unit 40 is not frequently accessed compared to the object feature quantity information storage unit 30, it does not necessarily have to be a storage device (such as a memory) that can read and write at high speed. For example, the object group information storage unit 40 may be a hard disk device.

[0039] The object tracking unit 50 executes a process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group calculated by the object grouping processing unit 20. The tracking ID is an identifier assigned to the same object across successive frames. This process is a batch process with a time period (time cycle) that is executed every time a predetermined time (for example, 5 minutes) elapses. This batch process acquires information on the objects belonging to the similar object group for which there has been an update from the object group information storage unit 40, and based on the acquired information, assigns tracking IDs to the objects belonging to the similar object group. At that time, the object tracking unit 50 performs a process of determining the same object using spatio-temporal similarity, for example, online tracking based on the overlap between the detection position of the object and the predicted position of the tracked object, and IoU (Intersection over Union). As this algorithm, for example, the Hungarian method can be used. The Hungarian method is an algorithm that calculates the cost from the degree of overlap between the predicted positions of the detected object and the tracked object, and determines the assignment that minimizes the cost. The Hungarian method will be described in more detail later. Note that this algorithm is not limited to the Hungarian method, and other algorithms, for example, the greedy method, can also be used. Note that in the determination of the same object by the object tracking unit 50, not only spatio-temporal similarity but also non-spatio-temporal similarity may be used.

[0040] The object tracking unit 50 exists in the same number (is generated in the same number) as the similar object groups calculated by the object grouping processing unit 20. Each object tracking unit 50 executes in parallel the process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group (one different similar object group for each) that each is in charge of. Thus, in the present embodiment, when the object grouping processing unit 20 calculates a plurality of similar object groups, the process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group can be executed in parallel. Note that the number of objects belonging to a similar object group may be one or a plurality. For example, in FIG. 3A, two persons U1 and U2 belong to the similar object group G1, one person U3 belongs to the similar object group G2, and one person U4 belongs to the similar object group G3.

[0041] The tracking ID assigned by the object tracking unit 50 is stored in the object tracking information storage unit 60. Specifically, in the object tracking information storage unit 60, for each object, the position of the object, the detection time of the object, and a group ID for identifying the similar object group to which the object belongs are stored. Since the object tracking information storage unit 60 is not accessed frequently compared to the object feature amount information storage unit 30, it does not have to be a storage device (such as a memory) that can read and write at high speed. For example, the object tracking information storage unit 60 may be a hard disk device.

[0042] Next, as an operation example of the object tracking processing device 1, the process of grouping similar human forms (the first-stage process) will be described.

[0043] FIG. 5 is a flowchart of the process of grouping the objects detected by the object detection unit 10. FIGS. 6 and 7 are conceptual diagrams of the process of grouping the objects detected by the object detection unit 10.

[0044] Hereinafter, on the premise that a frame constituting a single video captured by a camera 70 (see FIG. 12) is sequentially input to the object detection unit 10. For example, it is assumed that frames 1, 2, 3,... are sequentially input to the object detection unit 10 in this order. Also, it is assumed that nothing is initially stored in the object feature amount information storage unit 30, the object group information storage unit 40, and the object tracking information storage unit 60.

[0045] The following processing is executed for each frame (every time a frame is input).

[0046] First, the processing when frame 1 is input will be described.

[0047] First, when frame 1 is input, the object detection unit 10 executes a process of detecting an object to be tracked in the frame 1 (image) and detecting (calculating) the feature amount of the object to be tracked (step S10).

[0048] Here, as shown in FIG. 6, frame 1 (an image including persons U1 to U4) is input, and persons U1 to U4 in the frame 1 are detected as objects to be tracked (step S100), and it is assumed that the feature amounts of the detected persons U1 to U4 are detected.

[0049] Next, for each object detected in step S10, the object grouping processing unit 20 refers to the object feature amount information storage unit 30 and searches for similar objects having a similarity score higher than threshold 1 (step S11). Threshold 1 is a threshold representing the lower limit of the similarity score. At that time, all the feature amounts stored in the object feature amount information storage unit 30 (for example, the feature amounts for all frames) may be the search target, or a part of the feature amounts stored in the object feature amount information storage unit 30 (for example, the feature amounts for 500 frames stored within 30 seconds from the current time) may be the search target. By setting a part of the feature amounts stored in the object feature amount information storage unit 30 (for example, the feature amounts for 500 frames stored within 30 seconds from the current time) as the search target, it is possible to suppress the deterioration of the freshness of the feature amounts.

[0050] For example, for the person U1 detected in step S10 (step S100), no similar object is searched even if the process of step S11 is executed. This is because nothing is stored in the object feature amount information storage unit 30 at this point (see step S101 in FIG. 6).

[0051] Next, the object grouping processing unit 20 determines whether the number of similar objects in the search result in step S11 is 2 or more (step S12). The threshold value 2 is a threshold value representing the lower limit of the number of similar objects.

[0052] For the person U1 detected in step S10, no similar object is searched even if the process of step S11 is executed, so the determination result in step S12 is No.

[0053] In this case, for the person U1 detected in step S10, the object grouping processing unit 20 assigns a group ID (for example, 1) to a new object (person U1) (step S13), and stores the assigned group ID and related information (the position of person U1, the detection time of person U1) in the object group information storage unit 40 in association with each other (step S14, step S102 in FIG. 6). Further, the object grouping processing unit 20 stores the group ID assigned in step S13 and related information (the position of person U1, the detection time of person U1, the feature amount of person U1) in the object feature amount information storage unit 30 in association with each other (see step S103 in FIG. 6).

[0054] On the other hand, for the person U2 detected in step S10, when the process of step S11 is executed, person U1 is searched as a similar object. This is because the group ID of person U1 and related information (the position of person U1, the detection time of person U1, the feature amount of person U1) are stored in the object feature amount information storage unit 30 at this point (see step S104 in FIG. 6). Therefore, the determination result in step S12 is Yes (when the threshold value 2 is 0).

[0055] In this case, the object grouping processing unit 20 determines whether all the similar objects in the search result in step S11 have the same group ID (step S15).

[0056] Regarding the person U2 detected in step S10, since all the similar objects (person U1) in the search result in step S11 have the same group ID, the determination result in step S15 is Yes.

[0057] In this case, for the person U2 detected in step S10, the object grouping processing unit 20 associates the group ID of the similar object (person U1) detected in step S11 and the related information (the position of person U2, the detection time of person U2) with each other and stores them in the object group information storage unit 40 (step S14, step S105 in FIG. 6). Also, the object grouping processing unit 20 associates the group ID of the similar object (person U1) detected in step S11 and the related information (the position of person U1, the detection time of person U1, the feature amount of person U1) with each other and stores them in the object feature amount information storage unit 30 (see step S106 in FIG. 6).

[0058] On the other hand, regarding the person U3 detected in step S10, even if the process of step S11 is executed, no similar object is searched for, so the determination result in step S12 is No.

[0059] In this case, for the person U3 detected in step S10, the object grouping processing unit 20 assigns a new group ID (for example, 2) to the new object (person U3) (step S13), and associates the assigned group ID and the related information (the position of person U3, the detection time of person U3) with each other and stores them in the object group information storage unit 40 (step S14, step S108 in FIG. 6). Also, the object grouping processing unit 20 associates the group ID assigned in step S13 and the related information (the position of person U3, the detection time of person U3, the feature amount of person U3) with each other and stores them in the object feature amount information storage unit 30 (see step S109 in FIG. 6).

[0060] Similarly, for the person U4 detected in step S10, even if the process of step S11 is executed, no similar object is retrieved, so the determination result of step S12 is No.

[0061] In this case, for the person U4 detected in step S10, the object grouping processing unit 20 assigns a group ID (for example, 3) to a new object (person U4) (step S13), and associates the assigned group ID and related information (the position of person U4, the detection time of person U4) with each other and stores them in the object group information storage unit 40 (step S14, step S111 in FIG. 6). Further, the object grouping processing unit 20 associates the group ID assigned in step S13 and related information (the position of person U4, the detection time of person U4, the feature amount of person U4) with each other and stores them in the object feature amount information storage unit 30 (not shown).

[0062] Next, the processing when frames after frame 1 (for example, frame 2) are input will be described.

[0063] First, when frame 2 is input, the object detection unit 10 executes a process of detecting an object to be tracked in the frame 2 (image) and detecting (calculating) the feature amount of the object to be tracked (step S10).

[0064] Here, as shown in FIG. 7, frame 2 (an image including persons U1 to U4) is input, and persons U1 to U4 in the frame 2 are detected as objects to be tracked (step S200), and it is assumed that the feature amounts of the detected persons U1 to U4 are detected.

[0065] Next, for each object detected in step S10, the object grouping processing unit 20 refers to the object feature amount information storage unit 30 and searches for similar objects having a similarity score higher than threshold 1 (step S11). Threshold 1 is a threshold representing the lower limit of the similarity score. At this time, all the information stored in the object feature amount information storage unit 30 (for example, feature amounts for all frames) may be the search target, or a part of the information stored in the object feature amount information storage unit 30 (for example, feature amounts for 500 frames stored within 30 seconds from the current time) may be the search target. Note that by setting a part of the information stored in the object feature amount information storage unit 30 (for example, feature amounts for 500 frames stored within 30 seconds from the current time) as the search target, deterioration of the freshness of the feature amounts can be suppressed.

[0066] For example, for the person U1 detected in step S10 (step S200), when the process of step S11 is executed, the persons U1 and U2 are retrieved as similar objects. This is because at this time, the object feature amount information storage unit 30 stores the group ID and related information of the person U1 (the position of the person U1, the detection time of the person U1, the feature amount of the person U1) and the group ID and related information of the person U2 (the position of the person U2, the detection time of the person U2, the feature amount of the person U2) (see step S201 in FIG. 6). Therefore, the determination result in step S12 becomes Yes (when threshold 2 is 0).

[0067] In this case, the object grouping processing unit 20 determines whether all the similar objects in the search result in step S11 have the same group ID (step S15).

[0068] For the person U1 detected in step S10 (step S200), since all the similar objects (persons U1 and U2) in the search result in step S11 have the same group ID, the determination result in step S15 becomes Yes.

[0069] In this case, for the person U1 detected in step S10 (step S200), the object grouping processing unit 20 associates the group ID of the similar objects (persons U1, U2) detected in step S11 and the related information (the position of person U1, the detection time of person U1) with each other and stores them in the object group information storage unit 40 (step S14, step S202 in FIG. 6). Also, the object grouping processing unit 20 associates the group ID of the similar objects (persons U1, U2) detected in step S11 and the related information (the position of person U1, the detection time of person U1, the feature amount of person U1) with each other and stores them in the object feature amount information storage unit 30 (see step S203 in FIG. 7).

[0070] Incidentally, if all of the similar objects (persons U1, U2, U3) in the search result in step S11 do not have the same group ID, for example, if the group ID of person U1 is 1, the group ID of person U2 is 2, and the group ID of person U3 is 3, the determination result in step S15 is No. In this case, the object grouping processing unit 20 executes a process of integrating the group IDs. Specifically, the object grouping processing unit 20 integrates the group IDs in the search result and stores the integrated group ID in the object group information storage unit 40 (step S16). For example, the object grouping processing unit 20 changes all the persons (here, person U2) belonging to the similar object group with group ID 2 and all the persons (here, person U3) belonging to the similar object group with group ID 3 to group ID = 1.

[0071] Thereby, it is possible to integrate the persons (data) that have been erroneously classified into another similar object group (data cluster) during the process into the same similar object group.

[0072] When the process of integrating the group IDs is executed in this way, for the person U1 detected in step S10, the object grouping processing unit 20 associates the integrated group ID and related information (the position of person U1, the detection time of person U1) with each other and stores them in the object group information storage unit 40 (step S14). Also, the object grouping processing unit 20 associates the integrated group ID and related information (the position of person U1, the detection time of person U1, the feature amount of person U1) with each other and stores them in the object feature amount information storage unit 30. The same applies to persons U2 and U3.

[0073] Similarly, for the person U2 detected in step S10 (step S200), when the process of step S11 is executed, persons U1 and U2 are searched for as similar objects. At this point, in the object feature amount information storage unit 30, the group ID and related information of person U1 (the position of person U1, the detection time of person U1, the feature amount of person U1) and the group ID and related information of person U2 (the position of person U2, the detection time of person U2, the feature amount of person U2) are stored (see step S204 in FIG. 7). Therefore, the determination result in step S12 becomes Yes (when the threshold value 2 is 0).

[0074] In this case, the object grouping processing unit 20 determines whether all the similar objects in the search result in step S11 have the same group ID (step S15).

[0075] For the person U2 detected in step S10 (step S200), since all the similar objects (persons U1 and U2) in the search result in step S11 have the same group ID, the determination result in step S15 becomes Yes.

[0076] In this case, for the person U2 detected in step S10 (step S200), the object grouping processing unit 20 associates the group ID of the similar objects (persons U1, U2) detected in step S11 and the related information (the position of person U2, the detection time of person U2) with each other and stores them in the object group information storage unit 40 (step S14, step S205 in FIG. 7). Also, the object grouping processing unit 20 associates the group ID of the similar objects (persons U1, U2) detected in step S11 and the related information (the position of person U2, the detection time of person U2, the feature amount of person U2) with each other and stores them in the object feature amount information storage unit 30 (see step S206 in FIG. 7).

[0077] Similarly, for the person U3 detected in step S10 (step S200), when the process of step S11 is executed, person U3 is searched for as a similar object. This is because at this point, the object feature amount information storage unit 30 stores the group ID of person U3 and the related information (the position of person U3, the detection time of person U3, the feature amount of person U3) (see step S207 in FIG. 7). Therefore, the determination result in step S12 becomes Yes (when the threshold value 2 is 0).

[0078] In this case, the object grouping processing unit 20 determines whether all the similar objects in the search result in step S11 have the same group ID (step S15).

[0079] For the person U3 detected in step S10 (step S200), since all the similar objects (person U3) in the search result in step S11 have the same group ID, the determination result in step S15 becomes Yes.

[0080] In this case, for the person U3 detected in step S10 (step S200), the object grouping processing unit 20 associates the group ID of the similar object (person U3) detected in step S11 and the related information (the position of person U3, the detection time of person U3) with each other and stores them in the object group information storage unit 40 (step S14, step S208 in FIG. 7). Further, the object grouping processing unit 20 associates the group ID of the similar object (person U3) detected in step S11 and the related information (the position of person U3, the detection time of person U3, the feature amount of person U3) with each other and stores them in the object feature amount information storage unit 30 (see step S209 in FIG. 7).

[0081] Similarly, for the person U4 detected in step S10 (step S200), when the process of step S11 is executed, the person U4 is searched as a similar object. This is because at this point, the object feature amount information storage unit 30 stores the group ID of the person U4 and the related information (the position of the person U4, the detection time of the person U4, the feature amount of the person U4) (see step S210 in FIG. 7). Therefore, the determination result of step S12 becomes Yes (when the threshold value 2 is 0).

[0082] In this case, the object grouping processing unit 20 determines whether all the similar objects in the search result in step S11 have the same group ID (step S15).

[0083] For the person U4 detected in step S10 (step S200), since all the similar objects (person U4) in the search result in step S11 have the same group ID, the determination result of step S15 becomes Yes.

[0084] In this case, for the person U4 detected in step S10 (step S200), the object grouping processing unit 20 associates the group ID of the similar object (person U4) detected in step S11 and the related information (the position of person U4, the detection time of person U4) with each other and stores them in the object group information storage unit 40 (step S14, step S211 in FIG. 7). Further, the object grouping processing unit 20 associates the group ID of the similar object (person U4) detected in step S11 and the related information (the position of person U4, the detection time of person U4, the feature amount of person U4) with each other and stores them in the object feature amount information storage unit 30 (not shown).

[0085] Note that for frames after frame 2, the same processing as that for frame 2 is executed.

[0086] By executing the processing described in the above flowchart 1, the group ID and related information of each object detected in step S10 are stored in the object feature amount information storage unit 30 and the object group information storage unit 40 from moment to moment.

[0087] The example of executing the processing of the flowchart described in FIG. 5 above for each of the consecutive frames such as frame 1, frame 2, frame 3... has been described above, but it is not limited to this. For example, the processing of the flowchart described in FIG. 5 above may be executed for each of the frames skipped by 1 (or a plurality of) frames such as frame 1, frame 3, frame 5... Thereby, an improvement in throughput can be realized.

[0088] Next, as an operation example of the object tracking processing apparatus 1, a process (second-stage process) of assigning a tracking ID for identifying the object to the object belonging to the similar object group calculated by the object grouping processing unit 20 will be described. This process is executed by the object tracking unit 50.

[0089] The object tracking unit 50 exists in (is generated in) the same number as the similar object groups calculated by the object grouping processing unit 20. For example, as a result of executing the processing of the flowchart in FIG. 5 above, when three similar object groups are formed, as shown in FIG. 8, three object tracking units 50A to 50C exist (are generated). FIG. 8 shows a state where each of the object tracking units 50A to 50C is executing in parallel the process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group (one different similar object group for each) that each is in charge of.

[0090] The object tracking unit 50A executes the process of assigning a tracking ID for identifying the object to the objects (here, persons U1 and U2) belonging to the first similar object group (here, the similar object group with group ID 1). The object tracking unit 50B executes the process of assigning a tracking ID for identifying the object to the object (here, person U3) belonging to the second similar object group (here, the similar object group with group ID 2). The object tracking unit 50C executes the process of assigning a tracking ID for identifying the object to the object (here, person U4) belonging to the third similar object group (here, the similar object group with group ID 3). These processes are executed in parallel.

[0091] Hereinafter, by way of representative, the process in which the object tracking unit 50A assigns a tracking ID for identifying the object to the objects (here, persons U1 and U2) belonging to the first similar object group (the similar object group with group ID 1) will be described.

[0092] FIG. 9 is a flowchart of the process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group calculated by the object grouping processing unit 20. FIG. 10 is an image diagram of the process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group calculated by the object grouping processing unit 20.

[0093] First, when a predetermined time (e.g., 5 minutes) has elapsed, the object tracking unit 50A acquires from the object group information storage unit 40 the object group information (group ID and its related information) of all similar objects having the updated group ID (here, group ID = 1; the same applies hereinafter) (step S20).

[0094] "There has been an update" means the case where the same group ID and related information as the group ID already stored in the object group information storage unit 40 are additionally stored, the case where a new group ID and related information are additionally stored in the object group information storage unit 40, and also includes the case where the process of step S16 (the process of integrating group IDs) is executed and the result of that process is stored in the object group information storage unit 40 (step S14). Note that if there is no update, the processing of the flowchart shown in FIG. 9 is not executed even when a predetermined time (e.g., 5 minutes) has elapsed.

[0095] Next, the object tracking unit 50A sets the tracking ID of the object group information acquired in step S20 to unassigned (step S21).

[0096] Next, the object tracking unit 50A determines whether there is a next frame (step S24). Here, since there is a next frame (frame 2), the determination result of step S24 is Yes.

[0097] Next, the object tracking unit 50A determines whether the current frame (the frame to be processed) is frame 1 (step S25). Here, since the current frame (the frame to be processed) is frame 1 (the first frame), the determination result of step S25 is Yes.

[0098] Next, the object tracking unit 50A predicts the position in the next frame taking into account the current position of the object for the assigned tracked object (step S26).

[0099] For example, the object tracking unit 50A predicts the positions of persons U1 and U2 belonging to the similar object group with group ID 1 in frame 1 (the first frame) in the next frame (frame 2). As the prediction algorithm, for example, the one disclosed in https: / / arxiv.org / abs / 1602.00763 (code: https: / / github.com / abewley / sort, GPL v3) can be used. Here, as the predicted positions of persons U1 and U2, it is assumed that the positions of two rectangular frames A1 and A2 drawn with dotted lines in frame 2 in FIG. 10 are predicted.

[0100] Next, the object tracking unit 50A assigns a new tracking ID to an object without an assignment or with a cost higher than threshold 3 (step S27). Threshold 3 is a threshold representing the upper limit of the cost calculated based on the overlap of the object regions and the object similarity.

[0101] Here, since the person U1 belonging to the similar object group with group ID 1 in frame 1 (the first frame) has an unassigned tracking ID, the object tracking unit 50A assigns a new tracking ID (for example, 1) to the person U1 (step S27), and associates the assigned new tracking ID (=1) and related information (the position of person U1, the detection time of person U1) with each other and stores them in the object tracking information storage unit 60. Similarly, since the person U2 belonging to the similar object group with group ID 1 in frame 1 (the first frame) has an unassigned tracking ID, the object tracking unit 50A assigns a new tracking ID (for example, 2) to the person U2 (step S27), and associates the assigned new tracking ID (=2) and related information (the position of person U2, the detection time of person U2) with each other and stores them in the object tracking information storage unit 60.

[0102] Next, the object tracking unit 50A determines whether there is a next frame (step S24). Here, since there is a next frame (frame 2), the determination result of step S24 is Yes.

[0103] Next, the object tracking unit 50A determines whether the current frame (the frame to be processed) is Frame 1 (step S25). Here, since the current frame (the frame to be processed) is Frame 2, the determination result in step S25 is No.

[0104] Next, the object tracking unit 50A acquires all the object information of the current frame (Frame 2) and the predicted positions of the objects (persons U1, U2) that have been tracked up to the previous frame (Frame 1) (step S28). Here, assume that the positions of the two rectangular frames A1 and A2 drawn with dotted lines in Frame 2 in FIG. 10 (the positions predicted in step S26) are acquired as the predicted positions of the objects (persons U1, U2).

[0105] Next, the object tracking unit 50 assigns the tracking ID of the tracked object to the current object by the Hungarian method using the object region overlap and object similarity as a cost function (step S29). For example, the cost is calculated from the degree of overlap between the predicted positions of the detected object and the tracked object, and the assignment that minimizes the cost is determined.

[0106] Here, a specific example of the process of assigning the tracking ID of the tracked object to the current object by the Hungarian method will be described.

[0107] In this process, the matrix (table) shown in FIG. 11 is used. FIG. 11 is an example of a matrix (table) used in the process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group calculated by the object grouping unit 20. "Detection 1", "Detection 2", "Tracking 1", and "Tracking 2" in this matrix have the following meanings.

[0108] That is, in FIG. 10, the two rectangular frames A1 and A2 drawn with dotted lines in Frame 2 represent the predicted positions of the objects (persons U1, U2) predicted in the previous frame (Frame 1). One of these two rectangular frames A1 and A2 represents "Tracking 1", and the other represents "Tracking 2".

[0109] Also, in FIG. 10, the two rectangular frames A3 and A4 drawn with solid lines in the frame 2 represent the positions of the objects (persons U1, U2) detected in the current frame (frame 2). One of these two rectangular frames A3 and A4 represents "detection 1", and the other represents "detection 2".

[0110] Note that the matrix (table) shown in FIG. 11 is a 2×2 matrix, but it is not limited to this. Depending on the number of objects, it may be an N1×N2 matrix other than 2×2. N1 and N2 are integers of 1 or more, respectively.

[0111] The numerical values (hereinafter also referred to as costs) described in the matrix (table) shown in FIG. 11 have the following meanings.

[0112] For example, the 0.5 described at the intersection of "tracking 1" and "detection 1" is a numerical value obtained by subtracting the overlap degree (overlap area) / 2 between the predicted position representing "tracking 1" (one rectangular frame A1 drawn with a dotted line in frame 2 in FIG. 10) and the position representing "detection 1" (one rectangular frame A3 drawn with a solid line in frame 2 in FIG. 10) from 1.0. This numerical value represents that the two positions completely overlap when it is 0, and that the two positions do not overlap at all when it is 1. Also, the smaller this numerical value (the closer to 0), the greater the overlap degree of the two positions, and conversely, the larger this numerical value (the closer to 1), the smaller the overlap degree of the two positions. The same applies to the other numerical values (0.9, 0.1) described in the matrix (table) shown in FIG. 11.

[0113] In the case of the matrix (table) shown in FIG. 11, the object tracking unit 50A determines an assignment that minimizes the cost (has a large overlap degree). Specifically, the object tracking unit 50A assigns the tracking ID of "tracking 1" with the minimum cost (cost of 0.5) as the tracking ID of detection 1 (for example, person U1). In this case, the object tracking unit 50A associates the assigned tracking ID (=1) and related information (the position of person U1, the detection time of person U1) with each other for person U1 and stores them in the object tracking information storage unit 60.

[0114] On the other hand, the object tracking unit 50A assigns, as the tracking ID of the detection 2 (for example, the person U2), the tracking ID of "Tracking 2" with the minimum cost (the cost is 0.1). In this case, the object tracking unit 50A associates the assigned tracking ID (=2) and the related information (the position of the person U2, the detection time of the person U2) with each other for the person U2 and stores them in the object tracking information storage unit 60.

[0115] Next, the object tracking unit 50A predicts the position in the next frame while taking into account the current position of the object for the assigned tracked object (step S26).

[0116] For example, the object tracking unit 50A predicts the positions of the persons U1 and U2 belonging to the similar object group with the group ID of 1 in the frame 2 in the next frame (frame 3). Here, as the predicted positions of the persons U1 and U2, it is assumed that the positions of the two rectangular frames A5 and A6 drawn by the dotted lines in frame 3 in FIG. 10 are the predicted ones.

[0117] Next, the object tracking unit 50A assigns a new tracking ID to an object without an assignment or with a cost higher than the threshold 3 (step S27). The threshold 3 is a threshold representing the upper limit of the cost calculated based on the overlap of the object regions and the object similarity.

[0118] Here, since the persons U1 and U2 belonging to the similar object group with the group ID of 1 in frame 2 already have tracking IDs assigned and the cost is lower than the threshold 3, the process of step S26 is not executed.

[0119] Next, the object tracking unit 50A determines whether there is a next frame (step S24). Here, since there is a next frame (frame 3), the determination result of step S24 is Yes.

[0120] Next, the object tracking unit 50A determines whether the current frame (the frame to be processed) is frame 1 (step S25). Here, since the current frame (the frame to be processed) is frame 3, the determination result of step S25 is No.

[0121] Next, the object tracking unit 50A acquires all the object information of the current frame (frame 3) and the predicted positions of the objects (persons U1 and U2) that have been tracked up to the previous frame (frame 2) (step S28). Here, as the predicted positions of the objects (persons U1 and U2), it is assumed that the positions of the two rectangular frames A5 and A6 drawn with dotted lines in frame 3 in FIG. 10 (the positions predicted in step S26) are acquired.

[0122] Next, the object tracking unit 50A assigns the tracking ID of the tracked object to the current object by the Hungarian method using the overlap of the object regions and the object similarity as a cost function (step S29).

[0123] That is, as described above, the object tracking unit 50A determines the assignment that minimizes the cost (has a large overlap degree). Specifically, the object tracking unit 50A assigns the tracking ID of "tracking 1" with the minimum cost as the tracking ID of detection 1 (for example, person U1). In this case, the object tracking unit 50A associates the assigned tracking ID and related information (the position of person U1, the detection time of person U1) with each other for person U1 and stores them in the object tracking information storage unit 60.

[0124] On the other hand, the object tracking unit 50A assigns the tracking ID of "tracking 2" with the minimum cost as the tracking ID of detection 2 (for example, person U2). In this case, the object tracking unit 50A associates the assigned tracking ID and related information (the position of person U2, the detection time of person U2) with each other for person U2 and stores them in the object tracking information storage unit 60.

[0125] The above processing is repeatedly executed until there is no next frame (step S24: No).

[0126] Next, a hardware configuration example of the object tracking processing apparatus 1 (information processing apparatus) described in the above-described Embodiment 2 will be described. FIG. 12 is a block diagram showing a hardware configuration example of the object tracking processing apparatus 1 (information processing apparatus).

[0127] As shown in FIG. 12, the object tracking processing apparatus 1 is an information processing apparatus such as a server including a processor 80, a memory 81, a storage device 82, and the like. The server may be a physical machine or a virtual machine. Further, one camera 70 is connected to the object tracking processing apparatus 1 via a communication line (for example, the Internet).

[0128] By executing software (computer program) read from a memory 81 such as a RAM, the processor 80 functions as an object detection unit 10, an object grouping processing unit 20, and an object tracking unit 50. These functions may be implemented on a single server or may be distributed and implemented on a plurality of servers. Even in the case where they are distributed and implemented on a plurality of servers, the processing of each of the above flowcharts can be realized by the plurality of servers communicating with each other via a communication line (for example, the Internet). Note that some or all of these functions may be realized by hardware.

[0129] Also, the number of object tracking units 50 is the same as (generated to be the same as) the number of similar object groups divided by the object grouping processing unit 20. Each object tracking unit 50 may be implemented on a single server or may be distributed and implemented on a plurality of servers. Even in the case where they are distributed and implemented on a plurality of servers, the processing of each of the above flowcharts can be realized by the plurality of servers communicating with each other via a communication line (for example, the Internet).

[0130] The processor 80 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor may include a plurality of processors.

[0131] The memory 81 is composed of a combination of a volatile memory and a non-volatile memory. The memory may include a storage arranged separately from the processor. In this case, the processor may access the memory via an I / O interface (not shown).

[0132] The memory device 82 is, for example, a hard disk device.

[0133] In the example of FIG. 11, the memory is used to store a group of software modules. The processor can perform processes such as the object tracking processing device described in the above embodiments by reading and executing these groups of software modules from the memory.

[0134] The object feature amount information storage unit, the object group information storage unit, and the object tracking information storage unit may be provided in one server or may be distributed and provided in a plurality of servers.

[0135] As described above, according to the second embodiment, the tracking accuracy of the objects appearing in the video can be improved.

[0136] This is achieved by executing a two-stage process of detecting an object to be tracked in a frame and classifying the detected object to be tracked into a similar object group (a process using non-spatiotemporal similarity), and for each of these classified similar object groups, assigning a tracking ID for identifying the object to the objects belonging to the similar object group (a process using spatiotemporal similarity). That is, high tracking accuracy can be realized by achieving both the collation of the same object for a wide range of frames and time and the consideration of spatiotemporal similarity.

[0137] Further, according to the second embodiment, by executing a process (batch process) of assigning a tracking ID for identifying an object to the objects belonging to the similar object group calculated by the object grouping processing unit 20, it is possible to realize the discovery of frequently appearing persons in near real time. For example, by referring to the object tracking information storage unit 60, it is possible to easily discover objects (for example, persons) that frequently appear in a specific period and at a specific location. For example, it is possible to list the top 20 persons who frequently appear in the office in the past 7 days from now.

[0138] Moreover, according to Embodiment 2, the following effects can be achieved.

[0139] That is, in object tracking, detection omissions and tracking losses may occur due to occlusion by obstacles from the camera's field of view. In contrast, according to Embodiment 2, tracking losses can be improved by matching the same object over a wide range of frames and time.

[0140] In addition, object tracking that takes into account spatio-temporal similarity requires sequential processing in chronological order. Therefore, it is impossible to improve the throughput by parallelizing the processing per input unit. In contrast, according to Embodiment 2, by classifying the objects to be tracked into similar object groups, for each such similar object group, the process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group can be executed in parallel. As a result, the throughput can be improved. That is, by minimizing the sequential processing part in chronological order in the overall processing flow, it is possible to improve the throughput by parallelizing most of the processing.

[0141] On the other hand, in tracking based only on non-spatial similarity, mis-tracking that violates spatio-temporal constraints occurs, and the tracking accuracy deteriorates. In contrast, according to Embodiment 2, by executing the two-stage processing as described above, the tracking accuracy of the objects appearing in the video can be improved.

[0142] In the above example, the program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (such as flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (such as magneto-optical disks). Further, examples of non-transitory computer readable media include CD-ROM (Read Only Memory), CD-R, CD-R / W. Further, examples of non-transitory computer readable media include semiconductor memory. Semiconductor memory includes, for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory). Also, the program may be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or wireless communication channels. Note that the present disclosure is not limited to the above embodiments and can be appropriately changed without departing from the spirit. Also, the present disclosure may be implemented by appropriately combining each embodiment.

Explanation of Signs

[0143] 1…Object tracking processing device 10…Object detection unit 20…Object grouping processing unit 30…Object feature amount information storage unit 40…Object group information storage unit 50(50A~50B)…Object tracking unit 60…Object tracking information storage unit 70…Camera 80… Processor 81… Memory 82… Storage device

Claims

1. An object grouping processing unit that calculates at least one similar object group including at least one object having a feature amount similar to that of the object to be tracked, based on at least the feature amount of the object to be tracked; An object tracking unit that assigns a tracking ID for identifying the object to the objects belonging to the similar object group, and Further includes an object group information storage unit that stores, as information regarding the objects belonging to the similar object group, the position of the object, the detection time of the object, and a group ID for identifying the similar object group to which the object belongs. The object tracking unit performs batch processing at a predetermined cycle, The batch processing is an object tracking processing device that is a process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group for which an update has occurred.

2. The object tracking unit is provided for each of the similar object groups, The object tracking processing device according to claim 1, wherein each of the object tracking units executes in parallel a process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group for which each is responsible.

3. The object tracking processing device according to claim 1 or 2, further comprising an object tracking information storage unit in which the tracking ID assigned by the object tracking unit is stored.

4. An object detection unit that detects the object to be tracked and the feature amount of the object to be tracked in each frame constituting the video, For each object detected by the object detection unit, further includes an object feature amount storage unit that stores the position of the object, the detection time of the object, the feature amount of the object, and the group ID assigned to the object. The object grouping processing unit calculates at least one similar object group including at least one object having a feature amount similar to that of the object to be tracked, based on at least the feature amount of the object to be tracked, by referring to the object feature amount storage unit. The object tracking processing device according to any one of claims 1 to 3.

5. An object grouping processing step of calculating at least one similar object group including at least one object having a feature amount similar to that of the object to be tracked, based on at least the feature amount of the object to be tracked; An object tracking step of assigning a tracking ID for identifying the object to the objects belonging to the similar object group, and The object tracking step performs batch processing at a predetermined cycle, The object tracking processing method, wherein the batch processing is a process of assigning a tracking ID for identifying the object to the objects belonging to the similar object group having an update.

6. For each frame constituting the video, a step of detecting an object to be tracked in the frame and a feature amount of the object to be tracked; By referring to an object feature amount storage unit that stores the position of the object, the detection time of the object, the feature amount of the object, and a group ID for identifying the group to which the object belongs, based on at least the feature amount of the detected object to be tracked, calculating at least one similar object group including at least one object having a similar feature amount to the object to be tracked; For the detected object to be tracked, storing the position of the object, the detection time of the object, the feature amount of the object, and a group ID for identifying the group to which the object belongs in the object feature amount storage unit; For the detected object to be tracked, storing the position of the object, the detection time of the object, and a group ID for identifying the group to which the object belongs in an object group information storage unit; An object tracking processing method comprising: executing a batch process for assigning a tracking ID for identifying the object to the objects belonging to the similar object group at predetermined intervals.

7. An object grouping processing step of calculating at least one similar object group including at least one object having a similar feature amount to the object to be tracked based on at least the feature amount of the object to be tracked; An object tracking step of assigning a tracking ID for identifying the object to the objects belonging to the similar object group, which is a program for causing a computer to execute, The object tracking step performs a batch process at predetermined intervals, The batch process is a program for assigning a tracking ID for identifying the object to the objects belonging to the similar object group having an update.

Images