Tracking system for moving object
The system addresses the challenge of tracking moving objects across multiple cameras by generating a graph connecting camera nodes and tracking IDs, ensuring accurate and reliable tracking of the same object over a wide area.
Patent Information
- Application Number
- JP2023083394
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2043-05-19
AI Technical Summary
Existing tracking systems using multiple cameras struggle to accurately track a moving object over a wide area due to limitations in generating graphs that connect cameras with large Euclidean distances, leading to separate databases for node and edge graphs, which hinders the tracking of the same object across multiple cameras.
A moving object tracking system that generates a graph connecting nodes representing single cameras and tracking identification numbers for the same object across multiple cameras, using a processor to process video data from at least two cameras and store a graph in a database, linking nodes with edges indicating relationships between cameras and common tracking IDs.
Enables accurate tracking of the same moving object across a wide area using multiple cameras, ensuring reliable connections and real-time tracking with high precision by verifying and correcting inconsistencies in the graph connections.
Smart Images

Figure 2024166949000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a system that uses video data acquired by multiple cameras to track a moving object captured in the video data. [Background technology]
[0002] Japanese Patent Application Laid-Open Publication No. 2017-021753 discloses a technique for graphing images of people captured in video data acquired by multiple cameras. In this conventional technique, frames in which people are detected are extracted from among frames (images) included in the video data. Next, a rectangular area (i.e., a bounding box) containing the detected person is trimmed from this extracted frame. The extraction of frames in which people are detected and the trimming of the bounding box are performed on multiple frames that differ in at least one of the time and place of capture. Therefore, multiple bounding boxes are extracted.
[0003] In the technique described in the above publication, an image sequence consisting of five bounding boxes is then extracted from the extracted bounding boxes with reference to the Euclidean distance between the cameras. Then, a similarity between the person images included in the image sequence is calculated based on the feature quantities of the person images. This similarity is used to determine whether the person images included in the five bounding boxes are the same person.
[0004] In the technology described in the above publication, when it is determined that the human images contained in five bounding boxes are of the same person, a graph is generated for the five bounding boxes. This graph is expressed using nodes (vertices, nodes) and edges (sides, branches) in graph theory. Figures 7-9 of the above publication show an example of a graph in which five bounding boxes (human images) determined to be of the same person through data processing of video data acquired by multiple cameras installed in different locations are treated as nodes, and these bounding boxes are connected by edges.
[0005] JP 2022-086650 A discloses a technology for dividing surveillance camera video data into predetermined time intervals and generating a graph from the video data for each divided time interval. In this conventional technology, the graph includes a graph for nodes and a graph for edges. The graph for nodes includes a unique node ID assigned to a monitored element captured in the surveillance camera video data, and tracking information and attribute information of the element linked to the node ID. The graph for edges includes information on the interaction behavior of the monitored elements when the monitored elements include multiple people, and information on the relationship between the monitored elements when the monitored elements include people and objects.
[0006] In the technique described in the above publication, the generated graphs are stored in a database. The node graph is stored in a database for nodes, and the edge graph is stored in a database for edges. In other words, the generated node graphs and edge graphs are stored in separate databases. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] JP 2017-021753 A [Patent Document 2] Patent Publication No. 2022-086650 Summary of the Invention [Problem to be solved by the invention]
[0008] Consider tracking moving objects (people, robots, vehicles, etc.) captured in video data acquired by multiple cameras. According to the technology described in JP 2017-021753 A, a graph is generated for the same person captured in the video data. Therefore, by using this graph, it is possible to track the same moving object captured in the video data. However, in the technology of this publication, image sequences are extracted based on Euclidean distance, and therefore, this extraction is not performed between cameras with a large Euclidean distance. Therefore, the graph generated by the technology of this publication is not suitable for tracking moving objects moving in a wide range.
[0009] In addition, JP 2022-086650 A describes generating a graph for a single surveillance camera, but does not describe multiple surveillance cameras. Even if a graph is created for each of multiple surveillance cameras, the relationship between these graphs is not defined, and the graphs for nodes and edges generated for each surveillance camera are likely to be stored in separate databases. Therefore, the technology of this publication cannot track the same moving object reflected in the video data acquired by multiple cameras.
[0010] One objective of the present disclosure is to provide a technology capable of tracking the same moving object captured in video data acquired by multiple cameras arranged over a wide range, using the video data. [Means for solving the problem]
[0011] The present disclosure is a moving object tracking system having the following features. The tracking system includes a storage device, which stores image data captured by at least two cameras, and a processor, which is configured to perform data processing based on the image data. In the data processing, the processor generates a graph consisting of at least two nodes and at least one edge indicating the relationship between the at least two nodes, and stores the graph in a database in the storage device. In the graph, a node representing a single camera included in the at least two cameras is connected to a node representing a tracking identification number (tracking ID) given to a moving object captured in video data acquired by the single camera via at least one edge. The tracking identification number includes a common tracking identification number (common tracking ID) given to the same moving object captured in video data acquired by the single camera. In the graph, at least two of the single cameras have a relationship between them, and among the nodes representing these single cameras, the nodes having the relationship are connected to each other via at least one edge indicating the relationship between the single cameras. In the graph, nodes that represent the at least two common tracking identification numbers and that are recognized as being the same moving object are connected to each other via at least one edge that indicates that the at least two moving objects captured in each image data captured by the at least two single cameras are the same moving object. Effect of the Invention
[0012] According to the present disclosure, a graph consisting of at least two nodes and at least one edge indicating a relationship between the at least two nodes is generated and stored in a database in a storage device. In this graph, a node representing a single camera and a node representing a tracking identification number assigned to a moving object captured in video data acquired by the single camera are connected via at least one edge. In addition, the tracking identification number includes a common tracking identification number assigned to the same moving object captured in video data acquired by the single camera.
[0013] In this graph, nodes that have a relationship among the at least two single cameras are connected to each other through at least one edge that indicates the relationship between the at least two single cameras. In this graph, nodes that are recognized as the same moving object among the nodes that represent at least two common tracking identification numbers are connected to each other through at least one edge that indicates that at least two moving objects captured in each video data acquired by the at least two single cameras are the same moving object.
[0014] Therefore, by using such a graph, it becomes possible to track the same moving object captured in the video data obtained from at least two single cameras. This effect is expected to be obtained even when multiple cameras are placed over a wide area. [Brief description of the drawings]
[0015] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a tracking system according to an embodiment. [Diagram 2] 11A and 11B are diagrams illustrating a person detection process and a person re-identification process performed by a graph generation processing unit. [Diagram 3] 13 is a diagram illustrating an example of a basic configuration of a graph generated by a graph generation processing unit. FIG. [Figure 4] 13 is a diagram illustrating an example of a detailed configuration of a graph generated by a graph generation processing unit; FIG. [Diagram 5] 11 is a diagram illustrating an example of a verification process performed by a graph verification processing unit. FIG. [Figure 6] 11 is a diagram illustrating an example of a verification process performed by a graph verification processing unit. FIG. [Figure 7] 11 is a diagram illustrating an example of a verification process performed by a graph verification processing unit. FIG. [Figure 8] 11 is a diagram illustrating an example of a verification process performed by a graph verification processing unit. FIG. [Figure 9]11A and 11B are diagrams illustrating an example of a tracking process performed by a tracking processing unit. [Figure 10] 11A and 11B are diagrams illustrating an example of a tracking process performed by a tracking processing unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and the description thereof will be simplified or omitted.
[0017] 1. Overall configuration example FIG. 1 is a diagram for explaining an example of the overall configuration of a tracking system (hereinafter, also simply referred to as "system") according to an embodiment. The system according to the embodiment is a system for tracking a moving object moving inside a city CT. There is no limitation on the scale of the city CT in the present disclosure. A so-called smart city is an example of a large-scale city CT, an underground mall is an example of a medium-scale city CT, and a large building is an example of a small-scale city CT. Examples of the moving object include a person, a robot, and a vehicle. In the embodiment, it is assumed that the moving object is a person (pedestrian PD). In FIG. 1, pedestrians PD1-PD3 are depicted as examples of pedestrians PD.
[0018] The system according to the embodiment includes at least two cameras arranged in the city CT. In FIG. 1, cameras CA1-CA6 are illustrated as an example of the at least two cameras. Camera CA1 acquires video data VD_CA1. Like camera CA1, cameras CA2-CA6 acquire video data VD_CA2-VD_CA6, respectively. Video data VD_CAn of any one of cameras CAn (n is a natural number) arranged in the city CT is transmitted to the management server 10 via a communication network. The communication network is not particularly limited, and wired and wireless networks can be used.
[0019] The management server 10 is a computer including at least one processor 11, at least one storage device 12, and at least one interface 13. The processor 11 performs various data processing. The processor 11 includes a CPU (Central Processing Unit). The storage device 12 stores various data required for data processing. Examples of the storage device 12 include HDD, SSD, volatile memory, and non-volatile memory. The interface 13 receives various data from the outside and outputs various data to the outside. The various data received by the interface 13 from the outside includes video data VD_CAn. This video data VD_CAn is stored in the storage device 12. A graph DB (database) 17 is formed in the storage device 12. The graph DB 17 may be formed in an external device that can communicate with the management server 10.
[0020] 2. Management Server Configuration Example Fig. 1 shows an example of the functional configuration of the management server 10. In the example shown in Fig. 1, the management server 10 includes a graph generation processing unit 14, a graph verification processing unit 15, and a tracing processing unit 16. These functions are realized by the processor 11 executing various programs stored in the storage device 12.
[0021] 2-1.Graph generation processing section The graph generation processing unit 14 performs a process of generating the graph GPH based on the video data VD_CAn. In order to generate the graph GPH, the graph generation processing unit 14 performs a person detection and extraction process and a person re-identification process. Fig. 2 is a diagram for explaining the person detection process and the person re-identification process performed by the graph generation processing unit 14.
[0022] 2 illustrates video data VD_CAi from camera CAi and video data VD_CAj from camera CAj as an example of video data VD_CAn. The video data VD_CAi and VD_CAj are each separated by a predetermined time width, and FIG 2 illustrates a set of frames FR in a section time bt. Each frame FR of the video data VD_CAi is, for example, CAi As with each frame FR of the video data VD_CAi, each frame FR of the video data VD_CAj also includes the ID of the camera CAj. CAj and data at timestamp ts.
[0023] In the detection and extraction process, first, a frame FR in which a person is detected is extracted. Next, the detected person is extracted from this extracted frame. In the example shown in FIG. 2, pedestrians PDx and PDy are detected in each frame within section time bt1 (timestamp ts1-ts5) of video data VD_CAi. Furthermore, pedestrian PDz is detected in each frame within section time bt2 (timestamp ts6-ts10) of video data VD_CAj. Bounding boxes surrounding the pedestrians PDx, PDy, and PDz are added to the positions where the pedestrians PDx, PDy, and PDz are detected. By cropping this bounding box, an image IM of the pedestrians PDx, PDy, and PDz is obtained. PDx , I.M. PDy and I.M. PDz The image IM of the pedestrian PD is extracted. PDx and I.M. PDy For example, the ID of the camera CAi CAi The data of the time stamp ts and the coordinates CD in the frame FR of the extracted image PD The data includes:
[0024] In the re-identification process, each image IM extracted in the detection and extraction process is PDFrom each image IM, a feature for re-identification processing (hereinafter also referred to as "Re-ID feature") is extracted. The Re-ID feature is extracted using a Re-ID model based on machine learning. Note that the technology for extracting Re-ID features using the Re-ID model is well known, and the method is not particularly limited. PD Once the Re-ID features are extracted, the Re-ID features are compared to determine whether the people in the image sequences are the same person.
[0025] In the example shown in FIG. 2, each image IM PD Since the Re-ID feature values of the people extracted from the images IM and PD are similar to each other, it is determined that the pedestrians PDx and PDy included in the image sequence in the section time bt1 are the same person. PD Since the Re-ID feature values of the people extracted from are similar to each other, the pedestrian PDz included in the image sequence in the section time bt2 is determined to be the same person.
[0026] When the detection process is performed, a tracking ID is assigned to the person captured in the video data VD_CAn. PD Among these, if a person is determined to be the same person through the re-identification process, a common tracking ID for this person is assigned. PD (hereinafter, "Common Tracking ID PD ") is assigned to the shipment. PD is also called a universally unique ID (UUID). In the example shown in FIG. 2, for pedestrians PDx, PDy, and PDz, a common tracking ID PDx(bt1) , ID PDy(bt1) and ID PDz(bt2) Each common tracking ID is assigned. PD is a combination of data of a section time bt and data representative of the data of the Re-ID feature amount extracted from the image sequence of this section time bt. Note that there is no particular limitation on the example of selecting the representative data of the Re-ID feature amount, and any method can be adopted.
[0027] Common Tracking IDPD is generated for each section time bt. Therefore, if the same pedestrian PD continues to be captured by one camera (single camera), the common tracking ID PD may occur separately for the number of section times bt. Therefore, in the re-identification process, the Re-ID feature values may be compared between multiple image sequences that differ in the section time bt. For example, the Re-ID feature values are compared between two image sequences of section times bt that have close timestamps ts. If the Re-ID feature values are similar between multiple image sequences, it is determined that the pedestrians PD included in these image sequences are the same person, and the common tracking ID that was assigned separately to each pedestrian PD is used. PD may be combined into one.
[0028] The graph generation processing unit 14 uses the common tracking ID given to the pedestrian PD by the above-mentioned re-identification process. PD and the IDs of at least two cameras arranged in the city CT. The generated graph GRH is stored in the graph DB 17. As already explained, the graph GRH is expressed using nodes (vertices, nodes) and edges (sides, branches) in graph theory. FIG. 3 is a diagram showing an example of the basic configuration of the graph GRH generated by the graph generation processing unit 14. FIG. 3 shows a node N_ID representing each ID of the cameras CA1-CA6 (however, CA2 is omitted) shown in FIG. 1, CA1 -N_ID CA6 and edges E that indicate relationships between these nodes.
[0029] Here, the installation position of the camera CA1 is close to that of the camera CA3. Therefore, there is a relationship between these cameras. Therefore, in the graph GRH1 shown in FIG. 3, a node N_ID representing the ID of the camera CA1 is CA1 and a node N_ID representing the ID of the camera CA3. CA3are connected via edge E_CA1-3. The meaning of this edge E_CA1-3 is "NEARBY". The "NEARBY" relationship also exists between camera CA1 and camera CA4, between camera CA4 and camera CA5, and between camera CA5 and camera CA6. Therefore, the nodes N representing the IDs of two related cameras CA are connected by one edge E (edge E_CA1-4, edge E_CA4-5, and edge E_CA5-6).
[0030] Another example of the relationship between two cameras CA is that the shooting ranges of these cameras overlap in part or in whole. Here, part of the shooting range of camera CA3 overlaps with that of camera CA5. Therefore, in the graph GRH1 shown in FIG. 3, the node N_ID representing the ID of camera CA3 CA3 and a node N_ID representing the ID of the camera CA5. CA5 are connected via an edge E_CA3-5, which has the meaning "OVERLAPPED".
[0031] A common tracking ID assigned to pedestrian PD PD Represents the node N_ID PD This common tracking ID PD Node N_ID representing the camera CAn that acquired the video data VD_CAn that was the basis for the assignment CAn , and are connected to each other via at least one edge E. FIG. 3 shows the common tracking IDs assigned to pedestrians PDp, PDq, PDr, PDs, and PDu. PD Represents the node N_ID PDp , N_ID PDq , N_ID PDr , N_ID PDs and N_ID PDu is drawn. Node N_ID PDp connects to node N_ID via edge E_CA1 CA1 Node N_ID PDq and N_ID PDr connects to node N_ID via two edges E_CA4 CA4Node N_ID PDs connects to node N_ID via edge E_CA5 CA5 Node N_ID PDu connects to node N_ID via edge E_CA3 CA3 is tied to.
[0032] As already explained, the common tracking ID PD The re-identification process combines data of the section time bt and representative data of the Re-ID feature of the image sequence. In the embodiment, the representative data of the Re-ID feature is used to perform a re-identification process of the person captured by at least two cameras. In this re-identification process, the same process as the comparison of the Re-ID feature between a plurality of different image sequences in the section time bt is performed. However, while the comparison of the Re-ID feature between a plurality of image sequences is performed for one camera, the comparison of the representative data of the Re-ID feature is performed for two cameras. If the representative data of the Re-ID feature between the two cameras is similar, it is determined that the pedestrian PD captured separately by these cameras is the same person.
[0033] When it is determined that the pedestrians PD captured by the two cameras are the same person, the graph generation processing unit 14 uses the common tracking IDs that have been assigned to these pedestrians PD separately. PD Represents the node N_ID PD are connected via an edge E. In the graph GRH1 shown in FIG. 3, the pedestrians PDp and PDq are determined to be the same person, and the pedestrians PDr and PDs are determined to be the same person. Therefore, the node N_ID PDp and node N_ID PDq are connected via edges E_IDp-q, and node N_ID PDr and node N_ID PDs are connected via edge E_IDr-s. The meaning of edges E_IDp-q and E_IDr-s is “SAME PERSON.”
[0034] In the graph GRH1 shown in FIG. 3, it is also determined that pedestrians PDq and PDr are the same person. This determination is based on the result of a comparison of Re-ID features between multiple image sequences that are different in the interval time bt, performed for a single camera. For this reason, PDq and node N_ID PDr are connected via an edge E_IDq-r, which means "SAME PERSON".
[0035] Fig. 4 is a diagram showing a detailed configuration example of a graph generated by the graph generation processing unit 14. In the graph GRH2 shown in Fig. 4, a node N representing additional information ADD is added to the graph GRH1 shown in Fig. 3. This additional information ADD includes a common tracking ID PD The additional information ADD is the image IM of the pedestrian PD. PD , Appearance characteristics of pedestrian PD AP PD , Pedestrian PD's behavior AC PD , pedestrian PD's face image IMF PD Examples include:
[0036] Pedestrian PD image IM PD was used to extract the Re-ID features. Examples of the appearance features of the pedestrian PD include the color, clothing, and body type of the pedestrian PD. These appearance features are estimated using a pre-trained appearance model. Examples of the actions of the pedestrian PD include "walking," as well as "carrying" and "opening" actions that the pedestrian PD takes with respect to stationary objects such as luggage. These actions are estimated using a pre-trained action model. These actions also include interaction actions such as "conversation" and "handing over" that multiple people take together. The facial image IMF of the pedestrian PD PD IM is an image of a pedestrian PD PD The face image may be an image in which the face portion has been trimmed from the original image, or may be an externally provided face image for tracking processing by the tracking processing unit 16 (described later).
[0037] In the graph GRH2 shown in FIG. 4, there is a node N_IDPDp Three edges E_ID extending from PDp Next, the image IM of the pedestrian PDp PDp Represents the node N_IM PDp(bt1) and N_IM PDp(bt2) and node N_AC which represents the behavior of pedestrian PDp. PDp(bt1) In addition, the node N_ID PDq Three edges E_ID extending from PDq Next, the face image IMF of pedestrian PDq PDq Node N_IMF PDq and the image IM of the pedestrian PDq PDq Represents the node N_IM PDq(bt3) and node N_AC which represents the behavior of pedestrian PDq. PDq(bt3) and are arranged.
[0038] Although detailed explanation is omitted, in the graph GRH2, the node N_ID PDr , N_ID PDs and N_ID PDu For each pedestrian PD, a node N representing additional information ADD is connected to a node N_ID via an edge E. PD In addition, the node N_ID PDs and node N_ID PDu is an edge E_IA that means the interaction action "TALKING" PDs-PDu Also, the node N_ID PDs to edge E_ID PDs Node N_AP connected via PDs(bt6) is the appearance feature AP of pedestrian PDs in section time bt6 PDs Represents.
[0039] 2-2. Graph verification processing section The graph verification processing unit 15 verifies the judgment made by the graph generation processing unit 14, specifically, the judgment made using the representative data of the Re-ID features. According to the judgment using the representative data of the Re-ID features, a certain degree of certainty is obtained that the pedestrian PD captured separately by the two cameras is the same person. However, when the positional relationship between the two cameras and the temporal or spatial positional relationship of the pedestrian PD judged to be the same person are taken into consideration, a contradiction may occur in the judgment results. The graph verification processing unit 15 detects this contradiction. Then, if it is judged that there is a contradiction, it deletes the edge E that means "SAME PERSON". In other words, the two nodes N_ID based on the representative data of the Re-ID features are PD The binding is released.
[0040] FIG. 5-8 is a diagram for explaining an example of the verification process performed by the graph verification processing unit 15. The verification process is performed using the graph GRH. The graphs GRH3-6 shown in FIG. 5-8 each include a node N_ID representing each ID of at least two cameras. CA and an edge E indicating the relationship between these nodes. Also, at least two nodes N_ID CA A common tracking ID PD Represents the node N_ID PD are connected to each other via edge E.
[0041] In the graph GRH3 shown in FIG. 5, the node N_ID CA1 Node N_ID connected to PDp connects to node N_ID via edge E_CA4 CA4 Node N_ID connected to PDq and are connected via an edge E_IDp-q, which means "SAME PERSON". PDp also connects to node N_ID via edge E_CA3 CA3 Node N_ID connected to PDv and are connected via an edge E_IDp-v, which means "SAME PERSON".
[0042] However, when considering the relative positions of the cameras CA1, CA3, and CA4, and the predicted movement direction of the pedestrian PDp based on the section time bt (the direction of the arrows of the edges E_IDp-q and E_IDp-v), it is a contradiction for the edges E_IDp-q and E_IDp-v to exist simultaneously. This is because the graph GRH3 has connections that make it seem as if one pedestrian PDp had split into two pedestrians PDq and PDv. Therefore, in this case, the edges E_IDp-q and E_IDp-v are deleted. In a more advanced example, the node N_ID representing the pedestrian PD who has been determined to be the same person as the pedestrian PDp PD Or, the predicted moving direction of the pedestrian PDp is further taken into consideration. In this case, the inappropriate edge E can be deleted and the appropriate edge E can be left.
[0043] The structure of the graph GRH4 shown in Fig. 6 is basically the same as that of the graph GRH3 shown in Fig. 5. The difference between these graphs is in the predicted movement direction of the pedestrian PDp based on the section time bt (the direction of the arrows of the edges E_IDp-q and E_IDp-v). However, the verification result is the same. This is because the graph GRH3 has a connection as if the two pedestrians PDq and PDv were merged into one pedestrian PDp. Therefore, in this case as well, the edges E_IDp-q and E_IDp-v are deleted.
[0044] In the graph GRH5 shown in FIG. 7, the node N_ID CA4 Node N_ID connected to PDr connects to node N_ID via edge E_CA5 CA5 Node N_ID connected to PDs and is connected via edge E_IDr-s, which means "SAME PERSON". PDr Also, the node N_ID PDv and are connected via an edge E_IDr-v, which means "SAME PERSON".
[0045] Unlike the results of the verification described with reference to Figures 5 and 6, the result of the verification based on the graph GRH5 shown in Figure 7 is "not contradictory". CA5 and node N_ID CA3 This is because they are connected via edge E_CA3-5, which means "OVERLAPPED." Therefore, it is as if one pedestrian PDr is split into two pedestrians PDs and PDv, but the verification result shows that there is no contradiction in the fact that pedestrians PDs and PDv are the same person.
[0046] In the graph GRH6 shown in FIG. 8, the node N_ID PDp(bt1) and node N_ID PDq(bt3) In the graph GRH6, the nodes N_ID and N_IDp-q are connected via an edge E_IDp-q, which means "SAME PERSON". In the graph GRH6, the nodes N_ID and N_IDp-q are connected via an edge E_CA3. CA3 Node N_ID connected to PDu(bt7) and node N_ID via edge E_CA7 CA7 Node N_ID connected to PDw(bt8) are connected via an edge E_IDu-w, which means "SAME PERSON".
[0047] In graph GRH6, we also have node N_ID PDq(bt3) and node N_ID PDw(bt8) However, there is no relationship between the positions of the cameras CA4 and CA7 (i.e., the node N_ID CA4 and node N_ID CA7 Considering the fact that there is no edge E between the cameras CA4 and CA7, and the distance between the cameras CA4 and CA7, it is determined that there is a contradiction in the existence of the edge E_IDu-w. In this case, the edge E_IDu-w is deleted.
[0048] 2-3.Tracking processing section The tracking processing unit 16 tracks the target object using the graph GPH stored in the graph DB 17. Figures 9 and 10 are diagrams for explaining an example of the tracking process performed by the tracking processing unit 16. In the example shown in Figure 9, an image IM of the target object is TGT The graph DB 17 is referenced with the query Q (input information). TGT is selected from the frame of the pedestrian PD used to extract the Re-ID features. TGT is the face image IMF of the target TGT In this case, the face image IMF TGT may be cropped from the frame of pedestrian PD used to extract the Re-ID features, or may be provided from outside the tracking system.
[0049] In the example shown in FIG. TGT and its image IM PD The tracking data TRC of the pedestrian PD that matches the image IM is output as a search result R. TGT Matching images IM PD Represents the node N_IM PD and each node N_IM PD Node N_ID connected to via edge E PD and each node N_ID PD Node N_ID connected to via edge E CA In another example, each node N_IM PD Additional information ADD (i.e., appearance feature AP PD , behavioral AC PD A set of nodes N representing the above-mentioned nodes, such as the nodes 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158
[0050] In the example shown in FIG. 10, the date and time DT, the location AR, and the image IM TGT In this example, the graph DB 17 is referenced with the query Q (input information). TGT and its image IM PDThe tracking data TRC of the pedestrian PD that matches the query Q is output as a search result R. However, in the example shown in Fig. 10, since the date and time DT and the location AR are included in the query Q, the tracking data TRC with limited date and time and location is output.
[0051] 3.Effects According to the embodiment described above, two common tracking IDs are generated based on the video data VD_CA obtained from at least two cameras. PD Represents the node N_ID PD A graph GRH is generated in which the images are connected by edges E. Therefore, by performing tracking processing using this graph GRH, it becomes possible to track the same person (same moving object) that appears in these video data VD_CA. This effect is also expected to be obtained when multiple cameras are placed over a wide area.
[0052] Also, according to an embodiment, two common tracking IDs PD Represents the node N_ID PD The connections are verified. If the results of the verification performed during the connection are found to be inconsistent, the connection is removed. This type of verification is expected to ensure the reliability of the graph GRH.
[0053] Furthermore, the generation of the graph GRH and the verification of the association can be performed in parallel with the acquisition of the video data VD_CA. This makes it possible to track the same person who appears in the video data VD_CA in real time with high accuracy. [Explanation of symbols]
[0054] 10 Management server, 11 Processor, 12 Storage device, 14 Graph generation processing unit, 15 Graph verification processing unit, 16 Tracking processing unit, 17 Graph DB, CA, CA1-CA6, CAi, CAj, CAn Camera, FR Frame, PD, PDp-PDs, PDu-PDz Pedestrian, VD, VD_CA1-CA6, VD_CAn Video data, AC PD behavior, ap PDAppearance features, IM PD ,IM PDp ,IM PDq ,IM PDx -IM PDz ,IM TGT Image: IMF PD ,IMF PDq ,IMF TGT Face image, E,E_CA1,E_CA3-CA7,E_CA1-3,E_CA1-4,E_CA3-5,E_CA4-5,E_CA5-6,E_CA3-7,E_ID PDp -E_ID PDs ,E_ID PDu ,E_IDp-q,E_IDq-r,E_IDr-s,E_IDq-w,E_IDu-w,E_IA PDs - PDu Edge, N, N_AC PDp -N_AP PDs ,N_ID CA ,N_ID CA1 -N_ID CA7 ,N_ID PD ,N_ID PDp -N_ID PDs ,N_ID PDu -N_ID PDw ,N_IMF PDq Node, ADD additional information, GRH, GRH1-GRH6 graph
Claims
1. a storage device in which each image data captured by at least two cameras is stored; a processor configured to perform data processing based on each of the video data; Equipped with The processor, in the data processing, A graph is generated that includes at least two nodes and at least one edge that indicates a relationship between the at least two nodes; storing said graph in a database in said storage device; In the graph, a node representing a single camera included in the at least two cameras and a node representing a tracking identification number assigned to a moving object captured in video data acquired by the single camera are connected via at least one edge; The tracking identification number includes a common tracking identification number that is assigned to the same moving object captured in the video data acquired by the single camera, At least two of the single cameras are connected to each other through at least one edge indicating a relationship between the single cameras, among the nodes representing the single cameras, the nodes having the relationship; At least two of the nodes representing the common tracking identification numbers are connected to each other through at least one edge that indicates that at least two moving objects captured in each of the image data acquired by the at least two single cameras are the same moving object. A moving object tracking system comprising:
2. In the graph, a node representing the common tracking identification number and a node representing additional information about the same moving object to which the common tracking identification number is assigned are connected via at least one edge; The additional information includes at least one of an image of the same moving object to which the common tracking identification number is assigned, an appearance feature of the same moving object, a behavior of the same moving object, and a face image of the person when the same moving object is a person.
2. The moving object tracking system according to claim 1 .
3. The processor, in generating the graph, determining whether the at least two moving objects are the same moving object based on the feature amounts of the at least two moving objects; If it is determined that the at least two moving objects are the same moving object, the common tracking identification numbers assigned to the moving objects are linked together; When the common tracking identification numbers respectively assigned to the at least two moving objects are linked, the nodes representing the common tracking identification numbers are connected to each other in the graph via at least one edge indicating that the at least two moving objects are the same moving object.
3. A moving object tracking system according to claim 1 or 2.
4. The processor, in generating the graph, further comprises: Verifying the determination based on the feature amounts of the at least two moving objects; If it is determined that there is a contradiction in the determination, the association between the nodes representing the common tracking identification numbers respectively assigned to the at least two moving objects is released.
4. The moving object tracking system according to claim 3.
5. The processor further comprises: The method is configured to perform a tracking process of a tracking target by referring to the graph using a query as an input, The query includes at least one of a date and time, a location, an image of the tracked object, and, if the tracked object is a person, a face image of the person.
3. A moving object tracking system according to claim 1 or 2.
Citation Information
Patent Citations
Pedestrian tracking method and device and storage medium
CN113935358A
Knowledge information processing server system with image recognition system
JP2013088906A
Object identification device and method based on plurality of camera images
JP2019066925A
Three-dimensional position estimation device and program
JP2020122754A
Abnormality detection device and abnormality detection method
JP2022086627A
