Moving object tracking system

Abstract

To prevent tracking from being interrupted when the appearance of a moving object changes during tracking the moving object appearing in each video data acquired by a plurality of cameras.SOLUTION: Processing to generate a graph and processing to search for a tracking target by referring to the graph are performed. In the processing to search for the tracking target, a feature quantity of the tracking target is extracted from an image of the tracking target. A moving object having the feature quantity that is most similar to the tracking target feature quantity is specified from moving object feature quantities extracted from at least two moving object images represented by at least two nodes constituting the graph. A tracking target graph including a node representing a tracking identification number assigned to the identified moving object and at least one node connected to the node representing the tracking identification number via at least one edge is specified.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a system that uses video data acquired by multiple cameras to track a moving object captured in the video data. [Background technology]

[0002] International Publication No. 2022 / 185521 discloses a technique for searching the movement paths of people captured by multiple cameras. This conventional search technique detects people in video data acquired by a camera, and then extracts features of the detected people. This feature information is combined with the time the detected people were photographed and the ID number of the camera that captured the detected people, and then registered in a database.

[0003] When searching for a travel route, the search range (region and time) is set in addition to the feature values ​​of the person to be searched for. Within this set search range, the similarity between the feature values ​​of the person to be searched for and the feature values ​​of people registered in the database is calculated. Then, a person with a similarity value equal to or greater than a threshold value is considered to be the person to be searched for. When searching for a travel route, information on the time when such a person was photographed and information on the location of the camera that took the photograph are output as search results.

[0004] In conventional search techniques, at least some elements of the search results are extracted and arranged in chronological order to generate candidate travel routes. In conventional search techniques, the cost of travel between cameras is further calculated from a graph showing the positions of multiple cameras and the relative positions of these cameras. In a search for a travel route, the candidate travel routes are evaluated using this travel cost. If a candidate travel route that is worth the travel cost is found, this candidate travel route is determined as the travel route of the person being searched for.

[0005] In addition to WO 2022 / 185521, WO 2014 / 132841 and WO 2014 / 045843 can be exemplified as documents showing the technical state of the art in the technical field related to the present disclosure. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2022 / 185521 [Patent Document 2] International Publication No. 2014 / 132841 [Patent Document 3] International Publication No. 2014 / 045843 Summary of the Invention [Problem to be solved by the invention]

[0007] Consider tracking moving objects (people, robots, vehicles, etc.) captured in video data acquired by multiple cameras. In the search technology described in International Publication No. 2022 / 185521, the calculation of similarity to the feature values ​​of the person being searched is performed for all people within a set search range. Therefore, if the similarity threshold is low, the number of search results output increases, making it difficult to generate candidate movement paths. In this regard, the number of search results output can be reduced by increasing the similarity threshold. However, in this case, if the person being searched for changes their appearance, such as taking off their jacket or hat, the similarity may be determined to be low. This may cause the person's movement path to be interrupted, making it difficult to track the person being searched for.

[0008] One object of the present disclosure is to provide a technology that, when tracking a moving object captured in video data acquired by multiple cameras, prevents tracking from being interrupted when the appearance of the moving object changes. [Means for solving the problem]

[0009] The present disclosure is a moving object tracking system having the following features. The tracking system includes a storage device and a processor. The storage device stores video data captured by at least two cameras. The processor is configured to perform the following operations: generate a graph based on the video data, the graph including at least two nodes and at least one edge indicating a relationship between the at least two nodes; and search for the tracked object by referring to the graph, using a query including an image of the tracked object as input. In the graph, a node representing a single camera included in the at least two cameras is connected via at least one edge to a node representing a tracking identification number assigned to a moving object captured in video data acquired by the single camera, and the tracking identification number includes a common tracking identification number assigned to the same moving object captured in video data acquired by the single camera. In the graph, at least one edge indicating a relationship between at least two of the single cameras connects nodes representing these single cameras that have the relationship. In the graph, nodes that represent the at least two common tracking identification numbers and that are recognized as the same moving object are connected via at least one edge that indicates that the at least two moving objects captured in each of the video data are the same moving object. In the graph, a node representing the common tracking identification number and a node representing an image of the same moving object to which the common tracking identification number is assigned are connected via at least one edge. In the process of searching for the tracking target, the processor extracting a feature amount of the tracked object from an image of the tracked object; identifying a moving object having a feature quantity most similar to the feature quantity of the tracked target from feature quantities of at least two moving objects respectively extracted from images of the at least two moving objects represented by at least two nodes constituting the graph; A tracking target graph is identified, which indicates a graph including a node representing the tracking identification number assigned to the identified moving object and at least one node connected to the node representing the tracking identification number via at least one edge. [Effects of the Invention]

[0010] According to the present disclosure, a process is performed to generate a graph consisting of at least two nodes and at least one edge indicating the relationship between the at least two nodes. In this graph, a node representing a single camera is connected to a node representing a tracking identification number assigned to a moving object captured in video data acquired by the single camera via at least one edge. Furthermore, this tracking identification number includes a common tracking identification number assigned to the same moving object captured in video data acquired by the single camera.

[0011] In this graph, nodes representing at least two single cameras that have a relationship are connected to each other via at least one edge indicating the relationship between these single cameras.In this graph, nodes representing at least two common tracking identification numbers that are recognized as the same moving object are connected to each other via at least one edge indicating that at least two moving objects captured in the video data acquired by the at least two single cameras are the same moving object.In this graph, nodes representing common tracking identification numbers are also connected to nodes representing images of the same moving object to which the common tracking identification number is assigned via at least one edge.

[0012] In this way, the graph generation process makes it possible to generate a graph in which nodes representing the tracking identification numbers before and after the appearance of a moving object changes are connected to each other.

[0013] According to the present disclosure, a process for searching for a tracking target is also performed. In this process, a feature of the tracking target is extracted from an image of the tracking target included in the query. Then, from among feature of at least two moving objects extracted from images of these moving objects represented by at least two nodes constituting a graph, a moving object having feature most similar to the feature of the tracking target is identified. Then, a tracking target graph is identified, which indicates a graph including a node representing a tracking identification number assigned to the identified moving object and at least one node connected to the node representing the tracking identification number via at least one edge.

[0014] A moving object having feature quantities most similar to those of a tracking target is likely to be the tracking target. In this regard, the process of searching for a tracking target can identify a moving object having feature quantities most similar to the appearance of the moving object before or after a change in appearance, and identify a tracking target graph including a node representing the tracking identification number of the identified moving object. Therefore, according to the present disclosure, when tracking a moving object captured in video data acquired by multiple cameras, it is possible to prevent tracking from being interrupted when the appearance of the moving object changes. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a tracking system according to an embodiment; [Figure 2] 10A and 10B are diagrams illustrating a person detection process and a person re-identification process performed by a graph generation processing unit. [Figure 3] FIG. 2 is a diagram illustrating an example of a basic configuration of a graph generated by a graph generation processing unit. [Figure 4] FIG. 10 is a diagram illustrating an example of a detailed configuration of a graph generated by a graph generation processing unit. [Figure 5] FIG. 10 is a diagram illustrating an example of processing performed by a search processing unit. [Figure 6] FIG. 2 is a diagram illustrating a root node and a leaf node. [Figure 7]10A and 10B are diagrams illustrating an example of processing performed by a graph rearrangement processing unit. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and the description thereof will be simplified or omitted.

[0017] 1. Overall configuration example FIG. 1 is a diagram illustrating an example of the overall configuration of a tracking system (hereinafter also simply referred to as "system") according to an embodiment. The system according to the embodiment is a system for tracking a moving object moving within a city CT. There is no limitation on the size of the city CT in the present disclosure. A so-called smart city is an example of a large-scale city CT, an underground mall is an example of a medium-scale city CT, and a large building is an example of a small-scale city CT. Examples of moving objects include people, robots, and vehicles. In the embodiment, the moving object is assumed to be a person (pedestrian PD). In FIG. 1, pedestrians PD1-PD3 are depicted as examples of pedestrians PD.

[0018] The system according to the embodiment includes at least two cameras arranged in the city CT. In FIG. 1, cameras CA1-CA6 are depicted as an example of the at least two cameras. Camera CA1 acquires video data VD_CA1. Like camera CA1, cameras CA2-CA6 acquire video data VD_CA2-VD_CA6, respectively. Video data VD_CAn of any one camera CAn (n is a natural number) arranged in the city CT is transmitted to the management server 10 via a communication network. The communication network is not particularly limited, and wired and wireless networks can be used.

[0019] The management server 10 is a computer including at least one processor 11, at least one storage device 12, and at least one interface 13. The processor 11 performs various types of data processing. The processor 11 includes a CPU (Central Processing Unit). The storage device 12 stores various types of data required for data processing. Examples of the storage device 12 include an HDD, an SSD, a volatile memory, and a non-volatile memory. The interface 13 receives various types of data from the outside and outputs various types of data to the outside. The various types of data received from the outside by the interface 13 include video data VD_CAn. This video data VD_CAn is stored in the storage device 12. A graph DB (database) 17 is formed in the storage device 12. The graph DB 17 may be formed in an external device that can communicate with the management server 10.

[0020] 2. Management Server Configuration Example Fig. 1 shows an example of the functional configuration of the management server 10. In the example shown in Fig. 1, the management server 10 includes a graph generation processing unit 14, a search processing unit 15, and a graph organization processing unit 16. These functions are realized by the processor 11 executing various programs stored in the storage device 12.

[0021] 2-1.Graph generation processing section The graph generation processing unit 14 performs processing to generate a graph GPH based on the video data VD_CAn. To generate the graph GPH, the graph generation processing unit 14 performs person detection and extraction processing and person re-identification processing. Figure 2 is a diagram explaining the person detection processing and person re-identification processing performed by the graph generation processing unit 14.

[0022] 2 shows video data VD_CAi from camera CAi and video data VD_CAj from camera CAj as an example of video data VD_CAn. The video data VD_CAi and VD_CAj are each separated by a predetermined time interval, and FIG. 2 shows a set of frames FR in a section time bt. Each frame FR of the video data VD_CAi is, for example, CAi As with each frame FR of the video data VD_CAi, each frame FR of the video data VD_CAj also includes the ID of the camera CAj. CAj and timestamp ts.

[0023] In the detection and extraction process, first, a frame FR in which a person is detected is extracted. Next, the detected person is extracted from this extracted frame. In the example shown in FIG. 2, pedestrians PDx and PDy are detected in each frame within section time bt1 (timestamp ts1-ts5) of video data VD_CAi. Furthermore, pedestrian PDz is detected in each frame within section time bt2 (timestamp ts6-ts10) of video data VD_CAj. Bounding boxes surrounding these pedestrians PD are added to the positions where pedestrians PDx, PDy, and PDz are detected. By trimming this bounding box, an image IM of pedestrians PDx, PDy, and PDz is obtained. PDx , I.M. PDy and IM PDz The image IM of the pedestrian PD is extracted. PDx and IM PDy For example, the ID of the camera CAi CAi The data of the timestamp ts and the coordinate CD in the frame FR of the extracted image PD and data of

[0024] In the re-identification process, each image IM extracted in the detection and extraction process is PDFrom each image, a feature for re-identification processing (hereinafter also referred to as "Re-ID feature") is extracted. The Re-ID feature is extracted using a Re-ID model based on machine learning. Note that the technique for extracting Re-ID feature using the Re-ID model is well known, and the method is not particularly limited. PD Once the Re-ID features are extracted, it is determined whether the people included in the image sequences are the same person by comparing the Re-ID features.

[0025] In the example shown in FIG. 2, each image IM PD Since the Re-ID feature values ​​of the people extracted from the image sequence in the section time bt1 are similar, the pedestrians PDx and PDy included in the image sequence in the section time bt1 are determined to be the same person. PD Since the Re-ID features of the people extracted from are similar to each other, the pedestrian PDz included in the image sequence in the interval time bt2 is determined to be the same person.

[0026] When the detection process is performed, a tracking ID is assigned to the person captured in the video data VD_CAn. PD Among these, if a person is determined to be the same person through the re-identification process, a tracking ID common to this person is assigned. PD (hereinafter referred to as "Common Tracking ID PD ") will be assigned a common tracking ID. PD is also called a universally unique ID (UUID). In the example shown in FIG. 2, for pedestrians PDx, PDy, and PDz, a common tracking ID PDx(bt1) , ID PDy(bt1) and ID PDz(bt2) Each common tracking ID is assigned. PD is a combination of data for a section time bt and data representative of the Re-ID feature data extracted from the image sequence for this section time bt. Note that there are no particular limitations on the example of selecting the representative data for the Re-ID feature, and any method can be used.

[0027] Common Tracking IDPD is generated for each interval time bt. Therefore, if the same pedestrian PD continues to be captured by one camera (single camera), the common tracking ID assigned to this pedestrian PD PD can occur separately for as many interval times bt as there are interval times bt. Therefore, in the re-identification process, the Re-ID feature values ​​may be compared between multiple image sequences that differ in the interval time bt. For example, the Re-ID feature values ​​are compared between two image sequences of interval times bt that have close timestamps ts. If the Re-ID feature values ​​are similar between multiple image sequences, it is determined that the pedestrians PD included in these image sequences are the same person, and the common tracking IDs that were assigned separately to each pedestrian PD are used. PD may be combined into one.

[0028] The graph generation processing unit 14 uses the common tracking ID assigned to the pedestrian PD by the above-described re-identification process. PD and the IDs of at least two cameras placed in the city CT. The generated graph GRH is stored in the graph DB 17. As already explained, the graph GRH is expressed using nodes (vertices, nodal points) and edges (sides, branches) in graph theory. FIG. 3 is a diagram showing an example of the basic configuration of the graph GRH generated by the graph generation processing unit 14. In FIG. 3, a node N_ID representing each ID of the cameras CA1 to CA6 (however, CA2 is omitted) shown in FIG. 1 is shown. CA1 -N_ID CA6 and an edge E indicating the relationship between these nodes.

[0029] Here, the installation position of the camera CA1 is close to that of the camera CA3. Therefore, there is a relationship between these cameras. Therefore, in the graph GRH1 shown in Figure 3, there is a node N_ID representing the ID of the camera CA1. CA1 and a node N_ID representing the ID of camera CA3. CA3are connected via edge E_CA1-3. The meaning of this edge E_CA1-3 is "NEARBY." The "NEARBY" relationship also exists between camera CA1 and camera CA4, between camera CA4 and camera CA5, and between camera CA5 and camera CA6. Therefore, the nodes N representing the IDs of two related cameras CA are connected by one edge E (edge ​​E_CA1-4, edge E_CA4-5, and edge E_CA5-6).

[0030] Another example of the relationship between two cameras CA is that the imaging ranges of these cameras overlap in part or in whole. Here, part of the imaging range of camera CA3 overlaps with that of camera CA5. Therefore, in the graph GRH1 shown in FIG. 3, the node N_ID representing the ID of camera CA3 CA3 and a node N_ID representing the ID of the camera CA5. CA5 are connected via an edge E_CA3-5. The meaning of this edge E_CA3-5 is "OVERLAPPED."

[0031] A common tracking ID assigned to pedestrian PDs PD Represents the node N_ID PD This common tracking ID PD Node N_ID representing the camera CAn that acquired the video data VD_CAn that was the basis for the assignment CAn PDp, PDq, PDr, PDs, and PDu are connected to each other via at least one edge E. Figure 3 shows the common tracking IDs assigned to pedestrians PDp, PDq, PDr, PDs, and PDu. PD Represents the node N_ID PDp , N_ID PDq , N_ID PDr , N_ID PDs and N_ID PDu is drawn. Node N_ID PDp connects to node N_ID via edge E_CA1 CA1 Node N_ID PDq and N_ID PDr connects to node N_ID via two edges E_CA4 CA4Node N_ID PDs connects to node N_ID via edge E_CA5 CA5 Node N_ID PDu connects to node N_ID via edge E_CA3 CA3 is tied to.

[0032] As mentioned above, the common tracking ID PD The re-identification process combines data from the interval time bt and representative data of the Re-ID features from the image sequences. In the embodiment, the representative data of the Re-ID features is used to perform a re-identification process for the person captured by at least two cameras. This re-identification process is the same as the comparison of the Re-ID features between multiple different image sequences in the interval time bt. However, while the comparison of the Re-ID features between multiple image sequences targets one camera, the comparison of the representative data of the Re-ID features targets two cameras. If the representative data of the Re-ID features between the two cameras is similar, the pedestrian PD captured separately by these cameras is determined to be the same person.

[0033] When it is determined that the pedestrians PD captured by the two cameras are the same person, the graph generation processing unit 14 uses the common tracking IDs assigned to the pedestrians PD separately. PD Represents the node N_ID PD are connected via edge E. In the graph GRH1 shown in FIG. 3, pedestrians PDp and PDq are determined to be the same person, and pedestrians PDr and PDs are determined to be the same person. Therefore, the node N_ID PDp and node N_ID PDq are connected via edges E_IDp-q, and node N_ID PDr and node N_ID PDs are connected via edge E_IDr-s. The meaning of edges E_IDp-q and E_IDr-s is "SAME PERSON."

[0034] In the graph GRH1 shown in Figure 3, it is also determined that pedestrians PDq and PDr are the same person. This determination is based on the result of comparing Re-ID features between multiple image sequences that are different in the interval time bt, performed using a single camera. For this reason, the node N_ID PDq and node N_ID PDr are connected via an edge E_IDq-r which means "SAME PERSON".

[0035] Even if the comparison of Re-ID features determines that the pedestrians PD captured by the two cameras are not the same person, a common tracking ID assigned to each pedestrian PD is generated when certain movement conditions are met. PD Represents the node N_ID PD may be connected via an edge E_ID meaning “SAME PERSON.” The predetermined movement conditions include, for example, the following conditions (i) to (iii). (i) The similarity of the Re-ID feature is equal to or greater than a reference value. (ii) Two node N_IDs PD Each additional information ADD (image IM PD ) the interval between timestamps ts is within a specified time (iii) Node N_ID of the two points in condition (i) PD The distance between the installation positions of the two cameras CA connected to each other is within a specified distance.

[0036] 4 is a diagram showing a detailed configuration example of a graph generated by the graph generation processing unit 14. In the graph GRH2 shown in FIG. 4, a node N representing additional information ADD is added to the graph GRH1 shown in FIG. 3. This additional information ADD includes a common tracking ID PD The additional information ADD is the image IM of the pedestrian PD. PD , Pedestrian PD appearance characteristics AP PD , Pedestrian PD behavior AC PD , pedestrian PD's face image IMF PD Examples include:

[0037] Pedestrian PD Image IM PD are used to extract Re-ID features. Examples of appearance features of pedestrian PD include the color, clothing, and body shape of the pedestrian PD. These appearance features are estimated using a pre-trained appearance model. Examples of pedestrian PD behavior include "walking," as well as "carrying" and "opening" actions that pedestrian PD take with respect to stationary objects such as luggage. These actions are estimated using a pre-trained behavior model. These actions also include interaction actions such as "conversation" and "handing over" that multiple people take together. The facial image IMF of pedestrian PD PD is a pedestrian PD image IM PD The face image may be one in which the face portion has been trimmed from the original image, or may be a face image provided from outside in a search process (described later) by the search processing unit 15.

[0038] The graph GRH2 shown in Figure 4 has a node N_ID PDp Three edges E_ID extend from PDp Ahead, an image of a pedestrian PDp is displayed. PDp Node N_IM representing PDp(bt1) and N_IM PDp(bt2) and node N_AC representing the behavior of pedestrian PDp. PDp(bt1) Also, the node N_ID PDq Three edges E_ID extend from PDq Next, the face image IMF of pedestrian PDq PDq Node N_IMF representing PDq and the image IM of the pedestrian PDq PDq Node N_IM representing PDq(bt3) and node N_AC representing the behavior of pedestrian PDq. PDq(bt3) and are arranged.

[0039] Although detailed explanation is omitted, in the graph GRH2, the node N_ID PDr , N_ID PDs and N_ID PDu For example, the node N representing the additional information ADD for each pedestrian PD is connected to the node N_ID via the edge E. PDThe node N_ID is connected to the node PDs and node N_ID PDu is the edge E_IA, which means the interaction action "TALKING". PDs-PDu Also, the node N_ID PDs Edge E_ID PDs Node N_AP connected via PDs(bt6) is the appearance feature AP of pedestrian PDs in section time bt6. PDs Represents.

[0040] 2-2. Search processing section The search processing unit 15 performs a process of searching for the tracking target using the graph GPH stored in the graph DB 17. Fig. 5 is a diagram for explaining an example of the search process performed by the search processing unit 15. In the example shown in Fig. 5, an image IM of the tracking target is TGT The graph DB 17 is referenced using the query Q (input information). TGT is selected from the frame of the pedestrian PD used to extract the Re-ID features. TGT may be provided externally to the tracking system. TGT is the face image IMF of the tracked object. TGT Facial image IMF TGT is, for example, cropped from the frame of the pedestrian PD used to extract the Re-ID features. TGT may be provided externally to the tracking system.

[0041] In the example shown in FIG. TGT The image IM of the pedestrian PD that has the Re-ID feature most similar to the Re-ID feature extracted from PD In this search, the image IM TGT Re-ID features extracted from the common tracking ID PD Represents the node N_ID PD Node N_IM connected to PD The image IM of the pedestrian PD is represented by PDThe Re-ID feature values ​​extracted from the image IM of the pedestrian PD are compared. PD The Re-ID features extracted from, for example, the common tracking ID PD Representative data of the Re-ID features selected during the assignment is used.

[0042] Figure 5 shows the common tracking IDs of the pedestrian PDA, PDB, PDC, and PDD. PD Represents the node N_ID PD and each node N_ID PD Edge E_ID PD These pedestrian images are linked via IM PD Node N_IM representing PD The graph of the pedestrian PDB is a part of the graph GRH2 explained in Figure 4.

[0043] In the example shown in FIG. 5, the image IM of the pedestrians PDs PDs Therefore, in this example, the pedestrian PDs is identified as the person most likely to be the tracking target. When the pedestrian PDs is identified as the person most likely to be the tracking target, the common tracking ID assigned to the pedestrian PDs is PD Represents the node N_ID PDs Node N_ID is connected to the edge E_ID, which means "SAME PERSON". PD Group (Pedestrian PDB node N_ID PD The nodes in the cluster are identified.

[0044] In the example shown in Figure 5, node N_ID PDs is the node N_ID PDr and node N_ID via edge E_IDr-s. PDr is the node N_ID PDq and are connected via edge E_IDq-r. Also, node N_ID PDq is the node N_ID PDp and node N_ID via edge E_IDp-q. PDp is the node N_IDPDv and node N_ID via edge E_IDp-v. PDv is the node N_ID PDw and are connected via an edge E_IDv-w. Both of these edges E_ID mean "SAME PERSON."

[0045] Therefore, if the image IM of the pedestrian PDp PDp The Re-ID features extracted from the image IM TGT Even if the Re-ID feature extracted from the pedestrian PDs is not similar to the Re-ID feature extracted from the pedestrian PDs, the pedestrian PDs is identified as the person most likely to be the tracking target, and the tracking target graph (tracking target graph) GRH is obtained as the search result R(TRC). TGT It is possible to obtain

[0046] In the example shown in FIG. 5, the image IM to be tracked TGT The graph DB 17 is referenced using the query Q (input information). TGT In addition, information such as date, time, and location may be added to the query Q. In the example shown in FIG. 5, the common tracking ID of the tracking target is PD Represents the node N_ID PD and the image IM of the tracked object PD Node N_IM representing PD and the graph GRH containing TGT is output as the search result R(TRC). However, the node N_ID representing the camera CAn described in FIG. CA and the image IM explained in Fig. 4. PD The node N representing additional information ADD other than TGT may be included in

[0047] Another example of the tracking target search process is the common tracking ID PD Represents the node N_ID PD The search is narrowed down by node N_ID. PD By narrowing down the number of nodes, it is expected that the processing load of the search by the processor 11 will be reduced. PDThe narrowing down of the search results is performed according to predetermined narrowing down conditions, which include, for example, at least one of the following conditions (i) and (ii): (i) Node N_ID corresponding to the root node or leaf node PD Being (ii) Node N_ID with a connection degree equal to or greater than a predetermined degree PD Being

[0048] Regarding condition (i), Fig. 6 is a diagram for explaining the root node and the leaf node. PDp , N_ID PDq , N_ID PDr , N_ID PDs , N_ID PDv , and N_ID PDw This graph is a part of the pedestrian PDB graph explained in Figure 5. However, in the example shown in Figure 6, two adjacent nodes N_ID PD The direction of the arrow indicates the predicted movement direction of the pedestrian PDB based on the section time bt (or timestamp ts).

[0049] The root node is the node N_ID in the pedestrian PDB graph. PD (i.e., node N_ID in the Pedestrian PDB PD The node N_ID corresponding to the "root" of the group of nodes PD In the example shown in FIG. PDp The root node is, for example, the oldest node N_ID PD Typically, the root node is the node N_ID PD Node N_IM connected to PD The node N_ID that has the oldest data of timestamp ts (or data of interval time bt) PD If the query Q contains date and time information, the oldest node N_ID in this date and time range is PD is the root node.

[0050] The leaf nodes are the nodes N_ID in the pedestrian PDB graph. PD (i.e., node N_ID in the Pedestrian PDB PD The node N_ID corresponds to the "leaf" of the node group PD The leaf node is, for example, the newest node N_ID PD Typically, the leaf nodes are nodes N_ID PD Node N_IM connected to PD The node N_ID that has the latest data of timestamp ts (or data of interval time bt) PD The leaf node is the newest node N_ID PD Node N_ID excluding the root node PD Among the nodes in the group, node N_ID PD Node N_ID located at the end of PD Therefore, in the example shown in FIG. PDs and N_ID PDr corresponds to the leaf node.

[0051] Regarding condition (ii), node N_ID PD The "connection degree" of the node group is the node N_ID PD The total number of nodes that make up the node group. PD A large total number of nodes means a high degree of connection. PD By focusing only on the node N_ID, it is expected that the processing load of the search by the processor 11 will be reduced. PD This means that only the node N_ID with the lowest connection degree is considered. PD This is because it can be excluded from the search as noise data.

[0052] 2-3.Graph sorting processing unit The graph rearrangement processing unit 16 performs a process of rearranging the graph GPH generated by the graph generation processing unit 14. Specifically, the graph rearrangement processing unit 16 rearranges the connections of the edge E_ID that means "same person (SAME PERSON)." As already explained, the edge E_ID that means "same person (SAME PERSON)" is the common tracking ID of the pedestrian PD that has similar Re-ID feature values. PD Represents the node N_ID PD However, the node N_ID connected by this edge E_ID PD does not take into account time information. Therefore, the graph (tracking target graph) GRH TGT Although it is possible to roughly track the target, it is difficult to grasp the direction of movement of the target.

[0053] Therefore, the edge E_ID, which means "SAME PERSON", is reconnected. This reconnection of the edge E_ID is performed periodically independently of the generation process of the graph GRH. The reconnection of the edge E_ID is performed by the node N_ID. PD Node N_IM connected to PD This is performed based on the data of the timestamp ts (or the data of the interval time bt) held by the user.

[0054] FIG. 7 is a diagram illustrating an example of processing performed by the graph reorganization processing unit 16. FIG. 7 illustrates the graph of the pedestrian PDB described in FIG. 6. The upper part of FIG. 7 is an example of a graph before reorganization processing (before reconnection). As can be seen from the upper part of FIG. 7, before reorganization processing, the common tracking ID PD Represents the node N_ID PDp Branches to node N_ID PDq and N_ID PDv Such a connection is made by the node N_ID PDp and node N_ID PDq Between and node N_ID PDp and node N_ID PDv The Re-ID features are determined to be similar between nodes N_ID PDq and node N_ID PDrThis can occur when it is determined that the features between

[0055] The bottom part of Figure 7 is an example of a graph after the sorting process (after reconnection). As can be seen from the bottom part of Figure 7, the sorting process results in the common tracking IDs that are linked as belonging to the same person. PD Therefore, in the search process using the graph GRH after the sorting process, the edge E_ID of the node group of the node N_ID representing the target can be connected in chronological order. TGT This makes it possible to output the search result R(TRC). This contributes to improving the usability of the search result R(TRC). [Explanation of symbols]

[0056] 10 Management server, 11 Processor, 12 Storage device, 14 Graph generation processing unit, 15 Search processing unit, 16 Graph organization processing unit, 17 Graph DB, CA, CA1-CA6, CAi, CAj, CAn Camera, FR Frame, PD, PDp-PDs, PDu-PDz Pedestrian, VD, VD_CA1-CA6, VD_CAn Video data, AC PD behavior, AP PD Appearance features, IM PD ,IM PDp ,IM PDq ,IM PDx -IM PDz ,IM TGT Image, IMF PD ,IMF PDq ,IMF TGT Face image, E,E_CA1,E_CA3-CA7,E_CA1-3,E_CA1-4,E_CA3-5,E_CA4-5,E_CA5-6,E_CA3-7,E_ID PDp -E_ID PDs ,E_ID PDu ,E_IDp-q,E_IDq-r,E_IDr-s,E_IDp-v,E_IDv-w,E_IDv-q,E_IDr-w,E_IDw-s,E_IA PDs - PDu Edge, N, N_AC PDp -N_AP PDs ,N_IDCA ,N_ID CA1 -N_ID CA7 ,N_ID PD ,N_ID PDp -N_ID PDs ,N_ID PDu -N_ID PDw ,N_IMF PDq Node, ADD additional information, GRH, GRH1, GRH2, GRH TGT graph

Claims

1. a storage device in which video data captured by at least two cameras is stored; a processor; Equipped with The processor: A process of generating a graph including at least two nodes and at least one edge indicating a relationship between the at least two nodes based on the video data; a process of searching for the tracked object by referring to the graph, the query including an image of the tracked object being input; configured to: In the graph, a node representing a single camera included in the at least two cameras and a node representing a tracking identification number assigned to a moving object captured in video data acquired by the single camera are connected via at least one edge; the tracking identification number includes a common tracking identification number assigned to the same moving object captured in the video data acquired by the single camera, At least one edge indicating a relationship between the at least two single cameras connects nodes having the relationship among the nodes representing these single cameras, and nodes that are recognized as the same moving object among the nodes that represent the at least two common tracking identification numbers are connected to each other via at least one edge that indicates that the at least two moving objects reflected in each of the video data are the same moving object; a node representing the common tracking identification number and a node representing an image of the same moving object to which the common tracking identification number is assigned are connected via at least one edge; In the process of searching for the tracking target, the processor extracting a feature amount of the tracked object from an image of the tracked object; identifying a moving object having a feature quantity most similar to the feature quantity of the tracked target from feature quantities of the at least two moving objects respectively extracted from images of the at least two moving objects represented by at least two nodes constituting the graph; Identifying a tracking target graph that indicates a graph including a node representing the tracking identification number assigned to the identified moving object and at least one node connected to the node representing the tracking identification number via at least one edge. A moving object tracking system comprising:

2. In the process of searching for the tracking target, the processor before extracting feature quantities of the at least two moving objects from the images of the at least two moving objects, selecting at least two nodes from the at least two nodes constituting the graph in accordance with a predetermined narrowing-down condition; From the images of the at least two moving objects represented by the at least two selected nodes, feature amounts of the at least two moving objects are extracted, respectively.

2. The moving object tracking system according to claim 1.

3. In the process of generating the graph, the processor determining whether the at least two moving objects are the same moving object based on feature amounts of the moving objects extracted from images of the moving objects connected via at least one edge to a node representing a tracking identification number assigned to each of the moving objects; If it is determined that the at least two moving objects are similar, the nodes representing the common tracking identification numbers assigned to these moving objects are connected via at least one edge indicating that the at least two moving objects are the same moving object.

2. The moving object tracking system according to claim 1.

4. In the process of generating the graph, the processor further if it is determined that the at least two moving objects are not similar in the determination based on the feature amounts of the at least two moving objects, determining whether a predetermined movement condition is satisfied for the at least two moving objects; When it is determined that the movement condition is satisfied for the at least two moving objects, the nodes representing the common tracking identification numbers assigned to these moving objects are connected via at least one edge indicating that the at least two moving objects are the same moving object.

4. The moving object tracking system according to claim 3.

5. The processor further comprises: The graph generating process is configured to perform a process of arranging the graph generated by the process of generating the graph, In the process of organizing the graph, the processor extracting a node group in which the number of connections between nodes representing the common tracking identification numbers assigned to the at least two moving objects is at least three; At least one edge connecting the nodes constituting the node group is reconnected based on timestamp data for images of at least three moving objects each connected to the nodes constituting the node group via at least one edge.

5. A moving object tracking system according to claim 3 or 4.

Images