Tracking system for moving object
The tracking system generates a graph connecting camera nodes and tracking IDs to ensure continuous tracking of moving objects by identifying similar features across multiple cameras, addressing appearance changes and maintaining tracking continuity.
Patent Information
- Application Number
- JP2023083396
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2043-05-19
AI Technical Summary
Existing tracking systems struggle to maintain continuous tracking of a moving object when its appearance changes, such as when a person removes a jacket or hat, leading to interrupted tracking due to low similarity thresholds or excessive search results with high thresholds.
A tracking system that generates a graph connecting nodes representing cameras and tracking identification numbers for the same object across multiple cameras, using feature extraction and similarity comparison to identify the most similar moving objects, ensuring continuous tracking even with appearance changes.
The system effectively prevents tracking interruption by maintaining connections between nodes representing the same object before and after appearance changes, allowing for accurate tracking of moving objects across multiple cameras.
Smart Images

Figure 2024166951000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a system that uses video data acquired by multiple cameras to track a moving object captured in the video data. [Background technology]
[0002] WO 2022 / 185521 discloses a technique for searching the movement path of a person captured by multiple cameras. In this conventional search technique, a person captured in video data acquired by a certain camera is detected, and then the feature amount of the detected person is extracted. The information on the feature amount is registered in a database in combination with the time when the detected person was captured and the ID number of the camera that captured the detected person.
[0003] When searching for a travel route, in addition to the feature values of the person to be searched for, a search range (area and time) is set. In this set search range, the similarity between the feature values of the person to be searched for and the feature values of people registered in the database is calculated. Then, a person having a similarity value equal to or greater than a threshold value is a person who is highly likely to be the person to be searched for. When searching for a travel route, information on the time when such a person was photographed and information on the location of the camera that took the photograph are output as search results.
[0004] In the conventional search technology, at least some elements of the search results are extracted and arranged in chronological order to generate candidate movement routes. In the conventional search technology, the movement cost between the cameras is further calculated from a graph showing the positions of the multiple cameras and the positional relationships between these cameras. In the search for a movement route, the candidate movement routes are evaluated using this movement cost. When a candidate movement route that is worth the movement cost is found, this candidate is determined as the movement route of the person to be searched for.
[0005] In addition to WO 2022 / 185521, WO 2014 / 132841 and WO 2014 / 045843 can be exemplified as documents showing the technical state of the technical field related to the present disclosure. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] International Publication No. 2022 / 185521 [Patent Document 2] International Publication No. 2014 / 132841 [Patent Document 3] International Publication No. 2014 / 045843 Summary of the Invention [Problem to be solved by the invention]
[0007] Consider tracking moving objects (people, robots, vehicles, etc.) captured in video data acquired by multiple cameras. In the search technology described in International Publication No. 2022 / 185521, the calculation of similarity with the feature amount of the search target person is performed for all people in the set search range. Therefore, if the similarity threshold is low, the number of search results output increases and it becomes difficult to generate candidates for the movement path. In this regard, the number of search results output can be reduced by increasing the similarity threshold. However, in this case, if the appearance of the search target person changes, such as taking off a jacket or a hat, the similarity may be determined to be low. In that case, the movement path of the search target person is interrupted, making it difficult to track the search target person.
[0008] One objective of the present disclosure is to provide a technology that, when tracking a moving object captured in video data acquired by multiple cameras, prevents tracking from being interrupted when the appearance of the moving object changes. [Means for solving the problem]
[0009] The present disclosure is a moving object tracking system having the following features. The tracking system includes a storage device and a processor. The storage device stores image data captured by at least two cameras. The processor is configured to perform a process of generating a graph based on the image data, the graph including at least two nodes and at least one edge indicating a relationship between the at least two nodes, and a process of searching for the tracking target by referring to the graph using a query including an image of the tracking target as an input. In the graph, a node representing a single camera included in the at least two cameras is connected via at least one edge to a node representing a tracking identification number given to a moving object captured in video data acquired by the single camera. The tracking identification number includes a common tracking identification number given to the same moving object captured in video data acquired by the single camera. In the graph, at least two of the single cameras have a relationship between them, and among the nodes representing these single cameras, the nodes having the relationship are connected to each other via at least one edge indicating the relationship between the single cameras. In the graph, nodes that represent the at least two common tracking identification numbers and that are recognized as being the same moving object are connected to each other via at least one edge that indicates that the at least two moving objects captured in each of the video data are the same moving object. In the graph, a node representing the common tracking identification number and a node representing an image of the same moving object to which the common tracking identification number is assigned are connected via at least one edge. In the process of searching for the tracking target, the processor Extracting a feature amount of the tracked object from an image of the tracked object; Identifying a moving object having a feature quantity most similar to the feature quantity of the tracked target from feature quantities of at least two moving objects respectively extracted from images of the at least two moving objects represented by at least two nodes constituting the graph; A tracking target graph is identified, which indicates a graph including a node representing the tracking identification number assigned to the identified moving object and at least one node connected to the node representing the tracking identification number via at least one edge. Effect of the Invention
[0010] According to the present disclosure, a process is performed to generate a graph consisting of at least two nodes and at least one edge indicating a relationship between the at least two nodes. In this graph, a node representing a single camera and a node representing a tracking identification number given to a moving object captured in video data acquired by the single camera are connected via at least one edge. In addition, the tracking identification number includes a common tracking identification number given to the same moving object captured in video data acquired by the single camera.
[0011] In this graph, the nodes representing at least two single cameras are connected to each other through at least one edge indicating the relationship between the at least two single cameras. In this graph, the nodes representing at least two common tracking identification numbers that are recognized as the same moving object are connected to each other through at least one edge indicating that at least two moving objects captured in each video data acquired by at least two single cameras are the same moving object. In this graph, the nodes representing the common tracking identification numbers are further connected to the nodes representing the images of the same moving object to which the common tracking identification number is assigned through at least one edge.
[0012] In this way, the graph generation process makes it possible to generate a graph in which nodes representing the tracking identification numbers before and after a change in the appearance of a moving object are connected to each other.
[0013] According to the present disclosure, a process of searching for a tracking target is also performed. In this process, a feature of the tracking target is extracted from an image of the tracking target included in the query. A moving object having a feature most similar to the feature of the tracking target is identified from among the feature of at least two moving objects represented by at least two nodes constituting a graph, the feature of the moving object being extracted from the images of the moving objects. A tracking target graph is also identified, the tracking target graph indicating a graph including a node representing a tracking identification number given to the identified moving object and at least one node connected to the node representing the tracking identification number via at least one edge.
[0014] A moving object having a feature quantity most similar to that of a tracking target is highly likely to be the tracking target. In this regard, the process of searching for a tracking target can identify a moving object having a feature quantity most similar to the appearance of the moving object before or after a change in appearance, and identify a tracking target graph including a node representing the tracking identification number of the identified moving object. Therefore, according to the present disclosure, when tracking a moving object captured in each video data acquired by multiple cameras, it is possible to prevent tracking from being interrupted when the appearance of the moving object changes. [Brief description of the drawings]
[0015] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a tracking system according to an embodiment. [Diagram 2] 11A and 11B are diagrams illustrating a person detection process and a person re-identification process performed by a graph generation processing unit. [Diagram 3] 13 is a diagram illustrating an example of a basic configuration of a graph generated by a graph generation processing unit. FIG. [Figure 4] 13 is a diagram illustrating an example of a detailed configuration of a graph generated by a graph generation processing unit; FIG. [Diagram 5] 11 is a diagram illustrating an example of a process performed by a search processing unit. FIG. [Figure 6] FIG. 2 is a diagram for explaining a root node and a leaf node. [Figure 7]11A and 11B are diagrams illustrating an example of processing performed by a graph arrangement processing unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and the description thereof will be simplified or omitted.
[0017] 1. Overall configuration example FIG. 1 is a diagram for explaining an example of the overall configuration of a tracking system (hereinafter, also simply referred to as "system") according to an embodiment. The system according to the embodiment is a system for tracking a moving object moving inside a city CT. There is no limitation on the scale of the city CT in the present disclosure. A so-called smart city is an example of a large-scale city CT, an underground mall is an example of a medium-scale city CT, and a large building is an example of a small-scale city CT. Examples of the moving object include a person, a robot, and a vehicle. In the embodiment, it is assumed that the moving object is a person (pedestrian PD). In FIG. 1, pedestrians PD1-PD3 are depicted as examples of pedestrians PD.
[0018] The system according to the embodiment includes at least two cameras arranged in the city CT. In FIG. 1, cameras CA1-CA6 are illustrated as an example of the at least two cameras. Camera CA1 acquires video data VD_CA1. Like camera CA1, cameras CA2-CA6 acquire video data VD_CA2-VD_CA6, respectively. Video data VD_CAn of any one of cameras CAn (n is a natural number) arranged in the city CT is transmitted to the management server 10 via a communication network. The communication network is not particularly limited, and wired and wireless networks can be used.
[0019] The management server 10 is a computer including at least one processor 11, at least one storage device 12, and at least one interface 13. The processor 11 performs various data processing. The processor 11 includes a CPU (Central Processing Unit). The storage device 12 stores various data required for data processing. Examples of the storage device 12 include HDD, SSD, volatile memory, and non-volatile memory. The interface 13 receives various data from the outside and outputs various data to the outside. The various data received by the interface 13 from the outside includes video data VD_CAn. This video data VD_CAn is stored in the storage device 12. A graph DB (database) 17 is formed in the storage device 12. The graph DB 17 may be formed in an external device that can communicate with the management server 10.
[0020] 2. Management Server Configuration Example Fig. 1 shows an example of the functional configuration of the management server 10. In the example shown in Fig. 1, the management server 10 includes a graph generation processing unit 14, a search processing unit 15, and a graph organization processing unit 16. These functions are realized by the processor 11 executing various programs stored in the storage device 12.
[0021] 2-1.Graph generation processing section The graph generation processing unit 14 performs a process of generating the graph GPH based on the video data VD_CAn. In order to generate the graph GPH, the graph generation processing unit 14 performs a person detection and extraction process and a person re-identification process. Fig. 2 is a diagram for explaining the person detection process and the person re-identification process performed by the graph generation processing unit 14.
[0022] 2 illustrates video data VD_CAi from camera CAi and video data VD_CAj from camera CAj as an example of video data VD_CAn. The video data VD_CAi and VD_CAj are each separated by a predetermined time width, and FIG 2 illustrates a set of frames FR in a section time bt. Each frame FR of the video data VD_CAi is, for example, CAi As with each frame FR of the video data VD_CAi, each frame FR of the video data VD_CAj also includes the ID of the camera CAj. CAj and data at timestamp ts.
[0023] In the detection and extraction process, first, a frame FR in which a person is detected is extracted. Next, the detected person is extracted from this extracted frame. In the example shown in FIG. 2, pedestrians PDx and PDy are detected in each frame within section time bt1 (timestamp ts1-ts5) of video data VD_CAi. Furthermore, pedestrian PDz is detected in each frame within section time bt2 (timestamp ts6-ts10) of video data VD_CAj. Bounding boxes surrounding the pedestrians PDx, PDy, and PDz are added to the positions where the pedestrians PDx, PDy, and PDz are detected. By cropping this bounding box, an image IM of the pedestrians PDx, PDy, and PDz is obtained. PDx , I.M. PDy and I.M. PDz The image IM of the pedestrian PD is extracted. PDx and I.M. PDy For example, the ID of the camera CAi CAi The data of the time stamp ts and the coordinates CD in the frame FR of the extracted image PD The data includes:
[0024] In the re-identification process, each image IM extracted in the detection and extraction process is PDFrom each image IM, a feature for re-identification processing (hereinafter also referred to as "Re-ID feature") is extracted. The Re-ID feature is extracted using a Re-ID model based on machine learning. Note that the technology for extracting Re-ID features using the Re-ID model is well known, and the method is not particularly limited. PD Once the Re-ID features are extracted, the Re-ID features are compared to determine whether the people in the image sequences are the same person.
[0025] In the example shown in FIG. 2, each image IM PD Since the Re-ID feature values of the people extracted from the images are similar to each other, it is determined that the pedestrians PDx and PDy included in the image sequence in the section time bt1 are the same person. PD Since the Re-ID features of the people extracted from are similar to each other, it is determined that the pedestrian PDz included in the image sequence in the section time bt2 is the same person.
[0026] When the detection process is performed, a tracking ID is assigned to the person captured in the video data VD_CAn. PD Among these, if a person is determined to be the same person through the re-identification process, a common tracking ID for this person is assigned. PD (hereinafter, "Common Tracking ID PD ") is assigned to the shipment. PD is also called a universally unique ID (UUID). In the example shown in FIG. 2, for pedestrians PDx, PDy, and PDz, a common tracking ID PDx(bt1) , ID PDy(bt1) and ID PDz(bt2) Each common tracking ID is assigned. PD is a combination of data of a section time bt and data representative of the data of the Re-ID feature amount extracted from the image sequence of this section time bt. Note that there is no particular limitation on the example of selecting the representative data of the Re-ID feature amount, and any method can be adopted.
[0027] Common Tracking IDPD is generated for each section time bt. Therefore, if the same pedestrian PD continues to be captured by one camera (single camera), the common tracking ID PD may occur separately for the number of section times bt. Therefore, in the re-identification process, the Re-ID feature values may be compared between multiple image sequences that differ in the section time bt. For example, the Re-ID feature values are compared between two image sequences of section times bt that have close timestamps ts. If the Re-ID feature values are similar between multiple image sequences, it is determined that the pedestrians PD included in these image sequences are the same person, and the common tracking ID that was assigned separately to each pedestrian PD is used. PD may be combined into one.
[0028] The graph generation processing unit 14 uses the common tracking ID given to the pedestrian PD by the above-mentioned re-identification process. PD and the IDs of at least two cameras arranged in the city CT. The generated graph GRH is stored in the graph DB 17. As already explained, the graph GRH is expressed using nodes (vertices, nodes) and edges (sides, branches) in graph theory. FIG. 3 is a diagram showing an example of the basic configuration of the graph GRH generated by the graph generation processing unit 14. FIG. 3 shows a node N_ID representing each ID of the cameras CA1-CA6 (however, CA2 is omitted) shown in FIG. 1, CA1 -N_ID CA6 and edges E that indicate relationships between these nodes.
[0029] Here, the installation position of the camera CA1 is close to that of the camera CA3. Therefore, there is a relationship between these cameras. Therefore, in the graph GRH1 shown in FIG. 3, a node N_ID representing the ID of the camera CA1 is CA1 and a node N_ID representing the ID of the camera CA3. CA3are connected via edge E_CA1-3. The meaning of this edge E_CA1-3 is "NEARBY". The "NEARBY" relationship also exists between camera CA1 and camera CA4, between camera CA4 and camera CA5, and between camera CA5 and camera CA6. Therefore, the nodes N representing the IDs of two related cameras CA are connected by one edge E (edge E_CA1-4, edge E_CA4-5, and edge E_CA5-6).
[0030] Another example of the relationship between two cameras CA is that the shooting ranges of these cameras overlap in part or in whole. Here, part of the shooting range of camera CA3 overlaps with that of camera CA5. Therefore, in the graph GRH1 shown in FIG. 3, the node N_ID representing the ID of camera CA3 CA3 and a node N_ID representing the ID of the camera CA5. CA5 are connected via an edge E_CA3-5, which has the meaning "OVERLAPPED".
[0031] A common tracking ID assigned to pedestrian PD PD Represents the node N_ID PD This common tracking ID PD Node N_ID representing the camera CAn that acquired the video data VD_CAn that was the basis for the assignment CAn , and are connected to each other via at least one edge E. FIG. 3 shows the common tracking IDs assigned to pedestrians PDp, PDq, PDr, PDs, and PDu. PD Represents the node N_ID PDp , N_ID PDq , N_ID PDr , N_ID PDs and N_ID PDu is drawn. Node N_ID PDp connects to node N_ID via edge E_CA1 CA1 Node N_ID PDq and N_ID PDr connects to node N_ID via two edges E_CA4 CA4Node N_ID PDs connects to node N_ID via edge E_CA5 CA5 Node N_ID PDu connects to node N_ID via edge E_CA3 CA3 is tied to.
[0032] As already explained, the common tracking ID PD The data of the section time bt and the representative data of the Re-ID feature of the image sequence are combined in the re-identification process. In the embodiment, the representative data of the Re-ID feature is used to perform a re-identification process of the person captured in at least two cameras. In this re-identification process, the same process as the comparison of the Re-ID feature between different image sequences in the section time bt is performed. However, while the comparison of the Re-ID feature between the image sequences is performed for one camera, the comparison of the representative data of the Re-ID feature is performed for two cameras. If the representative data of the Re-ID feature between the two cameras is similar, it is determined that the pedestrian PD captured separately by these cameras is the same person.
[0033] When it is determined that the pedestrians PD captured by the two cameras are the same person, the graph generation processing unit 14 uses the common tracking IDs that have been assigned to these pedestrians PD separately. PD Represents the node N_ID PD are connected via an edge E. In the graph GRH1 shown in FIG. 3, the pedestrians PDp and PDq are determined to be the same person, and the pedestrians PDr and PDs are determined to be the same person. PDp and node N_ID PDq are connected via edges E_IDp-q, and node N_ID PDr and node N_ID PDs are connected via edge E_IDr-s. The meaning of edges E_IDp-q and E_IDr-s is “SAME PERSON.”
[0034] In the graph GRH1 shown in FIG. 3, it is also determined that pedestrians PDq and PDr are the same person. This determination is based on the result of a comparison of Re-ID features between multiple image sequences that are different in the interval time bt, performed for a single camera. For this reason, PDq and node N_ID PDr are connected via an edge E_IDq-r, which means "SAME PERSON".
[0035] Even if the comparison of the Re-ID features determines that the pedestrians captured by the two cameras are not the same person, a common tracking ID is assigned to each of these pedestrians when certain movement conditions are met. PD Represents the node N_ID PD may be connected via an edge E_ID that means “SAME PERSON.” The predetermined movement conditions include, for example, the following conditions (i) to (iii). (i) The similarity of the Re-ID feature is equal to or greater than a reference value. (ii) Two nodes N_ID PD Each additional information ADD (image IM PD ) The interval between the timestamps ts is within a specified time. (iii) Node N_ID of two points in condition (i) PD The distance between the installation positions of the two cameras CA connected to each other is within a specified distance.
[0036] Fig. 4 is a diagram showing a detailed configuration example of a graph generated by the graph generation processing unit 14. In the graph GRH2 shown in Fig. 4, a node N representing additional information ADD is added to the graph GRH1 shown in Fig. 3. This additional information ADD includes a common tracking ID PD The additional information ADD is the image IM of the pedestrian PD. PD , Appearance characteristics of pedestrian PD AP PD , Pedestrian PD's behavior AC PD , pedestrian PD's face image IMF PD Examples include:
[0037] Pedestrian PD image IM PD was used to extract the Re-ID features. Examples of the appearance features of the pedestrian PD include the color, clothing, and body type of the pedestrian PD. These appearance features are estimated using a pre-trained appearance model. Examples of the actions of the pedestrian PD include "walking," as well as "carrying" and "opening" actions that the pedestrian PD takes with respect to stationary objects such as luggage. These actions are estimated using a pre-trained action model. These actions also include interaction actions such as "conversation" and "handing over" that multiple people take together. The facial image IMF of the pedestrian PD PD IM is an image of a pedestrian PD PD The face image may be an image in which the face portion has been trimmed from the original image, or may be a face image provided from outside in a search process (described later) by the search processing unit 15.
[0038] In the graph GRH2 shown in FIG. 4, there is a node N_ID PDp Three edges E_ID extending from PDp Next, the image IM of the pedestrian PDp PDp Represents the node N_IM PDp(bt1) and N_IM PDp(bt2) and node N_AC which represents the behavior of pedestrian PDp. PDp(bt1) In addition, the node N_ID PDq Three edges E_ID extending from PDq Next, the face image IMF of pedestrian PDq PDq Node N_IMF PDq and the image IM of the pedestrian PDq PDq Represents the node N_IM PDq(bt3) and node N_AC which represents the behavior of pedestrian PDq. PDq(bt3) and are arranged.
[0039] Although detailed explanation is omitted, in the graph GRH2, the node N_ID PDr , N_ID PDs and N_ID PDu For each pedestrian PD, a node N representing additional information ADD is connected to a node N_ID via an edge E. PDIn addition, the node N_ID PDs and node N_ID PDu is an edge E_IA that means the interaction action "TALKING" PDs-PDu Also, the node N_ID PDs to edge E_ID PDs Node N_AP connected via PDs(bt6) is the appearance feature AP of pedestrian PDs in section time bt6 PDs Represents.
[0040] 2-2. Search processing section The search processing unit 15 performs a process of searching for the tracking target using the graph GPH stored in the graph DB 17. Fig. 5 is a diagram for explaining an example of the search process performed by the search processing unit 15. In the example shown in Fig. 5, an image IM of the tracking target is TGT The graph DB 17 is referenced with the query Q (input information). TGT is selected from the frame of the pedestrian PD used to extract the Re-ID features. TGT may be provided from outside the tracking system. TGT is the face image IMF of the target TGT Or face image IMF TGT is, for example, cropped from the frame of the pedestrian PD used to extract the Re-ID features. TGT may be provided externally to the tracking system.
[0041] In the example shown in FIG. TGT The image IM of the pedestrian PD that has the Re-ID feature most similar to the Re-ID feature extracted from PD In this search, image IM TGT The Re-ID feature extracted from the common tracking ID PD Represents the node N_ID PD Node N_IM connected to PD The image IM of the pedestrian PD is represented by PDThe Re-ID feature values extracted from the image IM of the pedestrian PD are compared. PD The Re-ID features extracted from, for example, a common tracking ID PD Representative data of the Re-ID features selected during the assignment are used.
[0042] Figure 5 shows the common tracking IDs of the pedestrian PDA, PDB, PDC and PDD. PD Represents the node N_ID PD and each node N_ID PD to edge E_ID PD These pedestrians are connected through the IM PD Represents the node N_IM PD The graph of the pedestrian PDB is a part of the graph GRH2 explained in Figure 4.
[0043] In the example shown in FIG. 5, the image IM PDs It is determined that the similarity of the Re-ID feature values extracted from the pedestrians PDs is the highest. Therefore, in this example, the pedestrians PDs are identified as the people most likely to be the tracking targets. When the pedestrians PDs are identified as the people most likely to be the tracking targets, the common tracking IDs assigned to the pedestrians PDs are PD Represents the node N_ID PDs Node N_ID is connected to node N_ID via edge E_ID, which means "SAME PERSON". PD Group of (Pedestrian PDB node N_ID PD A set of nodes (
[0044] In the example shown in FIG. PDs is node N_ID PDr and node N_ID via edge E_IDr-s. PDr is node N_ID PDq and E_IDq-r via edge E_IDq-r. Also, node N_ID PDq is node N_ID PDp and node N_ID via edge E_IDp-q. PDp is node N_IDPDv and node N_ID via edge E_IDp-v. PDv is node N_ID PDw and via an edge E_IDv-w. Both of these edges E_ID mean "SAME PERSON."
[0045] Therefore, if the image IM of a pedestrian PDp PDp The Re-ID features extracted from the image IM TGT Even if the Re-ID feature extracted from the pedestrian PDs is not similar to the Re-ID feature extracted from the pedestrian PDs, the pedestrian PDs is identified as the person most likely to be the tracking target, and the tracking target graph (tracking target graph) GRH is obtained as the search result R(TRC). TGT It is possible to obtain the following.
[0046] In the example shown in FIG. 5, the image IM TGT The graph DB 17 is referenced with the query Q (input information). TGT In addition, information such as date, time, and location may be added to the query Q. In the example shown in FIG. 5, the common tracking ID of the tracking target is PD Represents the node N_ID PD and the image IM of the tracked object PD Represents the node N_IM PD and the graph GRH TGT is output as the search result R(TRC). However, the node N_ID representing the camera CAn described in FIG. CA IM, as explained in Fig. 4. PD A node N representing additional information ADD other than TGT may be included in the
[0047] Another example of the search process for tracking targets is the common tracking ID PD Represents the node N_ID PD The search is narrowed down by node N_ID. PD By narrowing down the number of nodes, it is expected that the processing load of the search by the processor 11 will be reduced. PDThe narrowing down of the search results is performed according to a predetermined narrowing down condition. The predetermined narrowing down condition includes, for example, at least one of the following conditions (i) and (ii). (i) Node N_ID corresponding to the root node or leaf node PD To be (ii) Node N_ID with a connection degree equal to or greater than a given degree PD To be
[0048] Regarding condition (i), FIG. 6 is a diagram for explaining the root node and the leaf node. In FIG. 6, the node N_ID PDp , N_ID PDq , N_ID PDr , N_ID PDs , N_ID PDv , and N_ID PDw This graph is a part of the pedestrian PDB graph described in Fig. 5. However, in the example shown in Fig. 6, two adjacent nodes N_ID PD The direction of the arrow indicates the predicted movement direction of the pedestrian PDB based on the section time bt (or the timestamp ts).
[0049] The root node is the node N_ID in the pedestrian PDB graph. PD (i.e., node N_ID in the pedestrian PDB PD The ID of the node N_ID that corresponds to the "root" of the group of nodes PD In the example shown in FIG. PDp The root node is, for example, the oldest node N_ID PD Typically, the root node is the node N_ID PD Node N_IM connected to PD The node N_ID that has the oldest data of timestamp ts (or data of interval time bt) PD If the query Q contains date and time information, the oldest node N_ID in this date and time range is PD is the root node.
[0050] Leaf nodes are nodes N_ID in the pedestrian PDB graph. PD (i.e., node N_ID in the pedestrian PDB PD The node N_ID corresponds to the "leaf" of the node group PD The leaf node is, for example, the newest node N_ID PD Typically, the leaf nodes are the nodes N_ID PD Node N_IM connected to PD The node N_ID that has the latest data of timestamp ts (or data of interval time bt) PD The leaf node is the newest node N_ID PD It does not have to be the N_ID of the node excluding the root node. PD Among the nodes in the group, node N_ID PD Node N_ID located at the end of PD may correspond to a leaf node. Thus, in the example shown in FIG. PDs and N_ID PDr corresponds to a leaf node.
[0051] Regarding condition (ii), node N_ID PD The "connection degree" of the node N_ID that composes the node group is PD The total number of nodes that make up the node group. PD A large total number of nodes means a high degree of connection. PD By focusing only on the node N_ID, it is expected that the processing load of the search by the processor 11 can be reduced. PD This means that only the node with the lowest connection degree, N_ID PD This is because it can be excluded from the search as noise data.
[0052] 2-3.Graph sorting processing unit The graph rearrangement processing unit 16 performs a process of rearranging the graph GPH generated by the graph generation processing unit 14. Specifically, the graph rearrangement processing unit 16 rearranges the connections of the edge E_ID that means "SAME PERSON." As already explained, the edge E_ID that means "SAME PERSON" is the common tracking ID of the pedestrian PD with similar Re-ID feature values. PD Represents the node N_ID PD However, the node N_ID connected by this edge E_ID PD does not include time information. Therefore, the graph (tracking target graph) GRH TGT Although it is possible to roughly track the target, it is difficult to grasp the direction in which the target is moving.
[0053] Therefore, the edge E_ID, which means "SAME PERSON", is reconnected. This reconnection of the edge E_ID is performed periodically, independently of the generation process of the graph GRH. The reconnection of the edge E_ID is performed by the node N_ID. PD Node N_IM connected to PD This is performed based on the data of the time stamp ts (or the data of the section time bt) possessed by the
[0054] FIG. 7 is a diagram for explaining an example of processing performed by the graph rearrangement processing unit 16. FIG. 7 illustrates the graph of the pedestrian PDB explained in FIG. 6. The upper part of FIG. 7 is an example of a graph before rearrangement processing (before reconnection). As can be seen from the upper part of FIG. 7, before rearrangement processing, PD Represents the node N_ID PDp Branches to node N_ID PDq and N_ID PDv Such a connection is made by the node N_ID PDp and node N_ID PDq Between and node N_ID PDp and node N_ID PDv The Re-ID features are determined to be similar between node N_ID PDq and node N_ID PDrThis can occur when it is determined that the features between
[0055] The bottom part of Figure 7 is an example of a graph after sorting (reconnection). As can be seen from the bottom part of Figure 7, the sorting process results in a common tracking ID that is linked to the same person. PD Therefore, in the search process using the graph GRH after the sorting process, the graph GRH containing information on the moving direction of the tracking target is generated. TGT This makes it possible to output the search result R(TRC). This contributes to improving the usability of the search result R(TRC). [Explanation of symbols]
[0056] 10 Management server, 11 Processor, 12 Storage device, 14 Graph generation processing unit, 15 Search processing unit, 16 Graph sorting processing unit, 17 Graph DB, CA, CA1-CA6, CAi, CAj, CAn Camera, FR Frame, PD, PDp-PDs, PDu-PDz Pedestrian, VD, VD_CA1-CA6, VD_CAn Video data, AC PD behavior, ap PD Appearance features, IM PD ,IM PDp ,IM PDq ,IM PDx -IM PDz ,IM TGT Image: IMF PD ,IMF PDq ,IMF TGT Face image, E,E_CA1,E_CA3-CA7,E_CA1-3,E_CA1-4,E_CA3-5,E_CA4-5,E_CA5-6,E_CA3-7,E_ID PDp -E_ID PDs ,E_ID PDu ,E_IDp-q,E_IDq-r,E_IDr-s,E_IDp-v,E_IDv-w,E_IDv-q,E_IDr-w,E_IDw-s,E_IA PDs - PDu Edge, N, N_AC PDp -N_AP PDs ,N_IDCA ,N_ID CA1 -N_ID CA7 ,N_ID PD ,N_ID PDp -N_ID PDs ,N_ID PDu -N_ID PDw ,N_IMF PDq Node, ADD additional information, GRH, GRH1, GRH2, GRH TGT graph
Claims
1. a storage device in which each image data captured by at least two cameras is stored; A processor; Equipped with The processor, A process of generating a graph including at least two nodes and at least one edge indicating a relationship between the at least two nodes based on each of the video data; a process of searching for the tracked object by referring to the graph using a query including an image of the tracked object as an input; configured to: In the graph, a node representing a single camera included in the at least two cameras and a node representing a tracking identification number assigned to a moving object captured in video data acquired by the single camera are connected via at least one edge; The tracking identification number includes a common tracking identification number that is assigned to the same moving object captured in the video data acquired by the single camera, At least two of the single cameras are connected to each other through at least one edge indicating a relationship between the single cameras, among the nodes representing the single cameras, the nodes having the relationship; Nodes that are recognized as being the same moving object among the nodes that represent the at least two common tracking identification numbers are connected to each other via at least one edge that indicates that the at least two moving objects captured in each of the video data are the same moving object, a node representing the common tracking identification number and a node representing an image of the same moving object to which the common tracking identification number is assigned are connected via at least one edge; In the process of searching for the tracking target, the processor Extracting a feature amount of the tracked object from an image of the tracked object; Identifying a moving object having a feature quantity most similar to the feature quantity of the tracked target from feature quantities of at least two moving objects respectively extracted from images of the at least two moving objects represented by at least two nodes constituting the graph; Identifying a tracking target graph indicating a graph including a node representing the tracking identification number assigned to the identified moving object and at least one node connected to the node representing the tracking identification number via at least one edge. A moving object tracking system comprising:
2. In the process of searching for the tracking target, the processor before extracting feature quantities of the at least two moving objects from the images of the at least two moving objects, selecting at least two nodes from the at least two nodes constituting the graph in accordance with a predetermined narrowing-down condition; From the images of the at least two moving objects represented by the at least two selected nodes, feature amounts of the at least two moving objects are extracted, respectively.
2. The moving object tracking system according to claim 1 .
3. In the process of generating the graph, the processor determining whether the at least two moving objects are the same moving object based on feature amounts of the moving objects extracted from images of the moving objects connected via at least one edge to a node representing a tracking identification number assigned to each of the moving objects; When it is determined that the at least two moving objects are similar, the nodes representing the common tracking identification numbers respectively assigned to these moving objects are connected via at least one edge indicating that the at least two moving objects are the same moving object.
2. The moving object tracking system according to claim 1 .
4. The processor, in the process of generating the graph, further if it is determined that the at least two moving objects are not similar in the determination based on the feature amounts of the at least two moving objects, determining whether or not a predetermined movement condition is satisfied for the at least two moving objects; When it is determined that the movement condition is satisfied for the at least two moving objects, the nodes representing the common tracking identification numbers respectively assigned to these moving objects are connected to each other via at least one edge indicating that the at least two moving objects are the same moving object.
4. The moving object tracking system according to claim 3.
5. The processor further comprises: The graph generating process is configured to perform a process of arranging the graph generated by the process of generating the graph, In the process of arranging the graph, the processor extracting a node group in which the number of connections between nodes representing the common tracking identification numbers respectively assigned to the at least two moving objects is at least three; At least one edge connecting the nodes constituting the node group is reconnected based on time stamp data for images of at least three moving objects each connected to the nodes constituting the node group via at least one edge.
5. A moving object tracking system according to claim 3 or 4.
Citation Information
Patent Citations
Pedestrian tracking method and device and storage medium
CN113935358A
Knowledge information processing server system with image recognition system
JP2013088906A
Object identification device and method based on plurality of camera images
JP2019066925A
Three-dimensional position estimation device and program
JP2020122754A
Abnormality detection device and abnormality detection method
JP2022086627A
