Tracking system for moving object

The system addresses memory overflow in video and graph data by generating graphs and calculating probability distributions, ensuring efficient use of storage resources while preserving valuable tracking information.

JP2024166950AInactive Publication Date: 2024-11-29TOYOTA JIDOSHA KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023083395
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing systems that generate graphs from video data acquired by multiple cameras face memory overflow issues due to the accumulation of video and graph data, leading to potential waste of valuable tracking information.

Method used

A moving object tracking system that generates graphs using single or multiple cameras, calculates existence probability distributions, and compresses or deletes unnecessary data to manage memory capacity effectively.

Benefits of technology

Prevents waste of graph and associated data while reducing memory pressure by compressing or deleting data not essential for probability distribution calculations, thus optimizing storage usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166950000001_ABST
    Figure 2024166950000001_ABST
Patent Text Reader

Abstract

To suppress waste of data of a graph generated from video data and graph-associated data obtained in the generation of the graph, and prevent memory capacity of a storage device storing the data from running low.SOLUTION: A processor performs data processing based on video data acquired by a single camera, the data processing including generating a graph. The data processing further includes: calculating a presence probability distribution of an arbitrary identical moving object in each frame group of video data acquired by the single camera, by analyzing coordinate history of the identical moving objects in the frame groups of the video data including a plurality of identical moving objects with multiple common identification numbers assigned thereto; and compressing graph-associated data obtained in generating the graph, and obtained by removing data of the coordinate history of the identical moving objects used for calculating the presence probability distribution, or deleting the data from a storge device.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a system that uses video data acquired by a single camera or at least two cameras to track a moving object captured in the video data. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2017-021753 discloses a technique for graphing images of people captured in video data acquired by multiple cameras. In this conventional technique, frames in which people are detected are extracted from among frames (images) included in the video data. Next, a rectangular area (i.e., a bounding box) containing the detected person is trimmed from this extracted frame. The extraction of frames in which people are detected and the trimming of the bounding box are performed on multiple frames that differ in at least one of the time and place of capture. Therefore, multiple bounding boxes are extracted.

[0003] In the technique described in the above publication, an image sequence consisting of five bounding boxes is then extracted from the extracted bounding boxes with reference to the Euclidean distance between the cameras. Then, a similarity between the person images included in the image sequence is calculated based on the feature quantities of the person images. This similarity is used to determine whether the person images included in the five bounding boxes are the same person.

[0004] In the technology described in the above publication, when it is determined that the human images contained in five bounding boxes are of the same person, a graph is generated for the five bounding boxes. This graph is expressed using nodes (vertices, nodes) and edges (sides, branches) in graph theory. Figures 7-9 of the above publication show an example of a graph in which five bounding boxes (human images) determined to be of the same person through data processing of video data acquired by multiple cameras installed in different locations are treated as nodes, and these bounding boxes are connected by edges.

[0005] In addition to JP 2017-021753 A, examples of documents showing the technical state of the technical field related to the present disclosure include JP 2020-107171 A, JP 2020-149422 A, and Japanese Patent No. 7066362 A. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-021753 A [Patent Document 2] JP 2020-107171 A [Patent Document 3] JP 2020-149422 A [Patent Document 4] Patent No. 7066362 Summary of the Invention [Problem to be solved by the invention]

[0007] Although JP2017-021753A does not mention how to handle the video data captured by multiple cameras after graph generation, such video data is usually deleted from a storage device after a certain period of time has passed. The reason for this is to prevent the memory capacity of the storage device from being overwhelmed by video data that is constantly being captured.

[0008] Consider the case where a graph is generated from video data. In this case, if the video data used to generate the graph is deleted from the storage device after a certain period of time has passed, the memory capacity of the storage device can be prevented from being overloaded. However, when a graph is generated, in addition to the graph data, the graph-accompanying data associated with the generation of the graph is also stored in the storage device. Therefore, in the long term, the same problem as the problem of storage caused by video data may occur. However, deleting all of the graph data and the graph-accompanying data from the storage device after a certain period of time has passed is undesirable from the viewpoint of utilizing the graph and the graph-accompanying data, particularly from the viewpoint of tracking a moving object using the graph. Therefore, development from this viewpoint is required.

[0009] One object of the present disclosure is to provide a technology that can prevent waste of graph data generated from video data and graph-accompanying data associated with the generation of this graph, while minimizing pressure on the memory capacity of the storage device in which this data is stored. [Means for solving the problem]

[0010] A first aspect of the present disclosure is a moving object tracking system having the following features. The tracking system includes a storage device, the storage device storing image data captured by a single camera, and a processor, the processor configured to perform data processing based on the image data. In the data processing, the processor generates a graph consisting of at least two nodes and at least one edge indicating a relationship between the at least two nodes. In the graph, a node representing the single camera and a node representing an identification number given to a moving object captured in video data acquired by the single camera are connected via an edge. The identification number includes a common identification number given to the same moving object captured in video data acquired by the single camera. In the data processing, the processor further calculates the existence probability distribution of any of the same moving objects in the group of frames of the video data acquired by the single camera by analyzing the coordinate history of multiple identical moving objects, each of which is assigned a multiple number of common identification numbers, in each group of frames of the video data in which the multiple identical moving objects are captured, and compresses or erases from the storage device data, excluding the coordinate history data of the multiple identical moving objects used in calculating the existence probability distribution, from the graph-accompanying data associated with the generation of the graph.

[0011] A second aspect of the present disclosure is a moving object tracking system having the following features. The tracking system includes a storage device, which stores image data captured by at least two cameras, and a processor, which is configured to perform data processing based on the image data. In the data processing, the processor generates a graph consisting of at least two nodes and at least one edge indicating a relationship between the at least two nodes. In the graph, a node representing a single camera included in the at least two cameras is connected to a node representing an identification number given to a moving object captured in video data acquired by the single camera via at least one edge. The identification number includes a common identification number given to the same moving object captured in video data acquired by the single camera. In the graph, nodes having the relationship among the nodes representing the at least two single cameras are connected to each other via at least one edge indicating the relationship between the at least two single cameras. In the graph, nodes recognized as the same moving object among the nodes representing the at least two common identification numbers are further connected to each other via at least one edge indicating that at least two moving objects captured in each video data acquired by the at least two single cameras are the same moving object. In the data processing, the processor further calculates a presence probability distribution of any of the same moving objects between the at least two single cameras that acquired the video data in which multiple identical moving objects, each of which is assigned multiple common identification numbers, are captured by analyzing the movement history of these multiple identical moving objects between the at least two single cameras, and compresses or erases from the storage device data from the graph-accompanying data associated with the generation of the graph, excluding the data on the movement history of the multiple identical moving objects used in calculating the presence probability distribution. Effect of the Invention

[0012] According to the first aspect, not only is a graph generated in the data processing, but the existence probability distribution of any identical moving object in a group of frames of video data acquired by a single camera is calculated, and further, among the graph-accompanying data accompanying the generation of the graph, data excluding the coordinate history data of the multiple identical moving objects used in the calculation of the existence probability distribution is compressed or deleted from the storage device. Therefore, according to the first aspect, it is possible to prevent the data of the graph generated from the video data acquired by a single camera and the graph-accompanying data accompanying the generation of this graph from going to waste, while suppressing pressure on the memory capacity of the storage device in which these data are stored.

[0013] According to the second aspect, not only is a graph generated in the data processing, but the existence probability distribution of any identical moving object between at least two single cameras is calculated, and further, among the graph-accompanying data accompanying the generation of the graph, data excluding the data of the movement history of the multiple identical moving objects used in the calculation of the existence probability distribution is compressed or deleted from the storage device. Therefore, according to the second aspect, it is possible to prevent the data of the graph generated from each video data acquired by at least two single cameras and the graph-accompanying data accompanying the generation of this graph from going to waste, while suppressing pressure on the memory capacity of the storage device in which these data are stored. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a tracking system according to an embodiment. [Diagram 2] 11A and 11B are diagrams illustrating a person detection process and a person re-identification process performed by a graph generation processing unit. [Diagram 3] 13 is a diagram illustrating an example of a basic configuration of a graph generated by a graph generation processing unit. FIG. [Figure 4] 13 is a diagram illustrating an example of a detailed configuration of a graph generated by a graph generation processing unit; FIG. [Diagram 5] FIG. 11 is a diagram for explaining a first calculation example performed by a probability distribution calculation unit. [Figure 6] FIG. 11 is a diagram for explaining a second calculation example performed by the probability distribution calculation unit. [Figure 7] 11A and 11B are diagrams illustrating an example of a tracking process performed by a tracking processing unit. [Figure 8] 11A and 11B are diagrams illustrating an example of a tracking process performed by a tracking processing unit. [Figure 9] 11A and 11B are diagrams illustrating an example of a tracking process performed by a tracking processing unit. [Figure 10] 11A and 11B are diagrams illustrating an example of a tracking process performed by a tracking processing unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and the description thereof will be simplified or omitted.

[0016] 1. Overall configuration example FIG. 1 is a diagram for explaining an example of the overall configuration of a tracking system (hereinafter, also simply referred to as "system") according to an embodiment. The system according to the embodiment is a system for tracking a moving object moving inside a city CT. There is no limitation on the scale of the city CT in the present disclosure. A so-called smart city is an example of a large-scale city CT, an underground mall is an example of a medium-scale city CT, and a large building is an example of a small-scale city CT. Examples of the moving object include a person, a robot, and a vehicle. In the embodiment, it is assumed that the moving object is a person (pedestrian PD). In FIG. 1, pedestrians PD1-PD3 are depicted as examples of pedestrians PD.

[0017] The system according to the embodiment includes at least two cameras arranged in the city CT. In FIG. 1, cameras CA1-CA6 are illustrated as an example of the at least two cameras. Camera CA1 acquires video data VD_CA1. Like camera CA1, cameras CA2-CA6 acquire video data VD_CA2-VD_CA6, respectively. Video data VD_CAn of any one of cameras CAn (n is a natural number) arranged in the city CT is transmitted to the management server 10 via a communication network. The communication network is not particularly limited, and wired and wireless networks can be used.

[0018] The management server 10 is a computer including at least one processor 11, at least one storage device 12, and at least one interface 13. The processor 11 performs various data processing. The processor 11 includes a CPU (Central Processing Unit). The storage device 12 stores various data required for data processing. Examples of the storage device 12 include HDD, SSD, volatile memory, and non-volatile memory. The interface 13 receives various data from the outside and outputs various data to the outside. The various data received by the interface 13 from the outside includes video data VD_CAn. This video data VD_CAn is stored in the storage device 12. A graph DB (database) 18 and a probability distribution DB 19 are formed in the storage device 12. The graph DB 18 and the probability distribution DB 19 may be formed in an external device that can communicate with the management server 10.

[0019] 2. Management Server Configuration Example Fig. 1 shows an example of the functional configuration of the management server 10. In the example shown in Fig. 1, the management server 10 includes a graph generation processing unit 14, a probability distribution calculation unit 15, a data reduction unit 16, and a tracking processing unit 17. These functions are realized by the processor 11 executing various programs stored in the storage device 12.

[0020] 2-1.Graph generation processing section The graph generation processing unit 14 performs a process of generating the graph GPH based on the video data VD_CAn. In order to generate the graph GPH, the graph generation processing unit 14 performs a person detection and extraction process and a person re-identification process. Fig. 2 is a diagram for explaining the person detection process and the person re-identification process performed by the graph generation processing unit 14.

[0021] 2 illustrates video data VD_CAi from camera CAi and video data VD_CAj from camera CAj as an example of video data VD_CAn. The video data VD_CAi and VD_CAj are each separated by a predetermined time width, and FIG 2 illustrates a set of frames FR in a section time bt. Each frame FR of the video data VD_CAi is, for example, CAi As with each frame FR of the video data VD_CAi, each frame FR of the video data VD_CAj also includes the ID of the camera CAj. CAj and data at timestamp ts.

[0022] In the detection and extraction process, first, a frame FR in which a person is detected is extracted. Next, the detected person is extracted from this extracted frame. In the example shown in FIG. 2, pedestrians PDx and PDy are detected in each frame within section time bt1 (timestamp ts1-ts5) of video data VD_CAi. Furthermore, pedestrian PDz is detected in each frame within section time bt2 (timestamp ts6-ts10) of video data VD_CAj. Bounding boxes surrounding the pedestrians PDx, PDy, and PDz are added to the positions where the pedestrians PDx, PDy, and PDz are detected. By cropping this bounding box, an image IM of the pedestrians PDx, PDy, and PDz is obtained. PDx , I.M. PDy and I.M. PDz The image IM of the pedestrian PD is extracted. PDx and I.M. PDy For example, the ID of the camera CAi CAi The data of the time stamp ts and the coordinates CD in the frame FR of the extracted image PD The data includes:

[0023] In the re-identification process, each image IM extracted in the detection and extraction process is PDFrom each image IM, a feature for re-identification processing (hereinafter also referred to as "Re-ID feature") is extracted. The Re-ID feature is extracted using a Re-ID model based on machine learning. Note that the technology for extracting Re-ID features using the Re-ID model is well known, and the method is not particularly limited. PD Once the Re-ID features are extracted, the Re-ID features are compared to determine whether the people in the image sequences are the same person.

[0024] In the example shown in FIG. 2, each image IM PD Since the Re-ID feature values ​​of the people extracted from the images IM and PD are similar to each other, it is determined that the pedestrians PDx and PDy included in the image sequence in the section time bt1 are the same person. PD Since the Re-ID feature values ​​of the people extracted from are similar to each other, the pedestrian PDz included in the image sequence in the section time bt2 is determined to be the same person.

[0025] When the detection process is performed, a tracking ID is assigned to the person captured in the video data VD_CAn. PD Among these, if a person is determined to be the same person through the re-identification process, a common tracking ID for this person is assigned. PD (hereinafter, "Common Tracking ID PD ") is assigned to the shipment. PD is also called a universally unique ID (UUID). In the example shown in FIG. 2, for pedestrians PDx, PDy, and PDz, a common tracking ID PDx(bt1) , ID PDy(bt1) and ID PDz(bt2) Each common tracking ID is assigned. PD is a combination of data of a section time bt and data representative of the data of the Re-ID feature amount extracted from the image sequence of this section time bt. Note that there is no particular limitation on the example of selecting the representative data of the Re-ID feature amount, and any method can be adopted.

[0026] Common Tracking IDPD is generated for each section time bt. Therefore, if the same pedestrian PD continues to be captured by one camera (single camera), the common tracking ID PD may occur separately for the number of section times bt. Therefore, in the re-identification process, the Re-ID feature values ​​may be compared between multiple image sequences that differ in the section time bt. For example, the Re-ID feature values ​​are compared between two image sequences of section times bt that have close timestamps ts. If the Re-ID feature values ​​are similar between multiple image sequences, it is determined that the pedestrians PD included in these image sequences are the same person, and the common tracking ID that was assigned separately to each pedestrian PD is used. PD may be combined into one.

[0027] The graph generation processing unit 14 uses the common tracking ID given to the pedestrian PD by the above-mentioned re-identification process. PD and the IDs of at least two cameras placed in the city CT. The generated graph GRH is stored in the graph DB 18. As already explained, the graph GRH is expressed using nodes (vertices, nodes) and edges (sides, branches) in graph theory. FIG. 3 is a diagram showing an example of the basic configuration of the graph GRH generated by the graph generation processing unit 14. FIG. 3 shows a node N_ID representing each ID of the cameras CA1-CA6 (however, CA2 is omitted) shown in FIG. 1, CA1 -N_ID CA6 and edges E that indicate relationships between these nodes.

[0028] Here, the installation position of the camera CA1 is close to that of the camera CA3. Therefore, there is a relationship between these cameras. Therefore, in the graph GRH1 shown in FIG. 3, a node N_ID representing the ID of the camera CA1 is CA1 and a node N_ID representing the ID of the camera CA3. CA3are connected via edge E_CA1-3. The meaning of this edge E_CA1-3 is "NEARBY". The "NEARBY" relationship also exists between camera CA1 and camera CA4, between camera CA4 and camera CA5, and between camera CA5 and camera CA6. Therefore, the nodes N representing the IDs of two related cameras CA are connected by one edge E (edge ​​E_CA1-4, edge E_CA4-5, and edge E_CA5-6).

[0029] Another example of the relationship between two cameras CA is that the shooting ranges of these cameras overlap in part or in whole. Here, part of the shooting range of camera CA3 overlaps with that of camera CA5. Therefore, in the graph GRH1 shown in FIG. 3, the node N_ID representing the ID of camera CA3 CA3 and a node N_ID representing the ID of the camera CA5. CA5 are connected via an edge E_CA3-5, which has the meaning "OVERLAPPED".

[0030] A common tracking ID assigned to pedestrian PD PD Represents the node N_ID PD This common tracking ID PD Node N_ID representing the camera CAn that acquired the video data VD_CAn that was the basis for the assignment CAn , and are connected to each other via at least one edge E. FIG. 3 shows the common tracking IDs assigned to pedestrians PDp, PDq, PDr, PDs, and PDu. PD Represents the node N_ID PDp , N_ID PDq , N_ID PDr , N_ID PDs and N_ID PDu is drawn. Node N_ID PDp connects to node N_ID via edge E_CA1 CA1 Node N_ID PDq and N_ID PDr connects to node N_ID via two edges E_CA4 CA4Node N_ID PDs connects to node N_ID via edge E_CA5 CA5 Node N_ID PDu connects to node N_ID via edge E_CA3 CA3 is tied to.

[0031] As already explained, the common tracking ID PD The data of the section time bt and the representative data of the Re-ID feature of the image sequence are combined in the re-identification process. In the embodiment, the representative data of the Re-ID feature is used to perform a re-identification process of the person captured in at least two cameras. In this re-identification process, the same process as the comparison of the Re-ID feature between different image sequences in the section time bt is performed. However, while the comparison of the Re-ID feature between the image sequences is performed for one camera, the comparison of the representative data of the Re-ID feature is performed for two cameras. If the representative data of the Re-ID feature between the two cameras is similar, it is determined that the pedestrian PD captured separately by these cameras is the same person.

[0032] When it is determined that the pedestrians PD captured by the two cameras are the same person, the graph generation processing unit 14 uses the common tracking IDs that have been assigned to these pedestrians PD separately. PD Represents the node N_ID PD are connected via an edge E. In the graph GRH1 shown in FIG. 3, the pedestrians PDp and PDq are determined to be the same person, and the pedestrians PDr and PDs are determined to be the same person. Therefore, the node N_ID PDp and node N_ID PDq are connected via edges E_IDp-q, and node N_ID PDr and node N_ID PDs are connected via edge E_IDr-s. The meaning of edges E_IDp-q and E_IDr-s is “SAME PERSON.”

[0033] In the graph GRH1 shown in FIG. 3, it is also determined that pedestrians PDq and PDr are the same person. This determination is based on the result of a comparison of Re-ID features between multiple image sequences that are different in the interval time bt, performed for a single camera. For this reason, PDq and node N_ID PDr are connected via an edge E_IDq-r, which means "SAME PERSON".

[0034] Fig. 4 is a diagram showing a detailed configuration example of a graph generated by the graph generation processing unit 14. In the graph GRH2 shown in Fig. 4, a node N representing additional information ADD is added to the graph GRH1 shown in Fig. 3. This additional information ADD includes a common tracking ID PD The additional information ADD is the image IM of the pedestrian PD. PD , Appearance characteristics of pedestrian PD AP PD , Pedestrian PD's behavior AC PD , pedestrian PD's face image IMF PD Examples include:

[0035] Pedestrian PD image IM PD was used to extract the Re-ID features. Examples of the appearance features of the pedestrian PD include the color, clothing, and body type of the pedestrian PD. These appearance features are estimated using a pre-trained appearance model. Examples of the actions of the pedestrian PD include "walking," as well as "carrying" and "opening" actions that the pedestrian PD takes with respect to stationary objects such as luggage. These actions are estimated using a pre-trained action model. These actions also include interaction actions such as "conversation" and "handing over" that multiple people take together. The facial image IMF of the pedestrian PD PD IM is an image of a pedestrian PD PD The face image may be an image in which the face portion has been trimmed from the original image, or may be an externally provided face image for tracking processing by the tracking processing unit 17 (described later).

[0036] In the graph GRH2 shown in FIG. 4, there is a node N_IDPDp Three edges E_ID extending from PDp Next, the image IM of the pedestrian PDp PDp Represents the node N_IM PDp(bt1) and N_IM PDp(bt2) and node N_AC which represents the behavior of pedestrian PDp. PDp(bt1) In addition, the node N_ID PDq Three edges E_ID extending from PDq Next, the face image IMF of pedestrian PDq PDq Node N_IMF PDq and the image IM of the pedestrian PDq PDq Represents the node N_IM PDq(bt3) and node N_AC which represents the behavior of pedestrian PDq. PDq(bt3) and are arranged.

[0037] Although detailed explanation is omitted, in the graph GRH2, the node N_ID PDr , N_ID PDs and N_ID PDu For each pedestrian PD, a node N representing additional information ADD is connected to a node N_ID via an edge E. PD In addition, the node N_ID PDs and node N_ID PDu is an edge E_IA that means the interaction action "TALKING" PDs-PDu Also, the node N_ID PDs to edge E_ID PDs Node N_AP connected via PDs(bt6) is the appearance feature AP of pedestrian PDs in section time bt6 PDs Represents.

[0038] 2-2. Probability distribution calculation section The probability distribution calculation unit 15 calculates a common tracking ID PD The presence probability distribution of an arbitrary pedestrian PD is calculated by statistically analyzing the history data of the same pedestrian PD to which the presence probability distribution calculation unit 15 is assigned. The calculated presence probability distribution is stored in the probability distribution DB 18. There are two examples of the calculation of the presence probability distribution by the probability distribution calculation unit 15. These calculation examples will be described below.

[0039] In the first calculation example, the historical data subjected to the statistical analysis is a set of frames FR of a single camera (frame group), PD The coordinates CD of the same pedestrian PD are given PD The historical data is a common tracking ID that is assigned to each pedestrian PD included in multiple image sequences when the Re-ID features are similar between these image sequences. PD A common tracking ID after integration when PD The coordinates CD of the pedestrian PD to which PD Data from the above is also acceptable.

[0040] FIG. 5 is a diagram for explaining a first calculation example. In FIG. 5, a plurality of image sequences IMS of pedestrians PDx and PDy explained in FIG. 2 are captured in one frame FR(ID CAi ,TB) is shown above. CAi , TB) are shown for convenience of explanation. The section time TB is divided into time widths longer than the section time bt, for example, from section time bt1 to section time bt7. Examples of the time width of the section time TB include 30 minutes, 2-3 hours, 12 hours, and 1 day. Note that it is desirable that information on the start time and end time of the section time TB be associated with the data of the timestamp ts.

[0041] As already explained, in the detection and extraction process by the graph generation processing unit 14, the image IM PDx and I.M. PDy is extracted. Also, image IM PDx and I.M. PDy The coordinates CD of the pedestrians PDx and PDy in the frame FR of the extracted image are PD Therefore, the coordinates CD in the section time TB are PDx and CDs PDy It can be said that each of the time series data indicates the movement history of the pedestrians PDx and PDy in this section time TB.

[0042] Therefore, in the first calculation example, the movement vectors of multiple pedestrians PD including pedestrians PDx and PDy are calculated based on the movement history of these pedestrians PD. Then, by statistically analyzing these movement vectors, a presence probability distribution (PD, TB, CD) that indicates the probability that any pedestrian PD is present at coordinates CD in the frame FR of the camera CAn in the section time TB is calculated. Note that, when the pedestrian PD is identified, from the viewpoint of privacy protection, it is desirable to obtain consent from the pedestrian PD before using the presence probability distribution (PD, TB, CD) as input data for the calculation.

[0043] In the second calculation example, the history data used for the statistical analysis is data on the movement history of a pedestrian PD captured separately on at least two cameras when the pedestrian PD is the same person between these cameras. As already explained, in the graph generation processing unit 14, in the re-identification process of a person captured on two cameras, a determination is made using representative data of the Re-ID feature. Then, when it is determined that a pedestrian PD captured separately on two cameras is the same person, the common tracking IDs assigned separately to these pedestrians PD are used. PD Represents the node N_ID PD are connected via edge E.

[0044] Here, each node N_ID PD is a node N_ID that represents each ID of two cameras. CA Each common tracking ID is also linked to PD is an image IM of a pedestrian PD containing data at timestamp ts. PD Therefore, each node N_ID assigned to a pedestrian PD determined to be the same person PD can be ordered in chronological order, and the chronologically ordered nodes N_ID PD It can be said that indicates the movement history between these cameras of the pedestrian PD who is determined to be the same person.

[0045] FIG. 6 is a diagram for explaining a second calculation example. In FIG. 6, a graph GRH3 is drawn. The configuration of the graph GRH3 is basically the same as that of the graph GRH1 described in FIG. 3. The difference between the two is the presence or absence of a direction of the edges E_IDp-q, E_IDq-r, and E_IDr-s. The direction of the edge E in the graph GRH3 represents that the pedestrians PDp and PDq, who are the "SAME PERSON", have moved from the position of the camera CA1 to the position of the camera CA4. It also represents that the pedestrians PDq and PDr, who are the "SAME PERSON", have moved within the frame FR of the camera CA4, and that the PDr and PDs, who are the "SAME PERSON", have moved from the position of the camera CA4 to the position of the camera CA5.

[0046] The direction information of the edge E is the node N_ID that represents the pedestrians PDp, PDq, PDq, and PDr that are determined to be the same person. PD The person can be identified from the data of the time stamp ts. Therefore, in the second calculation example, a statistical analysis of the movement history between at least two cameras is performed for multiple pedestrians PD that are determined to be the same person. This allows the presence probability distribution (PD, CAn) that represents the probability that any pedestrian PD moving between at least two cameras is present in the frame FR of the camera CAn to be calculated.

[0047] 2-3. Data sorting section The data organizing unit 16 periodically organizes the video data VD_CAn. The organization by the data organizing unit 16 prevents the memory capacity in the storage device 12 from being overloaded. For example, the data organizing unit 16 deletes the video data VD_CAn after a certain period of time has passed since it was stored in the storage device 12. In another example, the data organizing unit 16 compresses the video data VD_CAn after a certain period of time has passed since it was stored in the storage device 12, and deletes the compressed video data VD_CAn after a further certain period of time has passed since this compression.

[0048] The data organizing unit 16 also organizes graph-accompanying data (such as data of Re-ID features) generated in association with various processes by the graph generation processing unit 14. The data organizing unit 16, for example, deletes graph-accompanying data for which a certain period of time has passed since it was stored in the graph DB 18. In another example, the data organizing unit 16 compresses the graph-accompanying data after a certain period of time has passed since it was stored in the graph DB 18, and deletes the compressed graph-accompanying data after another certain period of time has passed since this compression.

[0049] However, the data reduction unit 16 excludes data used in the calculation of the presence probability distribution from the graph accompanying data. The reason for this is that, in order to improve the accuracy of the presence probability distribution, the coordinates CD of the pedestrian PD in the frame FR of the extracted image generated in the detection and extraction process by the graph generation processing unit 14 are PD (First calculation example) and each node N_ID assigned to a pedestrian PD who was determined to be the same person in the re-identification process of a person captured by two cameras. PD This is because the accumulation of (Second Calculation Example) is required.

[0050] 2-4.Tracking processing section The tracking processing unit 17 tracks the target using the graph GPH stored in the graph DB 18. Figures 7 and 8 are diagrams for explaining an example of the tracking process performed by the tracking processing unit 17. In the example shown in Figure 7, an image IM of the target is TGT The graph DB 18 is referenced with the query Q (input information). TGT is selected from the frame of the pedestrian PD used to extract the Re-ID features. TGT is the face image IMF of the target TGT In this case, the face image IMF TGT may be cropped from the frame of pedestrian PD used to extract the Re-ID features, or may be provided from outside the tracking system.

[0051] In the example shown in FIG. TGT and its image IM PDThe tracking data TRC of the pedestrian PD that matches the image IM is output as a search result R. TGT Matching images IM PD Represents the node N_IM PD and each node N_IM PD Node N_ID connected to via edge E PD and each node N_ID PD Node N_ID connected to via edge E CA In another example, each node N_IM PD Additional information ADD (i.e., appearance feature AP PD , behavioral AC PD A set of nodes N representing the above-mentioned nodes, such as the nodes 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132,

[0052] In the example shown in FIG. 8, the date and time DT, the location AR, and the image IM TGT In this example, the graph DB 18 is referenced with the query Q (input information). TGT and its image IM PD The tracking data TRC of the pedestrian PD that matches the query Q is output as a search result R. However, in the example shown in Fig. 8, since the date and time DT and the location AR are included in the query Q, the tracking data TRC with limited date and time and location is output.

[0053] The tracking processing unit 17 predicts the movement of a person captured in a frame FR of a camera CAn within the frame FR, using the presence probability distribution stored in the probability distribution DB 19. Figs. 9 and 10 are diagrams for explaining an example of the prediction process performed by the tracking processing unit 17. In the example shown in Fig. 9, the probability distribution DB 19 is referenced with the camera CAn, the date and time DT, and the coordinates CD as a query Q (input information). In the example shown in Fig. 9, tracking data TRC_Q of a pedestrian PD matching the query Q is output as a search result R. The tracking data TRC_Q includes, for example, information on the movement line within the frame FR of the pedestrian PD matching the query Q.

[0054] The example shown in Fig. 9 is also a search example using the presence probability distribution (PD, TB, CD) obtained by the first calculation example described in Fig. 5. In this regard, the example shown in Fig. 10 is a search example using the presence probability distribution (PD, CAn) obtained by the second calculation example described in Fig. 6. In the example shown in Fig. 10, a probability distribution DB19 is referenced with a camera CAn and a date and time DT as a query Q (input information). In the example shown in Fig. 10, the tracking data TRC_Q output as the search result R includes information on the movement line between at least two cameras of the pedestrian PD captured by the camera CAn at the date and time DT. [Explanation of symbols]

[0055] 10 Management server, 11 Processor, 12 Storage device, 14 Graph generation processing unit, 15 Probability distribution calculation unit, 16 Data reduction unit, 17 Tracking processing unit, 18 Graph DB, 19 Probability distribution DB, CA, CA1-CA6, CAi, CAj, CAn Camera, FR Frame, PD, PDp-PDs, PDu-PDz Pedestrian, VD, VD_CA1-CA6, VD_CAn Video data, AC PD behavior, AP PD Appearance features, IM PD ,IM PDp ,IM PDq ,IM PDx -IM PDz ,IM TGT Image: IMF PD ,IMF PDq ,IMF TGT Facial image, IMS, IMS PDx ,IMS PDy Image sequence,E,E_CA1,E_CA3-CA7,E_CA1-3,E_CA1-4,E_CA3-5,E_CA4-5,E_CA5-6,E_CA3-7,E_ID PDp -E_ID PDs ,E_ID PDu ,E_IDp-q,E_IDq-r,E_IDr-s,E_IDq-w,E_IDu-w,E_IA PDs - PDu Edge, N, N_AC PDp -N_AP PDs ,N_ID CA ,N_IDCA1 -N_ID CA7 ,N_ID PD ,N_ID PDp -N_ID PDs ,N_ID PDu -N_ID PDw ,N_IMF PDq Node, ADD additional information, GRH, GRH1-GRH3 graph

Claims

1. A storage device in which video data captured by a single camera is stored; a processor configured to perform data processing based on the video data; Equipped with The processor, in the data processing, A graph is generated that includes at least two nodes and at least one edge that indicates a relationship between the at least two nodes; In the graph, the node representing the single camera and a node representing an identification number given to a moving object captured in the video data acquired by the single camera are connected via an edge; The identification number includes a common identification number that is assigned to the same moving object captured in the video data acquired by the single camera, The processor further includes, in the data processing, calculating a probability distribution of the presence of any one of the same moving objects in a group of frames of the video data acquired by the single camera by analyzing coordinate histories of the one of the same moving objects, each of which is assigned a common identification number, in each group of frames of the video data in which the one of the same moving objects is captured; Among the graph-accompanying data accompanying the generation of the graph, data other than the coordinate history data of the plurality of identical moving objects used in the calculation of the existence probability distribution is compressed or deleted from the storage device. A moving object tracking system comprising:

2. a storage device in which each image data captured by at least two cameras is stored; a processor configured to perform data processing based on each of the video data; Equipped with The processor, in the data processing, A graph is generated that includes at least two nodes and at least one edge that indicates a relationship between the at least two nodes; In the graph, a node representing a single camera included in the at least two cameras and a node representing an identification number given to a moving object captured in video data acquired by the single camera are connected via at least one edge; The identification number includes a common identification number that is assigned to the same moving object captured in the video data acquired by the single camera, At least two of the single cameras are connected to each other through at least one edge indicating a relationship between the single cameras, among the nodes representing the single cameras, the nodes having the relationship; At least two nodes that represent the common identification numbers and that are recognized as being the same moving object are connected to each other via at least one edge that indicates that at least two moving objects captured in each of the image data acquired by the at least two single cameras are the same moving object; The processor further includes, in the data processing, calculating a presence probability distribution of any identical moving object between the at least two single cameras by analyzing movement histories of the identical moving objects between the at least two single cameras that have acquired the video data in which the identical moving objects, each of which has been assigned a common identification number, are captured; Among the graph-accompanying data accompanying the generation of the graph, data other than the data on the movement history of the plurality of identical moving objects used in the calculation of the existence probability distribution is compressed or deleted from the storage device. A moving object tracking system comprising:

Citation Information

Patent Citations

  • Graph display device, graph display program and computer readable storage medium with graph display program stored

    JP2017021753A

  • Action pattern analysis support device, action pattern analysis support system, and action pattern analysis support method

    JP2020107171A

  • Image sensor system, calculation device, sensing method, and program

    JP2020149422A

  • Medical information processing device, system and method

    JP7066362B2