Real-time tracking system

Abstract

This paper proposes a technology that can improve tracking accuracy in a real-time tracking system that tracks a target in an image captured by a camera in real time. [Solution] The real-time tracking system performs real-time tracking of a target in images captured by one or more cameras. During this process, it retrieves a set of images from an image database for a specified past period and performs re-tracking on the retrieved image set. The real-time tracking system then compares the tracking result for the image set with the re-tracking result from the re-tracking process. If the tracking result for the image set and the re-tracking result differ, the system is configured to reflect the re-tracking result in the tracking result.

Description

Technical Field

[0001] This disclosure relates to a technique for performing real-time tracking of a target in an image captured by a camera.

Background Art

[0002] Patent Document 1 discloses a technique related to a tracking support device that supports the work of a monitor who performs tracking of a moving object by displaying captured images for each of a plurality of cameras on a display device. The tracking support device calculates an evaluation value representing the degree of identity between moving objects based on the pursuit information of the moving objects detected from the captured images for each of the plurality of cameras. Then, the tracking support device sequentially identifies the cameras that image the moving object to be tracked, and causes the display device to display a tracking target confirmation screen that displays the captured image of the moving object with the highest evaluation value for each of the identified cameras.

[0003] In addition, as documents indicating the technical level of this technical field, there are the following Patent Documents 2 and 3.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology for tracking targets in images captured by cameras in real time is being used in VR environments such as the metaverse, where real-time capabilities are required, and the need for such technology is increasing. On the other hand, ensuring tracking accuracy is a challenge in real-time tracking, and there is a need for technologies that can improve tracking accuracy. One of the purposes of this disclosure is to provide a technology that can improve the tracking accuracy of real-time tracking. [Means for solving the problem]

[0006] One aspect of this disclosure relates to a real-time tracking system. The real-time tracking system comprises one or more cameras, one or more processors that perform real-time tracking processing to track a target in images captured by the one or more cameras in real time, an image database that manages images captured by the one or more cameras up to the present time, and one or more storage devices that store the tracking results of the target by the real-time tracking processing. The one or more processors are configured to retrieve a group of images from the image database for a specified past period while the real-time tracking processing is being performed, and to perform re-tracking processing to track the target in the retrieved group of images. Furthermore, the one or more processors are configured to compare the tracking results for the group of images with the re-tracking results of the target by the re-tracking processing, and to reflect the re-tracking results in the tracking results when the tracking results for the group of images and the re-tracking results differ. [Effects of the Invention]

[0007] According to this disclosure, during the execution of real-time tracking, a set of images from a specified past period is retrieved from the image database, and re-tracking is performed on the retrieved image set. When the tracking result for the image set differs from the re-tracking result, the re-tracking result is reflected in the tracking result. This allows for the appropriate correction of errors in the tracking result using the more accurate re-tracking result. As a result, the tracking accuracy of real-time tracking can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating an example configuration of a real-time tracking system according to an embodiment. [Figure 2] This is a conceptual diagram illustrating the overview of the processing performed by the server in the real-time tracking system according to the embodiment. [Figure 3] This flowchart shows the processing flow related to the retracking process performed by the server. [Figure 4] This is a conceptual diagram illustrating specific examples of the timing and specified period for the retracking process. [Modes for carrying out the invention]

[0009] Embodiments of this disclosure will be described below with reference to the attached drawings. In each drawing, identical or corresponding components are denoted by the same reference numerals, and their descriptions are simplified or omitted.

[0010] 1. Real-time tracking system Figure 1 is a diagram illustrating an example configuration of the real-time tracking system 10 according to this embodiment. The real-time tracking system 10 tracks target 1 in real time within a predetermined area. Examples of target 1 include moving objects such as people (pedestrians), automobiles, animals, and robots. Examples of predetermined areas include streets, buildings, facilities, and shops. In the following explanation, we will consider the case where target 1 is a person. However, for generalization purposes, you can replace "person" with "moving object" in the following explanation.

[0011] The real-time tracking system 10 includes a server 100, one or more cameras 200, and an output device 300. The server 100 is communicatively connected to one or more cameras 200 and the output device 300 via a communication network. One or more cameras 200 are installed in a predetermined area.

[0012] Server 100 is a computer that provides the functions of the real-time tracking system 10. Server 100 acquires image IMGs captured by one or more cameras 200. Server 100 then performs real-time tracking of target 1 in the acquired image IMGs.

[0013] Server 100 may perform tracking of target 1 across multiple different cameras 200. This type of target 1 tracking is called multi-camera tracking. In particular, when there are multiple targets 1, it is also called MTMC (multi-target multi-camera tracking). In multi-camera tracking, re-identification is performed to identify the same target 1 from the image IMG of each camera 200. For re-identification, server 100 extracts feature quantities of target 1 (hereinafter referred to as "ReID features") based on the partial image corresponding to target 1 in the image IMG. Feature extraction is typically performed using a ReID model based on machine learning. The ReID model may also be a Transformer-based model. Server 100 then calculates the similarity between target 1s in the image IMG of each camera 200 based on the ReID features of target 1. If the similarity between target 1s exceeds a threshold, server 100 determines that those target 1s are identical. Unique identification information is assigned to identical target 1s. This re-identification process enables multi-camera tracking.

[0014] Server 100 may consist of a single node or multiple nodes. More specifically, Server 100 comprises one or more processors 110 (hereinafter simply referred to as "processor 110") and one or more storage devices 120 (hereinafter simply referred to as "storage devices 120"). Processor 110 performs various processes. Processor 110 consists of, for example, a general-purpose processor, an application-specific processor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), an integrated circuit, a conventional circuit, and one or more combinations thereof. Processor 110 can also be called processing circuitry.

[0015] The storage device 120 stores various information necessary for the processor 110 to execute processing. The storage device 120 is composed of recording media such as RAM (Random Access Memory), ROM (Read Only Memory), SSD (Solid State Drive), HDD (Hard Disk Drive), etc. The storage device 120 stores a computer program 121 that can be executed by the processor 110. The computer program 121 consists of multiple instruction codes that describe the processing to be executed by the processor 110. The computer program 121 is recorded on a computer-readable recording medium. The functions of the server 100 are realized through the cooperation of the processor 110, which executes the computer program 121, and the storage device 120.

[0016] The storage device 120 further stores an image database 122 and tracking results 123. The image database 122 manages images (IMG) captured up to the present time by one or more cameras 200. The tracking results 123 are the results of tracking target 1. The tracking results 123 are, for example, data in which tracks of target 1 are managed for each unique identifier. A track typically consists of a time-series dataset of bounding boxes indicating the position of target 1. A track may further include a time-series dataset of ReID features of target 1.

[0017] The output device 300 presents the output from the server 100 to the user through display and sound. The output device 300 consists of a display, speaker, touchscreen, etc. The user terminal of the user communicating with the server 100 (for example, a smartphone) may also function as the output device 300. The server 100 presents the user with at least real-time tracking results 123 via the output device 300.

[0018] 2. Overview of the processes performed by the server Server 100 executes "real-time tracking processing" for real-time tracking of target 1 in image IMG. Further, while executing the real-time tracking processing, server 100 acquires an image group of a past specified period from image database 122 and executes "re-tracking processing" for tracking target 1 in the acquired image group. And when the tracking result by the re-tracking processing (hereinafter referred to as "re-tracking result") is different from tracking result 123, server 100 reflects the re-tracking result in tracking result 123. Since the re-tracking processing performs tracking in a collective image group, the tracking accuracy is higher compared to the real-time tracking processing. Therefore, by reflecting the re-tracking result in tracking result 123, it becomes possible to appropriately correct the error of tracking result 123 by the real-time tracking processing. As a result, it is possible to realize real-time tracking system 10 with improved tracking accuracy.

[0019] FIG. 2 is a conceptual diagram for explaining an outline of processing executed by server 100 in real-time tracking system 10. The example shown in FIG. 2 shows a case of tracking target 1 in image IMG captured between time t1 and time t6. In the example shown in FIG. 2, two targets 1-A and 1-B are detected in image IMG. The example shown in FIG. 2 shows a case of particularly tracking target 1-A.

[0020] In the example shown in FIG. 2, it can be seen that from time t1 to time t3, the target 1-A is correctly tracked by the real-time tracking process. On the other hand, at time t4, occlusion (hiding) occurs due to target 1-B, and target 1-B is being tracked incorrectly. As a result, the tracking result 123 from time t4 to time t5 is an incorrect result. Here, in the example shown in FIG. 2, the server 100 executes a re-tracking process with the period from time t2 to time t5 as the specified period. The re-tracking result 124 can correctly track the target 1-A until time t5. At the time point of time t5, since the tracking result 123 and the re-tracking result 124 are different, the server 100 reflects the re-tracking result 124 in the tracking result 123. Then, the server 100 executes a real-time tracking process at time t6. By reflecting the re-tracking result 124, it can be seen that the tracking result 123 at time t6 can correctly track the target 1-A.

[0021] In this way, the server 100 executes a re-tracking process during the execution of the real-time tracking process. The real-time tracking process is a process executed online, and the re-tracking process is a process executed offline. And when the tracking result 123 is different from the re-tracking result 124 by the re-tracking process, the server 100 reflects the re-tracking result 124 in the tracking result 123. Thereby, the error of the tracking result 123 can be corrected appropriately, and the tracking accuracy of the real-time tracking process can be improved. In this embodiment, the tracking methods of the real-time tracking process and the re-tracking process are not particularly limited. For example, ByteTrack is used as the tracking algorithm.

[0022] The above describes the case where target 1 is tracked within an image IMG captured by a single camera 200, but the process described above is similarly applicable to multi-camera tracking and MTMC. In this case, the tracking result 123 and the retracking result 124 may each include a determination result indicating that the target 1 is the same among the image IMGs of each camera 200. The comparison between the tracking result 123 and the retracking result 124 may include a comparison of the determination results indicating that they are the same target 1.

[0023] 3. Processing Flow Figure 3 is a flowchart showing the processing flow related to the retracking process performed by the server 100 (more specifically, the processor 110). The processing flow shown in Figure 3 is repeatedly executed at predetermined processing cycles while the real-time tracking process is running.

[0024] First, in step S100, the server 100 determines whether or not it is time to execute the retracking process. The timing of the retracking process can be determined from several perspectives. Specific examples of the timing of the retracking process will be described later.

[0025] If it is not time to perform the retracking process (step S100; No), the server 100 terminates the current process without performing the retracking process. If it is time to perform the retracking process (step S100; Yes), the process proceeds to step S110.

[0026] In step S110, the server 100 retrieves a set of images from the image database 122 for a specified past period. The specified period can be determined in relation to the timing of the retracking process. Specific examples of the specified period will be described later.

[0027] Next, in step S120, the server 100 performs a retracking process on the acquired image set.

[0028] Next, in step S130, the server 100 compares the tracking result 123 for the image group with the retracking result 124 obtained through the retracking process.

[0029] Next, in step S140, the server 100 determines whether the tracking result 123 and the retracking result 124 for the image group are different.

[0030] If the tracking result 123 and the retracking result 124 for the image group are identical (step S140; No), the server 100 terminates the process without modifying the tracking result 123.

[0031] If the tracking result 123 and the retracking result 124 for the image group differ (step S130; Yes), the server 100 reflects the retracking result 124 in the tracking result 123 (step S150). After that, the server 100 terminates the current process.

[0032] 4. Specific examples of the timing and specified period for the retracking process. Below, with reference to Figure 4, we will explain some specific examples of the timing and specified period for the retracking process.

[0033] 4.1 Example 1 Figure 4(A) is a conceptual diagram showing a first example of the timing and specified period for the retracking process. In the first example, the server 100 executes the retracking process every predetermined time interval. The predetermined time interval can be determined as appropriate. In the example shown in Figure 4(A), the retracking process is executed at times t2 and t3. In the first example, the specified period is the time interval from a predetermined time in the past to the present. In the example shown in Figure 4(A), the retracking process at time t2 has a specified period from time t1 to time t2. The retracking process at time t3 has a specified period from time t2 to time t3.

[0034] In the first example, retracking is performed on all image IMGs that are subject to real-time tracking. This allows the tracking results 123 for all image IMGs to be compared with the retracking results 124, and any errors in the tracking results 123 can be corrected.

[0035] 4.2 Second Example Figure 4(B) is a conceptual diagram showing a second example of the timing and specified period for the retracking process. In the second example, compared to the first example, two predetermined times are provided: a first predetermined time and a second predetermined time. Here, the second predetermined time is set to be longer than the first predetermined time. In the second example, the server 100 executes the retracking process every time the first predetermined time has elapsed. Furthermore, the server 100 executes the retracking process every time the second predetermined time has elapsed. In the retracking process executed every time the first predetermined time has elapsed (hereinafter referred to as the "first retracking process"), the specified period is from a point in the past by the first predetermined time to the present time. On the other hand, in the retracking process executed every time the second predetermined time has elapsed (hereinafter referred to as the "second retracking process"), the specified period is from a point in the past by the second predetermined time to the present time.

[0036] In the second example, the first and second retracking processes are executed in parallel for all images (IMG) that are subject to real-time tracking. Offline tracking of target 1 becomes more accurate as the size of the target image group increases. In other words, the retracking result 124 from the second retracking process is more accurate than the retracking result 124 from the first retracking process. Therefore, the tracking accuracy can be further improved by the retracking result 124 from the second retracking process. On the other hand, since the second predetermined time is longer than the first predetermined time, the execution interval of the first retracking process is shorter than that of the second retracking process. Therefore, the retracking result 124 from the first retracking process can prevent the correction interval of the tracking result 123 from becoming excessively long. Thus, in the second example, by executing the first and second retracking processes in parallel, it is possible to achieve both improved tracking accuracy and sufficient correction intervals. The second example may be further modified so that more retracking processes are executed in parallel. For example, the retracking process may be performed at intervals of a third predetermined time that are longer than the second predetermined time.

[0037] 4.3 Third Example Figure 4(C) is a conceptual diagram showing a third example of the timing and specified period for the retracking process. In this third example, the server 100 (more specifically, the processor 110) determines a "low confidence period" in which the reliability of the tracking results 123 from the real-time tracking process is low. The server 100 may be configured to determine a low confidence period as the time while a specific scene is being detected. Examples of specific scenes in this case are (1) to (3) below. (1) Scenes where Target 1 in the image (IMG) is excessively numerous. (2) A scene in which multiple targets 1 that look similar (e.g., clothing) exist within the image IMG. (3) Scenes in tracking result 123 where the ReID feature of target 1 changes significantly over time.

[0038] In scene (1), the tracking result 123 is considered to have low reliability because occlusion is likely to cause errors in the tracking result 123. Scene (1) can be detected, for example, when the number of target 1 detected in the image IMG is greater than or equal to a predetermined number. In scene (2), the tracking result 123 is considered to have low reliability because the ReID features are similar to each other, which is likely to cause errors in the tracking result 123. Scene (2) can be detected, for example, when the similarity between the ReID features of each target 1 is greater than or equal to a predetermined value. In scene (3), the tracking result 123 is considered to have low reliability because the ReID features of the same target 1 usually change little over time. Scene (3) can be detected when the amount of change in the time-series data of the ReID features of target 1 in tracking result 123 is greater than or equal to a predetermined value.

[0039] As described above, in the third example, server 100 determines the low confidence period. Then, in the third example, server 100 performs retracking processing with the low confidence period as the specified period. Typically, server 100 performs retracking processing immediately after the end of the low confidence period. However, server 100 may also perform retracking processing after a certain period of time has elapsed after the end of the low confidence period.

[0040] In the third example, retracking is not performed on all images (IMG), but only during the low-confidence period. This allows for reduced processing costs while maintaining tracking accuracy. [Explanation of symbols]

[0041] 1. Target 10. Real-time tracking system 100 servers 200 Cameras 110 processors 120 Storage device 121 Computer Programs 122 Image Database 123 Tracking Results 124 Re-tracking results

Claims

1. One or more cameras, One or more processors that perform real-time tracking processing to track a target in real time within an image captured by the one or more cameras, An image database for managing the images captured by the one or more cameras up to the present time, and one or more storage devices for storing the tracking results of the target obtained by the real-time tracking process, Equipped with, The one or more processors described above are: While the real-time tracking process is being executed, a re-tracking process is performed to retrieve a set of images from the image database for a specified past period and to track the target using the retrieved set of images. The tracking results for the aforementioned image group and the retracking results of the target obtained by the retracking process are compared, When the tracking result and the retracking result for the aforementioned image group differ, the retracking result is reflected in the tracking result. It is configured in such a way A real-time tracking system characterized by the following features.

2. A real-time tracking system according to claim 1, The one or more processors are configured to execute the retracking process each time a first predetermined time has elapsed, with the period from a point in the past by the first predetermined time to the present time being the specified period. A real-time tracking system characterized by the following features.

3. A real-time tracking system according to claim 2, The one or more processors are configured to execute the retracking process each time a second predetermined time, which is longer than the first predetermined time, has elapsed, with the period from a point in the past by the second predetermined time to the present time set as the specified period. A real-time tracking system characterized by the following features.

4. A real-time tracking system according to claim 1, The one or more processors described above are: Determine the low-confidence period during which the reliability of the aforementioned tracking results decreases. The retracking process is executed with the aforementioned low-confidence period as the specified period. It is configured in such a way A real-time tracking system characterized by the following features.

Images