A multi-target tracking method and system based on a query mechanism

Abstract

The present application relates to a kind of multi-target tracking method and system based on query mechanism, belong to computer vision image processing technical field.Firstly, the scene video containing moving target is acquired.From the video frame obtained, the detection query of interest is acquired, for detecting the target of interest in current frame.Finally, the detection query and the last frame tracking result are used to track the target of current frame.The system includes video frame acquisition module, feature extraction module, detection query acquisition module and target query module.The present application simplifies the process of multi-target tracking, reduces the complexity of data association, improves the ease of use of multi-target tracking algorithm, improves the practicality of multi-target tracking method, and provides technical support for high-level video content analysis and processing.At the same time, the present application allows multi-target tracking to track the target of interest in an automatic identification or human designated manner, further improving the practicality.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-target tracking method and system based on a query mechanism, in particular to a method and system for treating a target to be tracked as a query and querying the target of interest in a given video through an attention mechanism, thereby realizing target tracking, belonging to the technical field of computer vision image processing. BACKGROUND

[0002] Multi-target tracking belongs to the category of high-level image processing of computer vision, and its purpose is to obtain the running track and identity information of the target of interest in a given video. In the tracking process, the target track is required to be as continuous as possible without interruption, and the identity information of the target does not change.

[0003] As a high-level image processing technology, multi-target tracking provides technical support for video content analysis and processing with broad application prospects. For example, this technology can be used for intelligent security monitoring, automatic driving, etc. Therefore, multi-target tracking has been widely concerned by researchers.

[0004] Multi-target tracking methods are usually realized based on target detection, and can be divided into offline multi-target tracking methods and online multi-target tracking methods according to the number of video frames processed at the same time in the tracking process. Among them, the offline multi-target tracking method usually adopts a sliding window mechanism, processes multiple video frames at a time, so the time delay is high, and the size of the window has no fixed limit, and can be the length of the entire video under the condition that the hardware memory is allowed. The online multi-target tracking method only processes one video frame at a time, and has lower time delay. Compared with the online multi-target tracking method, the offline multi-target tracking method can utilize more information when processing video frames each time, and allows some complex post-processing, so the tracking performance is generally better. Although the performance of the offline multi-target tracking method is generally better, the online multi-target tracking method has a wider range of applications due to its lower time delay, and has been paid more and more attention by researchers in recent years.

[0005] The existing online multi-target tracking method based on target detection implementation generally includes the following steps: (1) target detection: using a target detector to obtain a target of interest in a video frame, such as a pedestrian, a vehicle, etc.; (2) data association: associating and matching a newly detected target in a current frame with a tracked target, and obtaining the identity information of the newly detected target in the current frame by assigning the newly detected target to the tracked target, so as to realize tracking. Since target detection is not the focus of multi-target tracking research, the existing online multi-target tracking method mainly processes data association. When data association is performed, a linear assignment algorithm (such as the Hungarian algorithm) or a greedy algorithm is generally used to assign the newly detected target to the tracked target at the least cost. Therefore, the cost between the newly detected target and the tracked target greatly affects the matching result. On the other hand, in order to handle newly appearing targets and disappearing targets, some threshold values are generally set in the above matching process. When the cost is greater than the threshold value, the mutual matching between the targets is prohibited, at this time, the newly detected target that is not matched is considered to be a newly appearing target, and a new identity information is assigned to the newly detected target. The tracked target that is not matched is considered to be a disappearing target, and the information of the tracked target is deleted or cached, and the running track of the tracked target is no longer output. Obviously, the above data association and matching process is relatively cumbersome, and involves a lot of empirical parameter adjustment (such as cost calculation method and cost threshold value). These cumbersome and redundant settings result in complex use of the multi-target tracking method, and it is difficult to deploy in practical applications. SUMMARY

[0006] The purpose of the present application is to overcome the defects and deficiencies of the prior art, and to solve the technical problems of complex use and difficult deployment caused by the cumbersome and redundant data association process in multi-target tracking. A multi-target tracking method and system based on a query mechanism are creatively proposed. The present application is implemented based on a deep learning network, improves the ease of use of the multi-target tracking method, simplifies the process of multi-target tracking, and realizes multi-target tracking without complex data association. The present application can track the target of interest in an automatic recognition and human designation manner.

[0007] To achieve the above purpose, the present application adopts the following technical solutions.

[0008] A multi-target tracking method based on a query mechanism includes the following steps:

[0009] Step 1: using a shooting device to obtain a scene video containing a moving target.

[0010] The shooting device can be a common monitoring camera, a professional video recording device, or a mobile phone equipped with a camera, etc.

[0011] Step 2: obtaining a detection query of interest from the obtained video frame, for detecting a target of interest in the current frame.

[0012] The detection query can be automatically identified and obtained by the tracking method, or it can be obtained by a manual specification in the initial frame.

[0013] For automatic identification and acquisition methods, deep learning-based classifiers and regressors can be used to process deep features to obtain the content and location information of the query. The classifiers and regressors can be implemented using fully connected layers.

[0014] For manually specified acquisition methods, the location information of the target of interest needs to be provided manually. The detection query content is obtained through preprocessing, that is, the detection query content is directly extracted from the deep features.

[0015] Step 3: Use the detection query and the tracking results of the previous frame to track the target in the current frame.

[0016] The tracking result from the previous frame needs to be further processed into a tracking query similar to the detection query. The query content in the tracking query is the query content of the target in the previous frame, and the query position is the position of the target in the previous frame.

[0017] For tracking methods that automatically identify targets of interest, it is necessary to track newly emerging targets of interest. Therefore, detection queries and tracking queries are used together to query targets in the current frame, thereby achieving target tracking. In practice, most of the bounding boxes obtained by the detection query are duplicates of those obtained by the tracking query (when the intersection-union ratio between two bounding boxes is greater than a given threshold, it is considered that the two bounding boxes are duplicates, and the threshold is generally set to 0.7). Therefore, the duplicate bounding boxes obtained by the detection query are deleted, and the remaining bounding boxes obtained by the detection query are the newly emerging targets of interest.

[0018] For tracking methods that involve manually specifying targets of interest, only the manually specified targets are tracked. Therefore, the detection query is only used for target querying in the initial frame, while subsequent video frames only use the tracking query for target querying.

[0019] Furthermore, to achieve the objectives of this invention, this invention proposes a multi-target tracking system based on a query mechanism to implement the method, the system including a video frame acquisition module, a feature extraction module, a detection query acquisition module, and a target query module.

[0020] The video frame acquisition module is used to acquire scene video images containing moving targets using a shooting device;

[0021] The feature extraction module is used to extract deep features from the acquired scene video images and video frames using a deep learning network;

[0022] The detection query acquisition module is used to obtain detection queries from deep features based on automatic recognition or manual specification.

[0023] The target query module is used to query the location of the target in the obtained deep features based on detection query and tracking query, so as to realize target tracking.

[0024] Beneficial effects

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] This invention effectively solves the technical problem of complex tracking processes caused by data association in existing methods. By implementing multi-target tracking through a query mechanism, it can track targets of interest not only automatically but also manually. By simplifying the multi-target tracking process, reducing the complexity of data association, and improving the usability and practicality of the multi-target tracking algorithm, this invention provides technical support for the analysis and processing of high-definition video content. Furthermore, this invention allows for both automatic identification and manual specification of targets of interest, further enhancing its practicality. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0028] Figure 2 This is a schematic diagram of the detection, query, and acquisition method in the method of the present invention;

[0029] Figure 3 This is a schematic diagram of the target query in the method of the present invention;

[0030] Figure 4 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0031] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0032] Example

[0033] like Figure 1 As shown, a multi-target tracking method based on a query mechanism includes the following steps:

[0034] Step S10: Using a camera, capture video images of the scene containing moving targets to obtain video frames to be processed.

[0035] Specifically, in this embodiment of the invention, the shooting device is a roadside surveillance camera, and the shooting scene includes pedestrians and vehicles. However, the specific shooting device used depends on the application scenario.

[0036] Step S20: Obtain the detection query of interest from the acquired video frames, which is used to detect the target of interest in the current frame.

[0037] Specifically, such as Figure 2 As shown, the detection query acquisition method can be implemented using the following steps:

[0038] Step S21: Extract feature F from the obtained video frames. Let H represent the set of real numbers, and let H, W, and d represent the height, width, and number of channels of the extracted features, respectively.

[0039] Preferably, deep learning networks are used to extract depth features from video frames. In this invention, there are no specific limitations on the deep learning network used; various existing deep learning networks suitable for computer vision recognition tasks can be employed, such as deep residual networks.

[0040] Meanwhile, the present invention can also employ a manually designed feature extraction method to extract features from video frames, such as scale-invariant feature transformation (SIFT) and histogram of oriented gradient (HOG).

[0041] Step S22: Obtain the target of interest. This includes automatically identifying the target of interest and manually specifying the target of interest.

[0042] The automatic identification of the target of interest involves automatically identifying and obtaining the detection query from deep features. In this embodiment, this is achieved using a classifier and a regressor. The classifier can be implemented using fully connected layers commonly used in deep learning, or it can be implemented using traditional machine learning algorithms, such as support vector machines. The classifier classifies features to identify those belonging to the target of interest, uses these features as the query content, and then uses the regressor to obtain the query location based on these features. The regressor is similar to the classifier and can be implemented using relevant modules and algorithms from deep learning or traditional machine learning. In this embodiment, the target of interest is a pedestrian in a video frame.

[0043] The user-specified target of interest (OPI) is determined by extracting features from the depth features based on the OPI's location in the initial frame, and using these features as the query content. In this embodiment, the user-specified OPI is a group of pedestrians.

[0044] Step S30: Use the detection query and the tracking results of the previous frame to track the target in the current frame.

[0045] Specifically, such as Figure 3 As shown, the tracking method can be implemented using the following steps:

[0046] Step S31: Track the automatically identified target of interest;

[0047] Specifically, during the tracking process, target tracking is achieved using detection queries and the tracking results from the previous frame. The detection query is used to detect newly emerging targets of interest.

[0048] The tracking result of the previous frame needs to be processed into a tracking query similar to the detection query. The query content in the tracking query is the query content of the target in the previous frame, and the query position is the position of the target in the previous frame.

[0049] The detection query and the tracking query are used together to query the target in the features of the current frame. Non-maximum suppression is used to delete the location boxes that overlap with the location boxes obtained by the tracking query, so as to achieve target tracking. When the intersection-union ratio between two location boxes is greater than a given threshold, the two location boxes are considered to be duplicated. The threshold is generally set to 0.7.

[0050] The deleted bounding boxes are obtained from the detection query, and the remaining bounding boxes obtained from the detection query represent the newly appearing targets of interest. The bounding boxes of tracked targets are obtained from the tracking query.

[0051] This step is only used to automatically identify the tracking method for targets of interest.

[0052] Step S32: Track the user-specified target of interest;

[0053] Specifically, target tracking is achieved by using a tracking query. Since only human-specified targets are tracked, there is no need to identify other targets, thus eliminating the need for detection queries. The method for obtaining the tracking query is the same as that in step S31.

[0054] This step is only used for manually specifying the tracking method for targets of interest.

[0055] Step S33: Track the target by querying;

[0056] Specifically, set It is a feature of the i-th (detection or tracking) query, which includes the query content and query location information.

[0057] First, calculate the query feature f. i Similarity between the video frame feature F and the video frame feature F:

[0058]

[0059]

[0060] in, d represents the feature at coordinates (h, w) in the feature F of a video frame; d represents the number of channels in feature F, i.e., the feature vector F. h,w Dimension; a i,h,w Representing feature f i With F h,w similarity, Representing feature f i With F h,w Normalized similarity, a i,h′,w′ Representing feature f i With F h′,w′ Similarity; F h′,w′ In F (h) ′ ,w ′ Features at coordinates.

[0061] Then, the query features are updated using the similarity obtained from Equation 2:

[0062]

[0063] Finally, the updated query features The data is input into the regressor in step S22 to obtain a more precise bounding box of the target.

[0064] Figure 4 This is a schematic diagram of the system of the present invention.

[0065] like Figure 4 As shown, a multi-target tracking system based on a query mechanism includes a video frame acquisition module M10, a feature extraction module M20, a detection query acquisition module M30, and a target query module M40.

[0066] Among them, the video frame acquisition module M10 is used to capture scene images and videos containing moving targets using a shooting device to obtain video frames to be processed;

[0067] The feature extraction module M20 is used to extract features from the acquired video frames using deep learning networks or traditional hand-designed feature extraction methods;

[0068] The detection query acquisition module M30 is used to obtain the query content and location of the target of interest from video features. The detection query acquisition module M30 includes an automatic target of interest identification submodule M31 and a user-specified target of interest submodule M32.

[0069] The target query module M40 is used to query the location of the target in the features of the current frame based on the detection query and the tracking results of the previous frame.

[0070] The connection relationships between the above modules are as follows:

[0071] The output of the image acquisition module M10 is connected to the input of the feature extraction module M20.

[0072] The output of the feature extraction module M20 is connected to the input of the detection query acquisition module M30 and the input of the target query module M40.

[0073] The output of the detection query acquisition module M30 is connected to the input of the target query module M40. Within the detection query acquisition module M30, the automatic target identification submodule M31 and the manually specified target of interest submodule M32 are independent of each other, used for tracking methods of automatic target of interest identification and manually specified target of interest, respectively.

[0074] Therefore, this method can solve the technical problem of complex tracking processes caused by data association in existing methods. By using a query mechanism to achieve multi-target tracking, it can track targets of interest not only through automatic identification by the tracking method, but also through manual specification, thereby improving the practicality of multi-target tracking methods and providing certain technical support for the analysis and processing of high-level video content.

Claims

1. A multi-target tracking method based on a query mechanism, characterized in that, Includes the following steps: Step 1: Use a camera to capture video of the scene containing moving objects; Step 2: Obtain the detection query of interest from the acquired video frames, which is used to detect the target of interest in the current frame; Step 3: Use the detection query and the tracking results of the previous frame to track the target in the current frame; When tracking automatically identified targets of interest, the tracking process utilizes a detection query and the tracking results from the previous frame. The detection query detects newly emerging targets of interest. The tracking results from the previous frame are processed into a tracking query similar to the detection query. The query content in the tracking query is the target's content in the previous frame, and the query position is the target's position in the previous frame. The detection and tracking queries are used together to search for targets in the features of the current frame. Using non-maximum suppression, overlapping bounding boxes obtained from the tracking query are deleted. When the intersection-union ratio (IU) between two bounding boxes is greater than a given threshold, the two bounding boxes are considered duplicates, thus achieving target tracking. The deleted bounding boxes are obtained from the detection query, and the remaining bounding boxes obtained from the detection query are the newly emerging targets of interest. The bounding boxes of already tracked targets are obtained from the tracking query. When tracking a user-specified target of interest, a tracking query is used to query the target and achieve target tracking without the need for a detection query; the method for obtaining the tracking query is the same as the method for obtaining the tracking query when tracking automatically identified targets of interest; Finally, target tracking is achieved through querying. set up It is the feature of the i-th detection or tracking query, which includes query content and query location information; First, calculate the query feature f. i Similarity between the video frame feature F and the video frame feature F: in, d represents the feature at coordinates (h, w) in the feature F of a video frame; d represents the number of channels in feature F, i.e., the feature vector F. h,w Dimension; a i,h,w Representing feature f i With F h,w similarity, Representing feature f i With F h,w Normalized similarity, a i,h′,w′ Representing feature f i With F h′,w′ Similarity; F h′,w′ This represents the feature at coordinates (h′, w′) in F; Then, the query features are updated using the similarity obtained from Equation 2: Finally, the updated query features The data is input into the regressor to obtain a more precise bounding box of the target.

2. The multi-target tracking method based on a query mechanism as described in claim 1, characterized in that, In step 2, the detection query is automatically identified and obtained by the tracking method.

3. The multi-target tracking method based on a query mechanism as described in claim 2, characterized in that, In step 2, a deep learning-based classifier and regressor are used to process the deep features to obtain the content and location information of the detected query.

4. The multi-target tracking method based on a query mechanism as described in claim 3, characterized in that, In step 2, the classifier and regressor are implemented using fully connected layers.

5. The multi-target tracking method based on a query mechanism as described in claim 1, characterized in that, In step 2, the detection query is obtained in the initial frame through a method specified by the user, who provides the location information of the target of interest. The detection query content is obtained through preprocessing, that is, the detection query content is directly extracted from the deep features.

6. The multi-target tracking method based on a query mechanism as described in claim 1, characterized in that, In step 3, the intersection-union ratio between the two location boxes is greater than a given threshold of 0.

7.

7. A multi-target tracking system based on a query mechanism that implements the method of claim 1, characterized in that, It includes a video frame acquisition module, a feature extraction module, a detection query acquisition module, and a target query module; The video frame acquisition module is used to capture scene images and videos containing moving targets using a shooting device to obtain video frames to be processed; The feature extraction module is used to extract features from the acquired video frames; The detection query acquisition module is used to obtain the query content and query location of the target of interest from video features; The target query module is used to query the location of the target in the features of the current frame based on the detection query and the tracking results of the previous frame; The connection relationships between the above modules are as follows: The output of the image acquisition module is connected to the input of the feature extraction module; The output of the feature extraction module is connected to the input of the detection query acquisition module and the input of the target query module; The output of the detection query acquisition module is connected to the input of the target query.

8. The multi-target tracking system based on a query mechanism as described in claim 7, characterized in that, The detection, query and acquisition module includes an automatic target of interest identification submodule and a user-specified target of interest submodule. The automatic target of interest identification submodule and the user-specified target of interest identification submodule are independent of each other and are used for tracking methods of automatic target of interest identification and user-specified target of interest identification, respectively.

Images