A multi-target tracking method and device based on symmetric joint detection and Re-ID

Abstract

The present invention relates to the field of target tracking technology, and more particularly to a multi-target tracking method and apparatus based on symmetric joint detection and Re-ID. A multi-target tracking method based on symmetric joint detection and Re-ID includes: collecting raw data, preprocessing the raw data to obtain training data; establishing a multi-target tracking model to be trained; training the multi-target tracking model to be trained using the training data to obtain a multi-target tracking model; and inputting the data to be tracked into the multi-target tracking model to obtain a multi-target tracking result. The present invention is an efficient and accurate multi-target tracking method that effectively solves the problem of balancing detection tasks and Re-ID tasks.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a multi-target tracking method and device based on symmetric joint detection and Re-ID. Background Art

[0002] Object tracking is a crucial problem in computer vision. It involves continuously inferring the state of an object in a video sequence. The goal is to locate the object in each frame of the video, generate its trajectory, and provide a complete representation of the target area at each moment. Video tracking technology has a wide range of applications in both military and civilian fields. Military applications include unmanned aerial vehicles, precision guidance, airborne early warning, and battlefield surveillance; while civilian applications include mobile robots, intelligent video surveillance, intelligent transportation systems, human-computer interaction, and virtual reality.

[0003] With the development of multi-task learning technology, multi-object tracking has been dominated by joint detection and re-identification (JDR) models. The goal of the Re-ID task is to learn target appearance features for matching between frames. FairMOT is a landmark work within this paradigm. It argues that anchor-free methods are more suitable for building JDR models than anchor-based methods. MOTS and JDE also extend JDR models based on anchor-based detectors.

[0004] Despite the progress made by the JDR model, it still has room for improvement in the MOT task related to drone videos. FairMOT, a milestone work of the JDR model, is designed based on an anchor-free detector. It shows that anchor-based methods are not suitable for extracting Re-ID features because they cannot achieve perfect correspondence between Re-ID and anchor boxes. However, anchor-based detectors perform well in drone multi-target detection because anchor boxes improve the network's recall ability and significantly improve the detection accuracy of small objects. However, due to the problems caused by anchor boxes, the balance between detection and Re-ID tasks and the gradient problem between multi-task learning are also major challenges facing existing JDR models.

[0005] In the existing technology, there is a lack of an efficient and accurate multi-target tracking method that can well solve the problem of balancing detection tasks and Re-I D tasks. Summary of the Invention

[0006] The embodiment of the present invention provides a multi-target tracking method and apparatus based on symmetric joint detection and Re-ID. The technical solution is as follows:

[0007] Collecting raw data, and preprocessing the raw data to obtain training data;

[0008] Establish a multi-target tracking model to be trained;

[0009] Using the training data to train the multi-target tracking model to be trained, to obtain a multi-target tracking model;

[0010] The data to be tracked is input into the multi-target tracking model to obtain a multi-target tracking result.

[0011] Optionally, establishing a multi-target tracking model to be trained includes:

[0012] Based on the Anchor-based detector, the backbone structure of the YOLOv7 detector is used to build the network and obtain the backbone network;

[0013] Based on the Anchor-based detector, the YOLOv7 detector detection branch is used to build the network to obtain the detection branch;

[0014] Perform symmetric operations according to the network structure of the detection branch to obtain a Re-I D branch;

[0015] Based on the backbone network, the detection branch and the Re-ID branch, a multi-target tracking model to be trained is obtained.

[0016] Optionally, the anchor-based detector uses a YOLOv7 detector detection branch to construct a network to obtain a detection branch, including:

[0017] Based on the Anchor-based detector, the detection branch of the YOLOv7 detector is used to construct the detection branch Neck; the detection branch Neck is used for feature transformation and feature fusion;

[0018] Establish a detection branch output end; the detection branch output end includes three detection output heads; each of the three detection output heads includes a prediction box center position head, a prediction box size head, a confidence head, and a classification head; the number of channels output by each detection output head is calculated based on the number of categories in the detection task;

[0019] Based on the detection branch Neck and the detection branch output end, a detection branch is obtained.

[0020] Optionally, performing a symmetric operation according to the network structure of the detection branch to obtain a Re-ID branch includes:

[0021] Based on the network structure of the detection branch, a detection branch Neck is obtained; the network structure of the detection branch Neck is symmetrically applied to the Re-ID branch to obtain a Re-ID branch Neck; the network structure of the Re-ID branch Neck is consistent with the network structure of the detection branch Neck;

[0022] Establishing a Re-ID branch output terminal; the Re-ID branch output terminal includes three Re-ID branch output heads; the number of channels output by the Re-ID branch output head is 128;

[0023] A Re-ID branch is obtained based on the Re-ID branch Neck and the Re-ID branch output end.

[0024] Optionally, the using the training data to train the to-be-trained multi-target tracking model to obtain the multi-target tracking model includes:

[0025] Setting the network loss of the Re-ID branch to 0, using the training data to train the backbone network and the detection branch to obtain a trained backbone network and a trained detection branch, and solidifying the network parameters of the detection branch and the backbone network;

[0026] Based on the trained backbone network and the trained detection branch, the Re-ID branch is trained using the training data to obtain a trained Re-ID branch;

[0027] Based on the trained backbone network, the trained detection branch and the trained Re-ID branch, a multi-target tracking model is obtained.

[0028] Optionally, inputting the data to be tracked into the multi-target tracking model to obtain a multi-target tracking result includes:

[0029] Perform frame operations and labeling operations on the data to be tracked to obtain processed data;

[0030] Inputting the processed data into a trained backbone network to obtain feature data;

[0031] Input the feature data into the trained detection branch to obtain a detection result;

[0032] Input the feature data and the detection result into the trained Re-ID branch to obtain a Re-ID feature vector;

[0033] According to the Re-ID feature vector and the detection result, cascade matching is performed through the preset DeepSORT algorithm to obtain the multi-target tracking result.

[0034] On the other hand, a multi-target tracking device based on symmetric joint detection and Re-ID is provided, which is applied to a multi-target tracking method based on symmetric joint detection and Re-ID, and includes:

[0035] A data processing module is used to collect raw data, pre-process the raw data, and obtain training data;

[0036] Model building module, used to build the multi-target tracking model to be trained;

[0037] A model training module is used to train the multi-target tracking model to be trained using the training data to obtain a multi-target tracking model;

[0038] The target tracking module is used to input the data to be tracked into the multi-target tracking model to obtain the multi-target tracking results.

[0039] Optionally, the model building module is further used to:

[0040] Based on the Anchor-based detector, the backbone structure of the YOLOv7 detector is used to build the network and obtain the backbone network;

[0041] Based on the Anchor-based detector, the YOLOv7 detector detection branch is used to build the network to obtain the detection branch;

[0042] Perform symmetric operations according to the network structure of the detection branch to obtain a Re-I D branch;

[0043] Based on the backbone network, the detection branch and the Re-ID branch, a multi-target tracking model to be trained is obtained.

[0044] Optionally, the model building module is further used to:

[0045] Based on the Anchor-based detector, the detection branch of the YOLOv7 detector is used to construct the detection branch Neck; the detection branch Neck is used for feature transformation and feature fusion;

[0046] Establish a detection branch output end; the detection branch output end includes three detection output heads; each of the three detection output heads includes a prediction box center position head, a prediction box size head, a confidence head, and a classification head; the number of channels output by each detection output head is calculated based on the number of categories in the detection task;

[0047] Based on the detection branch Neck and the detection branch output end, a detection branch is obtained.

[0048] Optionally, the model building module is further used to:

[0049] Based on the network structure of the detection branch, a detection branch Neck is obtained; the network structure of the detection branch Neck is symmetrically applied to the Re-ID branch to obtain a Re-ID branch Neck; the network structure of the Re-ID branch Neck is consistent with the network structure of the detection branch Neck;

[0050] Establishing a Re-ID branch output terminal; the Re-ID branch output terminal includes three Re-ID branch output heads; the number of channels output by the Re-ID branch output head is 128;

[0051] A Re-I D branch is obtained based on the Re-I D branch Neck and the Re-I D branch output end.

[0052] Optionally, the model training module is further used to:

[0053] Setting the network loss of the Re-ID branch to 0, using the training data to train the backbone network and the detection branch to obtain a trained backbone network and a trained detection branch, and solidifying the network parameters of the detection branch and the backbone network;

[0054] Based on the trained backbone network and the trained detection branch, the Re-ID branch is trained using the training data to obtain a trained Re-ID branch;

[0055] Based on the trained backbone network, the trained detection branch and the trained Re-ID branch, a multi-target tracking model is obtained.

[0056] Optionally, the target tracking module is further configured to:

[0057] Perform frame operations and labeling operations on the data to be tracked to obtain processed data;

[0058] Inputting the processed data into a trained backbone network to obtain feature data;

[0059] Input the feature data into the trained detection branch to obtain a detection result;

[0060] Input the feature data and the detection result into the trained Re-ID branch to obtain a Re-ID feature vector;

[0061] According to the Re-ID feature vector and the detection result, cascade matching is performed through the preset DeepSORT algorithm to obtain the multi-target tracking result.

[0062] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned multi-target tracking method based on symmetric joint detection and Re-I D.

[0063] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned multi-target tracking method based on symmetric joint detection and Re-I D.

[0064] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0065] The present invention proposes a multi-target tracking method based on symmetric joint detection and Re-ID, and implements the multi-target tracking task based on the anchor-based detector; proposes a dual-branch network structure with symmetric structures of the detection branch and the Re-ID branch, achieving a balance between the detection task and the Re-ID task; learns Re-ID features for the detection anchor frame, solving the gradient collision problem between the detection task and the Re-ID task; and adopts a partial training method of first training the detection branch and then training the Re-ID branch to alleviate the learning and training conflict generated during the training process of the detection branch and the Re-ID branch. The present invention is an efficient and accurate multi-target tracking method that effectively solves the problem of balancing the detection task and the Re-ID task. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 This is a flow chart of a multi-target tracking method based on symmetric joint detection and Re-ID provided by an embodiment of the present invention;

[0068] Figure 2 Schematic diagram of a multi-target tracking model provided by an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of the correspondence between Re-ID features and the same-level output detection results provided by an embodiment of the present invention;

[0070] Figure 4 This is a block diagram of a multi-target tracking device based on symmetric joint detection and Re-ID provided by an embodiment of the present invention;

[0071] Figure 5 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0073] The embodiment of the present invention provides a multi-target tracking method based on symmetric joint detection and Re-ID, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of a multi-target tracking method based on symmetric joint detection and Re-ID is shown. The processing flow of this method may include the following steps:

[0074] S1. Collect raw data, pre-process the raw data, and obtain training data;

[0075] In one feasible implementation, a camera is used to shoot the target to obtain raw video data containing the tracking task target. The raw video data is frame-processed to obtain the training data required for model training.

[0076] S2. Establish a multi-target tracking model to be trained;

[0077] Optionally, a multi-target tracking model to be trained is established, including:

[0078] Based on the Anchor-based detector, the backbone structure of the YOLOv7 detector is used to build the network and obtain the backbone network;

[0079] Based on the Anchor-based detector, the YOLOv7 detector detection branch is used to build the network to obtain the detection branch;

[0080] Perform symmetric operations based on the network structure of the detection branch to obtain the Re-ID branch;

[0081] Based on the backbone network, detection branch and Re-ID branch, a multi-target tracking model to be trained is obtained.

[0082] In a feasible implementation, in order to synchronously realize the detection task and the pedestrian re-identification (Re-Identification, Re-ID) task in the present invention, the backbone network must realize parameter sharing of the detection target for these two tasks. If the Re-ID branch has too few convolutional layers, the burden on the backbone network is too heavy, and it needs to bear the parameter sharing of the two tasks. When the design of the two branches is asymmetric, it will cause an imbalance problem between the two tasks, resulting in poor accuracy of both tasks, so in the present invention, the two branches adopt a symmetrical structure. The present invention inputs the input image into the backbone network and extracts abstract features. Then, three feature maps of different scales (Strip = 8, 16, 32) are fused to the two branches, and the detection results and Re-ID features of three different scales are output. The model structure is as follows Figure 2 shown.

[0083] In order to achieve a good balance between accuracy and speed, the backbone network of the present invention adopts the backbone network of the 7th version (You Only Look Once, YOLOv7) detector of a deep learning algorithm for image detection.

[0084] Based on the E-ELAN module of YOLOV7, the backbone network improves the network's learning ability by expanding, shuffling, and merging bases without destroying the original gradient path. This helps to ensure the quality of the Re-ID feature vector when performing detection tasks.

[0085] Denote the size of the input feature map as C IN ×H IN ×W IN , then the shape of the output feature map of E-ELAN is C OUT ×H OUT ×W OUT , where C is the number of channels, W and H are the sizes of the feature maps.

[0086] In the above formula, C OUT =4×C IN , H OUT =H IN and W OUT =W IN The multiple uses of E-ELAN enable the network to learn various features at different levels.

[0087] Optionally, based on the Anchor-based detector, a YOLOv7 detector detection branch is used to construct a network to obtain a detection branch, including:

[0088] Based on the Anchor-based detector, the detection branch of the YOLOv7 detector is used to construct the detection branch Neck; the detection branch Neck is used for feature transformation and feature fusion;

[0089] Establish the detection branch output end; the detection branch output end includes three detection output heads; each of the three detection output heads includes a prediction box center position head, a prediction box size head, a confidence head, and a classification head; the number of channels output by each detection output head is calculated based on the number of categories in the detection task;

[0090] Based on the detection branch Neck and the detection branch output end, a detection branch is obtained.

[0091] In one implementation, the detection branch consists of a YOLOv7 anchor-based detector and three output heads, corresponding to the three levels of feature maps output from the backbone. The Neck portion of the network is used for feature transformation and fusion. Each output head includes a prediction box center location head, a prediction box size head, a confidence head, and a classification head.

[0092] The prediction box center position and size head are designed to determine the location and size of the object. The three output layers correspond to different sizes of objects. The higher the level of extracted object features, the richer the semantics, and the larger the corresponding detected object.

[0093] Each prediction frame in the present invention is counted as Among them, the front and back of the expression represent the center position and size of the prediction box respectively, and the area of the prediction box is For each GT box The front and back of the expression represent the center position and size of the GT box respectively, and the area of the GT box is a i =(w i ×h i ), the mathematical expressions of the loss functions for the prediction box center position head and the prediction box size head are as follows (1), (2), (3), and (4):

[0094]

[0095]

[0096]

[0097]

[0098] Where c represents the diagonal distance of the minimum closed area that can contain both the prediction box and the GT box, ρ represents the Euler distance between the two centers, and N is the total number of targets in the current training batch.

[0099] The confidence header is essentially a measure of the probability that an object exists in the region of interest. We can use the confidence header to quickly remove prediction boxes that do not contain the target. The classification header is used to distinguish the categories of different objects. This paper uses the BCEWithLogitsLoss function to calculate the loss of the two heads. The mathematical expressions of the confidence head and classification head loss functions are shown in the following formula (5):

[0100]

[0101] Among them, w is a hyperparameter, y i 、x i They correspond to the true values corresponding to the two heads respectively.

[0102] Optionally, a symmetric operation is performed according to the network structure of the detection branch to obtain a Re-ID branch, including:

[0103] Based on the network structure of the detection branch, the detection branch Neck is obtained; the network structure of the detection branch Neck is symmetrically transferred to the Re-ID branch to obtain the Re-ID branch Neck; the network structure of the Re-ID branch Neck is consistent with the network structure of the detection branch Neck;

[0104] Establish a Re-ID branch output terminal; the Re-ID branch output terminal includes three Re-ID branch output heads; the number of channels output by the Re-ID branch output head is 128;

[0105] Based on the Re-ID branch Neck and the Re-ID branch output end, a Re-ID branch is obtained.

[0106] In a feasible implementation, in order to ensure the performance balance between the detection task and the Re-ID task, the network structure of the Re-ID branch is designed to be similar to that of the detection branch. The feature maps output by different layers of the backbone network correspond to tracking targets of three different scales. Figure 3 As shown, in this paper, the appearance features of the target are extracted for each anchor box at the same level, and the Re-ID features are learned through the classification task. During the target tracking process, all objects with the same ID are considered to be of the same category. The Re-ID features correspond to the detection results output at the same level, and each sub-grid corresponds to three anchor boxes. Therefore, the Re-ID head is designed with 3×128 channels to extract appearance features for each anchor box.

[0107] For each center point c i The prediction target is extracted through the fully connected layer and Softmax function to extract its feature vector E i , and map it to the distribution vector P = {p(k), k∈[1,K]}, and adjust the target ID to one-hot form, expressed as Li (k) Focal Loss is used to calculate the Re-ID loss. The mathematical expression of its loss function is shown in the following formula (6):

[0108]

[0109] where K is the total number of identifiers in a single sequence.

[0110] S3. Use the training data to train the multi-target tracking model to obtain a multi-target tracking model;

[0111] Optionally, training the to-be-trained multi-target tracking model using the training data to obtain the multi-target tracking model includes:

[0112] Set the network loss of the Re-ID branch to 0, use the training data to train the backbone network and the detection branch, obtain the trained backbone network and the trained detection branch, and solidify the network parameters of the detection branch and the backbone network;

[0113] Based on the trained backbone network and the trained detection branch, the Re-ID branch is trained using the training data to obtain a trained Re-ID branch;

[0114] Based on the trained backbone network, the trained detection branch and the trained Re-ID branch, a multi-target tracking model is obtained.

[0115] In one feasible implementation, the present invention uses separate detection and Re-ID training based on the publicly available Visdrone-MOT2019 dataset. The Re-ID loss is set to 0 to train the detection branch, and then the network parameters of the detection branch are fixed to train the Re-ID branch.

[0116] Due to gradient conflicts in multi-task learning, when the losses correspond to different weights, the results of detection and Re-ID will be significantly different. In this paper, we quantitatively compare the impact of different weights. When the weight of detection and Re-ID is 1, it represents the separation of training detection branch and Re-ID branch, respectively. As shown in Table 1 (Separate training cosine metric mean table), we used weight ratios of 0.9-0.1, 0, 8-0.2, and 0.7-0.3 to train the detection module and Re-ID module separately, set 30 rounds respectively, and resize the input image to 640×640. The average precision (AP) of detection and the mean of the cosine metric of the Re-ID feature vector show that separate training can effectively alleviate the gradient conflict problem in our work.

[0117] Table 1

[0118]

[0119] S4. Input the data to be tracked into the multi-target tracking model to obtain the multi-target tracking result.

[0120] Optionally, the data to be tracked is input into a multi-target tracking model to obtain a multi-target tracking result, including:

[0121] Perform frame operations and labeling operations on the data to be tracked to obtain processed data;

[0122] Input the processed data into the trained backbone network to obtain feature data;

[0123] Input the feature data into the trained detection branch to obtain the detection results;

[0124] Input the feature data and detection results into the trained Re-ID branch to obtain the Re-ID feature vector;

[0125] According to the Re-ID feature vector and detection results, cascade matching is performed through the preset DeepSORT algorithm to obtain multi-target tracking results.

[0126] In one feasible implementation, the network of the present invention uses a 1920×1920 frame as the target input. The processed data is passed through a trained backbone network for feature extraction. The detection branch and the Re-ID branch, attached to the backbone network, synchronously learn the detected target features. The tracked target, derived from the target features and the detection branch, is processed through the Re-ID branch to obtain a Re-ID feature vector. DeepSORT cascade matching is performed on the Re-ID feature vectors obtained from the detection and Re-ID branches and the detection results. After three rounds of cascade matching, the final multi-target tracking result is obtained.

[0127] In the detection branch, the prediction box is selected based on the confidence score. The prediction box is selected by non-maximum suppression (NMS). The NMS is implemented by a simple 3×3 maximum pooling operation, and the prediction box with a confidence score greater than a certain threshold is used as the final result.

[0128] The data association process of this invention uses DeepSORT as the data association method. Multiple trajectories are first initialized based on the detection results of the first frame. Trajectories are then predicted using a Kalman filter. Objects are associated using cascade matching and IOU matching. DeepSORT's data association method can be referenced in other publicly available target tracking solutions and will not be detailed here.

[0129] The present invention proposes a multi-target tracking method based on symmetric joint detection and Re-ID, and implements the multi-target tracking task based on the anchor-based detector. A dual-branch network structure with symmetric detection and Re-ID branch structures is proposed to achieve a balance between the detection task and the Re-ID task. Re-ID features are learned for the detection anchor frame to solve the gradient collision problem between the detection task and the Re-ID task. In order to alleviate the learning and training conflict generated during the training of the detection branch and the Re-ID branch, a partial training method is adopted in which the detection branch is trained first and then the Re-ID branch is trained. The present invention is an efficient and accurate multi-target tracking method that effectively solves the problem of balancing the detection task and the Re-ID task.

[0130] Figure 4 FIG1 is a block diagram of a multi-target tracking device based on symmetric joint detection and Re-ID according to an exemplary embodiment. Figure 4 , the device comprises:

[0131] The data processing module 410 is used to collect raw data, pre-process the raw data, and obtain training data;

[0132] A model building module 420 is used to build a multi-target tracking model to be trained;

[0133] A model training module 430 is used to train the multi-target tracking model to be trained using the training data to obtain a multi-target tracking model;

[0134] The target tracking module 440 is used to input the data to be tracked into the multi-target tracking model to obtain the multi-target tracking results.

[0135] Optionally, the model building module 420 is further configured to:

[0136] Based on the Anchor-based detector, the backbone structure of the YOLOv7 detector is used to build the network and obtain the backbone network;

[0137] Based on the Anchor-based detector, the YOLOv7 detector detection branch is used to build the network to obtain the detection branch;

[0138] Perform symmetric operations based on the network structure of the detection branch to obtain the Re-I D branch;

[0139] Based on the backbone network, detection branch and Re-ID branch, the multi-target tracking model to be trained is obtained.

[0140] Optionally, the model building module 420 is further configured to:

[0141] Based on the Anchor-based detector, the detection branch of the YOLOv7 detector is used to construct the detection branch Neck; the detection branch Neck is used for feature transformation and feature fusion;

[0142] Establish the detection branch output end; the detection branch output end includes three detection output heads; each of the three detection output heads includes a prediction box center position head, a prediction box size head, a confidence head, and a classification head; the number of channels output by each detection output head is calculated based on the number of categories in the detection task;

[0143] Based on the detection branch Neck and the detection branch output end, a detection branch is obtained.

[0144] Optionally, the model building module 420 is further configured to:

[0145] Based on the network structure of the detection branch, the detection branch Neck is obtained; the network structure of the detection branch Neck is symmetrically transferred to the Re-ID branch to obtain the Re-ID branch Neck; the network structure of the Re-ID branch Neck is consistent with the network structure of the detection branch Neck;

[0146] Establish a Re-I D branch output terminal; the Re-I D branch output terminal includes three Re-I D branch output heads; the number of channels output by the Re-I D branch output head is 128;

[0147] Based on the Re-ID branch Neck and the Re-ID branch output, a Re-ID branch is obtained.

[0148] Optionally, the model training module 430 is further configured to:

[0149] Set the network loss of the Re-ID branch to 0, use the training data to train the backbone network and the detection branch, obtain the trained backbone network and the trained detection branch, and solidify the network parameters of the detection branch and the backbone network;

[0150] Based on the trained backbone network and the trained detection branch, the Re-ID branch is trained using the training data to obtain a trained Re-ID branch;

[0151] Based on the trained backbone network, the trained detection branch and the trained Re-ID branch, a multi-target tracking model is obtained.

[0152] Optionally, the target tracking module 440 is further configured to:

[0153] Perform frame operations and labeling operations on the data to be tracked to obtain processed data;

[0154] Input the processed data into the trained backbone network to obtain feature data;

[0155] Input the feature data into the trained detection branch to obtain the detection results;

[0156] Input the feature data and detection results into the trained Re-ID branch to obtain the Re-ID feature vector;

[0157] According to the Re-ID feature vector and detection results, cascade matching is performed through the preset DeepSORT algorithm to obtain multi-target tracking results.

[0158] This paper proposes a multi-target tracking method based on symmetric joint detection and Re-ID, which implements the multi-target tracking task based on an anchor-based detector. A dual-branch network structure with symmetric detection and Re-ID branch structures is proposed to achieve a balance between the detection and Re-ID tasks. Re-ID features are learned from the detection anchor boxes, solving the gradient collision problem between the detection and Re-ID tasks. To alleviate the learning and training conflicts that arise during the training of the detection and Re-ID branches, a partial training method is adopted, first training the detection branch and then training the Re-ID branch. This paper is an efficient and accurate multi-target tracking method that effectively solves the problem of balancing the detection and Re-ID tasks.

[0159] Figure 5 It is a structural diagram of an electronic device 500 provided in an embodiment of the present invention. The electronic device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 501 and one or more memories 502, wherein the memory 502 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 501 to implement the steps of the above-mentioned multi-target tracking method based on symmetric joint detection and Re-ID.

[0160] In an exemplary embodiment, a computer-readable storage medium is further provided, such as a memory including instructions. The instructions can be executed by a processor in a terminal to implement the multi-target tracking method based on symmetric joint detection and Re-ID. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0161] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-target tracking method based on symmetric joint detection and Re-ID, characterized in that: The method comprises: Collecting raw data, and preprocessing the raw data to obtain training data; The original data is the original video data containing the tracking task target shot by a camera; Establish a multi-target tracking model to be trained; The step of establishing a multi-target tracking model to be trained includes: Based on the Anchor-based detector, the backbone structure of the YOLOv7 detector is used to build the network and obtain the backbone network; Based on the Anchor-based detector, the YOLOv7 detector detection branch is used to build the network to obtain the detection branch; Perform symmetric operations according to the network structure of the detection branch to obtain a Re-ID branch; Based on the backbone network, the detection branch and the Re-ID branch, a multi-target tracking model to be trained is obtained; The anchor-based detector uses the YOLOv7 detector detection branch to construct a network, and obtains the detection branch, including: Based on the Anchor-based detector, the detection branch of the YOLOv7 detector is used to construct the detection branch Neck; the detection branch Neck is used for feature transformation and feature fusion; Establish a detection branch output end; the detection branch output end includes three detection output heads; each of the three detection output heads includes a prediction box center position head, a prediction box size head, a confidence head, and a classification head; the number of channels output by each detection output head is calculated based on the number of categories in the detection task; Obtaining a detection branch based on the detection branch Neck and the detection branch output end; The step of performing a symmetric operation based on the network structure of the detection branch to obtain a Re-ID branch includes: Based on the network structure of the detection branch, a detection branch Neck is obtained; the network structure of the detection branch Neck is symmetrically applied to the Re-ID branch to obtain a Re-ID branch Neck; the network structure of the Re-ID branch Neck is consistent with the network structure of the detection branch Neck; Establishing a Re-ID branch output terminal; the Re-ID branch output terminal includes three Re-ID branch output heads; the number of channels output by the Re-ID branch output head is 128; Obtaining a Re-ID branch based on the Re-ID branch Neck and the Re-ID branch output end; Using the training data to train the multi-target tracking model to be trained, to obtain a multi-target tracking model; The data to be tracked is input into the multi-target tracking model to obtain a multi-target tracking result.

2. The multi-target tracking method based on symmetric joint detection and Re-ID according to claim 1, characterized in that: The step of training the to-be-trained multi-target tracking model using the training data to obtain the multi-target tracking model includes: Setting the network loss of the Re-ID branch to 0, using the training data to train the backbone network and the detection branch to obtain a trained backbone network and a trained detection branch, and solidifying the network parameters of the detection branch and the backbone network; Based on the trained backbone network and the trained detection branch, the Re-ID branch is trained using the training data to obtain a trained Re-ID branch; Based on the trained backbone network, the trained detection branch and the trained Re-ID branch, a multi-target tracking model is obtained.

3. The multi-target tracking method based on symmetric joint detection and Re-ID according to claim 1, characterized in that: The step of inputting the data to be tracked into the multi-target tracking model to obtain a multi-target tracking result includes: Perform frame operations and labeling operations on the data to be tracked to obtain processed data; Inputting the processed data into a trained backbone network to obtain feature data; Input the feature data into the trained detection branch to obtain a detection result; Input the feature data and the detection result into the trained Re-ID branch to obtain a Re-ID feature vector; According to the Re-ID feature vector and the detection result, cascade matching is performed through a preset DeepSORT algorithm to obtain a multi-target tracking result.

4. A multi-target tracking device based on symmetric joint detection and Re-ID, characterized in that: The device comprises: A data processing module is used to collect raw data, pre-process the raw data, and obtain training data; The original data is the original video data containing the tracking task target shot by a camera; Model building module, used to build the multi-target tracking model to be trained; The step of establishing a multi-target tracking model to be trained includes: Based on the Anchor-based detector, the backbone structure of the YOLOv7 detector is used to build the network and obtain the backbone network; Based on the Anchor-based detector, the YOLOv7 detector detection branch is used to build the network to obtain the detection branch; Perform symmetric operations according to the network structure of the detection branch to obtain a Re-ID branch; Based on the backbone network, the detection branch and the Re-ID branch, a multi-target tracking model to be trained is obtained; The anchor-based detector uses the YOLOv7 detector detection branch to construct a network, and obtains the detection branch, including: Based on the Anchor-based detector, the detection branch of the YOLOv7 detector is used to construct the detection branch Neck; the detection branch Neck is used for feature transformation and feature fusion; Establish a detection branch output end; the detection branch output end includes three detection output heads; each of the three detection output heads includes a prediction box center position head, a prediction box size head, a confidence head, and a classification head; the number of channels output by each detection output head is calculated based on the number of categories in the detection task; Obtaining a detection branch based on the detection branch Neck and the detection branch output end; The step of performing a symmetric operation based on the network structure of the detection branch to obtain a Re-ID branch includes: Based on the network structure of the detection branch, a detection branch Neck is obtained; the network structure of the detection branch Neck is symmetrically applied to the Re-ID branch to obtain a Re-ID branch Neck; the network structure of the Re-ID branch Neck is consistent with the network structure of the detection branch Neck; Establishing a Re-ID branch output terminal; the Re-ID branch output terminal includes three Re-ID branch output heads; the number of channels output by the Re-ID branch output head is 128; Obtaining a Re-ID branch based on the Re-ID branch Neck and the Re-ID branch output end; A model training module is used to train the multi-target tracking model to be trained using the training data to obtain a multi-target tracking model; The target tracking module is used to input the data to be tracked into the multi-target tracking model to obtain the multi-target tracking results.

5. The multi-target tracking device based on symmetric joint detection and Re-ID according to claim 4, characterized in that: The model training module is further used to: Setting the network loss of the Re-ID branch to 0, using the training data to train the backbone network and the detection branch to obtain a trained backbone network and a trained detection branch, and solidifying the network parameters of the detection branch and the backbone network; Based on the trained backbone network and the trained detection branch, the Re-ID branch is trained using the training data to obtain a trained Re-ID branch; Based on the trained backbone network, the trained detection branch and the trained Re-ID branch, a multi-target tracking model is obtained.

6. The multi-target tracking device based on symmetric joint detection and Re-ID according to claim 4, characterized in that: The target tracking module is further configured to: Perform frame operations and labeling operations on the data to be tracked to obtain processed data; Inputting the processed data into a trained backbone network to obtain feature data; Input the feature data into the trained detection branch to obtain a detection result; Input the feature data and the detection result into the trained Re-ID branch to obtain a Re-ID feature vector; According to the Re-ID feature vector and the detection result, cascade matching is performed through a preset DeepSORT algorithm to obtain a multi-target tracking result.

Images