Multi-target tracking method based on double-branch feature enhancement and multi-level trajectory association

Abstract

The present disclosure discloses a multi-target tracking method based on double-branch feature enhancement and multi-level trajectory association, comprising: S100: obtaining an input image containing a plurality of to-be-tracked targets; S200: constructing a multi-target tracking model and training to obtain a trained multi-target tracking model; S300: inputting the input image into the trained multi-target tracking model to simultaneously track the plurality of to-be-tracked targets in the input image. The present disclosure also discloses a multi-target tracking device based on double-branch feature enhancement and multi-level trajectory association, a storage medium and an electronic device. The present disclosure can solve the problem of target tracking failure in complex scenes such as target occlusion and blur, thereby improving the tracking performance of the plurality of to-be-tracked targets in complex scenes.

Description

Technical Field

[0001] This disclosure pertains to the field of target tracking, specifically relating to a multi-target tracking method based on bi-branch feature enhancement and multi-level trajectory association. Background Technology

[0002] Multiple Object Tracking (MOT) is an important research direction in the field of computer vision. It aims to predict the positions of multiple objects in consecutive video frames, identify which objects belong to the same object, and generate the motion trajectory of the object. It has a wide range of applications in practical scenarios such as video analysis, autonomous driving, robotics, and action recognition.

[0003] In recent years, with the improvement of the accuracy and speed of target detectors, the MOT algorithm has developed rapidly. Currently, MOT algorithm models are divided into two types: two-stage models and single-stage models. Two-stage models follow a detection-then-tracking paradigm, such as SORT, DeepSORT, and POI. They divide MOT into two independent tasks: first, the detector detects video frames to obtain the bounding boxes of targets, and then performs correlation tracking between targets. While the tracking accuracy of two-stage models is leading, the models are relatively complex and computationally expensive, making it difficult to balance accuracy and tracking speed. Single-stage models follow a joint detection and tracking paradigm, performing detection and correlation within the same network. They redesign the detector head as a tracking branch, simultaneously obtaining the results of both the detection and tracking branches to achieve correlation between targets. For example, the typical single-stage MOT algorithm CenterTrack uses an anchor-free detector, CenterNet, and adds a regression prediction tracking branch to obtain the offset vectors between targets, thus jointly training detection and correlation. Compared with the two-stage model, the single-stage model simplifies the model complexity, improves the tracking speed, and can balance the accuracy of the algorithm and the tracking speed.

[0004] However, when faced with complex scenarios, single-stage models still suffer from problems such as insufficient target feature extraction, identity switching, and missing trajectories, thus reducing the performance of MOT. For most MOT algorithm detectors, insufficient target feature extraction will prevent the output of high-quality detection results, thereby limiting the tracker's performance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this disclosure is to provide a multi-target tracking method based on bi-branch feature enhancement and multi-level trajectory association. This method can solve the problem of target tracking failure in complex scenarios such as target occlusion and blurring, thereby improving the tracking performance of multiple targets in complex scenarios.

[0006] To achieve the above objectives, this disclosure provides the following technical solutions:

[0007] A multi-target tracking method based on bi-branch feature enhancement and multi-level trajectory association includes the following steps:

[0008] S100: Acquire an input image containing multiple targets to be tracked;

[0009] S200: Construct and train a multi-target tracking model to obtain a well-trained multi-target tracking model;

[0010] The multi-target tracking model utilizes a dual-branch feature learning network to alleviate the excessive competition between detection and tracking tasks, and introduces correlation matrix AM prediction to obtain a more accurate offset vector to reduce the number of identity switching of the target to be tracked in the input image.

[0011] S300: Input the input image into the trained multi-target tracking model to achieve simultaneous tracking of multiple targets in the input image.

[0012] Preferably, in step S200, the multi-target tracking model is trained using the following method:

[0013] S201: Obtain the dataset and divide it into training and test sets;

[0014] S202: Set training parameters, train the model using the training set, and the model training is complete when the set number of training rounds is reached;

[0015] S203: Test the trained model using the test set. During the test, multi-target tracking accuracy and IDF1 score are used as evaluation metrics to evaluate the model. The model passes the test when the tracking accuracy reaches 66.1% and the IDF1 score reaches 64.2%.

[0016] Preferably, in step S203, the multi-target tracking accuracy is expressed as follows:

[0017]

[0018] Here, FN represents false negative, FP represents false positive, IDS represents identity switching count, and GT stands for Ground Truth, which represents the number of targets labeled in the scene.

[0019] Preferably, in step S203, the IDF1 score is represented as:

[0020]

[0021] Among them, IDTP is the True Positive ID, which represents the number of correctly assigned targets in the entire video; IDFN is the False Negative ID, which represents the number of unassigned targets in the entire video; and IDFP is the False Positive ID, which represents the number of incorrectly assigned targets in the entire video.

[0022] This disclosure also provides a multi-target tracking device based on bi-branch feature enhancement and multi-level trajectory association, including:

[0023] The acquisition module is used to acquire an input image containing multiple targets to be tracked;

[0024] The model building and training module is used to build and train a multi-object tracking model to obtain a well-trained multi-object tracking model.

[0025] The multi-target tracking model utilizes a dual-branch feature learning network to alleviate the excessive competition between detection and tracking tasks, and introduces correlation matrix AM prediction to obtain a more accurate offset vector to reduce the number of identity switching of the target to be tracked in the input image.

[0026] The tracking module is used to input the input image into the trained multi-target tracking model to achieve simultaneous tracking of multiple targets in the input image.

[0027] Preferably, the model building and training module includes:

[0028] The sub-module is used to divide the dataset for model training into training and testing sets;

[0029] The training submodule is used to train the model using the training set;

[0030] The test submodule is used to test the trained model using a test set.

[0031] This disclosure also provides a computer storage medium storing computer-executable instructions for performing any of the methods described above.

[0032] This disclosure also provides an electronic device, including:

[0033] Memory, processor, and computer programs stored in memory and executable on the processor, wherein,

[0034] When the processor executes the program, it implements the method described in any of the preceding methods.

[0035] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0036] 1. This disclosure uses a dual-branch feature learning network to learn the specificity and correlation between the detection and tracking tasks, which alleviates the excessive competition between the two tasks and can extract sufficient target feature information.

[0037] 2. This disclosure introduces an association matrix, which uses more time-series information to predict the offset vector, thereby reducing the number of identity switching attempts;

[0038] 3. This disclosure adopts a multi-level trajectory association strategy, which associates high and low score detection boxes with trajectories by using different matching methods, thereby reducing the number of missing trajectories;

[0039] 4. Based on the above three improvements, this disclosure can improve the tracking performance of multiple targets in complex scenarios. Attached Figure Description

[0040] Figure 1 This is a flowchart of a multi-target tracking method based on bi-branch feature enhancement and multi-level trajectory association provided in one embodiment of this disclosure;

[0041] Figure 2 This is a schematic diagram of the structure of a multi-target tracking model provided in another embodiment of this disclosure;

[0042] Figure 3 yes Figure 2 A schematic diagram of the DFL network structure in the model shown;

[0043] Figure 4 yes Figure 2 The diagram shows the structure of the MTA strategy in the model shown.

[0044] Figure 5(a) is a schematic diagram of the tracking results of the CenterTrack algorithm on the MOT17-04 dataset;

[0045] Figure 5(b) is a schematic diagram of the tracking results of the multi-target tracking model provided in another embodiment of this disclosure on the dataset MOT17-04;

[0046] Figure 6(a) is a schematic diagram of the tracking results of the CenterTrack algorithm on the MOT17-09 dataset;

[0047] Figure 6(b) is a schematic diagram of the tracking results of the multi-target tracking model provided in another embodiment of this disclosure on the dataset MOT17-09;

[0048] Figure 7(a) is a schematic diagram of the tracking results of the CenterTrack algorithm on the MOT17-11 dataset;

[0049] Figure 7(b) is a schematic diagram of the tracking results of a multi-target tracking model provided in another embodiment of this disclosure on the dataset MOT17-11. Detailed Implementation

[0050] The following will refer to the appendix. Figures 1 to 7(b) Specific embodiments of this disclosure are described in detail. While specific embodiments of this disclosure are shown in the accompanying drawings, it should be understood that this disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0051] It should be noted that certain terms are used in the specification and claims to refer to specific components. Those skilled in the art will understand that different terms may be used to refer to the same component. This specification and claims do not distinguish components based on differences in terminology, but rather on differences in function. The terms "comprising" or "including" used throughout the specification and claims are open-ended and should be interpreted as "comprising but not limited to." The following descriptions are preferred embodiments for carrying out this disclosure; however, these descriptions are for the purpose of understanding the general principles of the specification and are not intended to limit the scope of this disclosure. The scope of protection of this disclosure is determined by the appended claims.

[0052] To facilitate understanding of the embodiments of this disclosure, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. The accompanying drawings do not constitute a limitation on the embodiments of this disclosure.

[0053] In one embodiment, such as Figure 1 As shown, this disclosure provides a multi-target tracking method based on bi-branch feature enhancement and multi-level trajectory association, including the following steps:

[0054] S100: Acquire an input image containing multiple targets to be tracked;

[0055] S200: Construct and train a multi-target tracking model;

[0056] S300: Input the input image into the multi-target tracking model to achieve simultaneous tracking of multiple targets in the input image.

[0057] In another embodiment, such as Figure 2 As shown, the multi-target tracking model includes: an input layer, a feature extraction layer, a feature enhancement layer, a parallel detection and tracking layer, a correlation layer, and an output layer. The following embodiment will provide a detailed description of each of the above layers.

[0058] 1. Feature extraction layer:

[0059] The feature extraction layer uses DLASeg (DLASeg is a segmentation network that adds deformable convolution to DLA (Deep Layer Aggregation)) as the backbone network. The input image is processed by the backbone network to generate basic features. (H F W represents the height of the input image after downsampling by a factor of 4. F H represents the width of the input image after downsampling by a factor of 4. F =H / 4, W F =W / 4.

[0060] 2. Feature enhancement layer:

[0061] The feature enhancement layer employs a dual-branch feature learning network (DFL) to learn features used by the detection layer for detection tasks and those used by the tracking layer for tracking tasks. This alleviates excessive competition between the two tasks and allows for the extraction of sufficient target feature information. The DFL network primarily enhances features by learning the specificity and correlation between the two tasks, and its structure is as follows: Figure 3 As shown:

[0062] The DFL network consists of two branches. These branches first employ different pooling functions (average pooling for the first branch and max pooling for the second) to reduce resolution. Then, each branch undergoes a non-shared convolutional combination (3×3 convolution + Instance Norm (a normalization method) + Leak ReLU (activation function)) to generate its respective feature maps A1 and A2 for cross-computation. These feature maps are then combined with the initial feature maps through matrix multiplication and addition to obtain the output. Specifically, the DFL network first obtains shared features from the backbone network. To reduce the computational cost of feature map matrix operations, it is first necessary to optimize the shared features. Pooling operations are performed, and different pooling methods are required for different tasks. Avgpool yields features that are more sensitive to background information and can be used for learning detection features in detection layers; Maxpool yields features that are more sensitive to texture information and can be used for learning tracking features in tracking layers. Shared features. After pooling, two types of features containing local information are obtained, namely detection features. and tracking features Next, f1 and f2 are encoded through 3×3 convolutional layers to generate feature maps A1 and A2 for detection and tracking, and then reshaped (using the Reshape function) into a size of C×H′. F W′ F Two-dimensional tensors M1 and M2. Then, for M1 and M2 and their corresponding transpose tensors... and Matrix multiplication is performed, and the results are normalized using the softmax function to calculate the task-specific response map s. k ∈R C×C The calculation method is as follows:

[0063]

[0064] Where · represents the dot product of two vectors. and M respectively k The i, j and l rows, For S k The value at position (i, j) represents the correlation between the i-th channel and the j-th channel in the feature map. C represents the number of feature channels, and C = 64.

[0065] Next, M1 and M2 and Matrix multiplication is performed separately, where T denotes the transpose operation, to learn the correlations between different tasks, and the correlation response map R between tasks is obtained after normalization. k ∈R C×C The calculation method is as follows:

[0066]

[0067] in, This represents the correlation between the i-th channel of Task 1 and the j-th channel of Task 2. Let (k, h) represent the correlation between the i-th channel of Task 2 and the j-th channel of Task 1, and let (k, h) represent different tensor combinations, such as (1, 2) representing tensors M1 and M2. The larger the value, the greater the degree to which the feature information of the channel is of common interest to both tasks.

[0068] Finally, the specificity response map and the correlation response map are fused using a trainable parameter λ to obtain the feature-enhanced response map W. k ∈R C×C The calculation method is as follows:

[0069] W k =λ k×S k +(1-λ k )×R k k∈{1,2}

[0070] Where, λ k Let λ1 represent the trainable parameters, and let λ2 represent the trainable parameters of the response graph W1.

[0071] The enhanced response maps W1 and W2 corresponding to different tasks are matrix-multiplied with the reshaped input features to obtain the enhanced features for each task. These enhanced features are then reshaped to match the input features. Three-dimensional tensors of the same shape are fused with f to prevent information loss. This results in two features for detecting and tracking branch inputs.

[0072] 3. Detection layer and tracking layer:

[0073] The detection layer consists of three output heads, each head comprising two convolutional layers and ReLU activation functions within those two convolutional layers. The first convolutional layer is a 3×3 convolution, used to increase the number of feature channels from 64 to 256; the second convolutional layer is a 1×1 convolution, used to reduce the number of feature channels from 256 to 1 or 2 (center point: H). F ×W F ×1; Center point offset: H F ×W F ×2; Width and Height: H F ×W F ×2).

[0074] The Re-identification network in the tracking layer consists of four convolutional layers. The first two layers are depthwise separable convolutions, namely a channel-wise convolution and a point-wise convolution, both of which are 1×1 convolutions, used to increase the number of feature channels from 64 to 128. After that, it goes through a BatchNorm2d normalization and a ReLU activation function. The third convolution is a 3×3 convolution with the same number of channels. After that, it goes through another BatchNorm2d normalization and a ReLU activation function. The fourth convolution is a 1×1 convolution with the same number of channels.

[0075] To utilize more temporal information, this embodiment also introduces an AM matrix in the tracking layer. The AM matrix uses extracted features to construct a similarity relationship between two frames, thereby predicting a more accurate offset vector and reducing the number of identity switching events of the target being tracked in the input image. The AM matrix is ​​represented as follows: It can be done (Features extracted through ReID) and e t-1The corresponding transpose tensor is multiplied by matrix to obtain matrix A, which represents the image I. t and I t-1 The similarity between them is calculated as follows:

[0076]

[0077] Where T is the transpose symbol, A i，j，m，n Let A represent the feature similarity between target point (i, j) in frame t and target point (m, n) in frame (t-1). For the center point (i, j) of target x in frame t, its corresponding two-dimensional correlation matrix can be obtained from matrix A. This represents the feature similarity between the target x and all points on the (t-1)th frame of the image.

[0078] Next, through A i，j Find the offset vector, which is to apply max pooling to A in both the horizontal and vertical directions. i，j The pooling kernels are H F ×1 and 1×W F This yields matrices in two directions. and Will and After normalization using the softmax function, two vectors are obtained. and in, and Let represent the probabilities of the target appearing in the horizontal and vertical directions in frame t-1, respectively. Offset templates for the two directions are defined based on the output image resolution, respectively. and This represents the offset value where the target actually appears in another location, calculated as follows:

[0079] X i，j，n = (nj)×s 1≤n≤W F

[0080] Y i，j，m = (mi)×s 1≤m≤H F

[0081] Where s represents the downsampling factor, set to 4, X i，j，n and Y i，j，m Let these represent the offsets of the target at positions (*, n) and (m, *) in frame t-1, respectively. The final tracking offset is obtained by the dot product of the actual target position offset and its offset probability at its corresponding position:

[0082]

[0083] The horizontal and vertical position offsets are learned through two channels respectively, and the final tracked offset is obtained as follows: Used for subsequent trajectory association.

[0084] Most existing trackers filter out low-scoring bounding boxes when their scores fall below a threshold. However, these low-scoring bounding boxes may have dropped due to occlusion or other reasons, so simply filtering them out can easily lead to missing tracks. To address this issue, this embodiment introduces a multi-level trajectory association (MTA) strategy in the association layer to reduce missing tracks and further improve tracking performance in complex scenarios. Figure 4 As shown, the MTA strategy reclassifies the detection boxes into high-scoring and low-scoring boxes based on a set threshold. High-scoring boxes can extract accurate target feature information, thereby achieving long-term association of the target; low-scoring boxes can be used to recover missing trajectories. Different matching methods are used for these two types of scoring boxes to achieve association with the trajectory.

[0085] During trajectory association, firstly, high-resolution detection boxes are matched with the trajectory of the previous frame using a simple greedy algorithm based on their offset vectors, generating unmatched detection boxes, successfully matched trajectories, and unmatched trajectories. Then, the cosine similarity between the features of the unmatched detection boxes and the tracking trajectory of the previous frame is calculated for secondary matching. If the similarity is below a threshold, a new trajectory is created, thus enabling re-association after target occlusion and reducing the number of IDs. If the similarity is above the threshold, a successful match is achieved, and the trajectories are added to the trajectory. The feature information f of the i-th trajectory in frame t is then updated. i t The calculation formula is shown below:

[0086]

[0087] in, ε represents the current frame image features extracted by the ReID network, and ε is the weight. Unmatched trajectories may be caused by low-scoring detection boxes being filtered out due to occlusion. Therefore, a secondary matching process is performed between low-scoring detection boxes and unmatched trajectories to recover lost trajectories in scenarios such as occlusion.

[0088] In another embodiment, this disclosure uses the MOT17 dataset to train the model. The MOT17 dataset consists of 7 video sequences for training and 7 sequences for testing. MOT17 includes bounding boxes generated by three different object detectors: DPM, Faster R-CNN, and SDP.

[0089] In this embodiment, the first half of each video in the MOT17 dataset is used as the training set to train the model, and the second half of each video is used as the test set to test the model. The model is trained using the following method:

[0090] The model is trained using the training set, and the training parameters are set as follows: batch size is set to 32, and initial learning rate is set to 1.25 × 10⁻⁶. -4 The number of training rounds is set to 70. Training is complete when the model has been trained 70 times.

[0091] The trained model was tested using a test set. During the testing process, multi-object tracking accuracy (MOTA) and IDF1 score were used as evaluation metrics. MOTA represents the overall performance of the tracker and is measured by evaluating three sources: false negatives (FN), false positives (FP), and identity switching counts (IDs). The calculation method is shown below:

[0092]

[0093] GT stands for Ground Truth, which represents the number of objects labeled in the scene.

[0094] The IDF1 score represents the correlation performance of the tracker, which is the ratio of the number of correct target detections to the average of the sum of the true number and the calculated number of detections. The calculation method is as follows:

[0095]

[0096] Among them, IDTP is the True Positive ID, which represents the number of correctly assigned targets in the entire video; IDFN is the False Negative ID, which represents the number of unassigned targets in the entire video; and IDFP is the False Positive ID, which represents the number of incorrectly assigned targets in the entire video.

[0097] In summary, the model passes the test when the tracking accuracy reaches 66.1% and the IDF1 score reaches 64.2%.

[0098] The following is a combination of the present disclosure Figures 5(a) to 7(b) Tables 1 and 2 provide a detailed explanation of the effectiveness of the model described in this disclosure.

[0099] First, the effectiveness of each module in this model is verified through ablation experiments, as shown in Table 1:

[0100] Table 1

[0101]

[0102] In Table 1, the first row of data represents the ablation experimental results using the benchmark algorithm CenterTrack.

[0103] The second row of data presents the ablation experiment results with the AM matrix. Compared to the first row, using the AM matrix for offset prediction improved the MOTA value by 1.0%, the IDF1 score by 4.4%, and reduced IDs from 528 to 369. The baseline algorithm CenterTrack uses regression learning to predict the offset vector from the center point of the current frame to the center point of the previous frame. This offset vector does not fully utilize temporal information. The AM matrix, however, is composed of similarity relationships between adjacent frames, containing more temporal information, thus predicting a more accurate offset vector. Therefore, using the offset vector predicted by AM for trajectory association can significantly reduce the number of target IDs, thereby improving the association capability IDF1.

[0104] The third row of data shows the ablation experiment results using the DFL network. Compared to the second row, MOTA improved by 0.3% and IDF1 by 0.5%, thus improving the overall performance of the model. This is mainly because the two branches of the DFL network enhance the shared features of the input separately to obtain detection features and tracking features, thereby alleviating the competition between detection and tracking.

[0105] The fourth row of data presents the ablation experiment results using the MTA strategy after introducing the AM matrix. Compared to the second row, MOTA improved performance by 0.8%, but the IDF1 score decreased by 0.8%, indicating an overall improvement in tracker performance but a decrease in correlation ability. In the experiment, detection boxes were divided using confidence η, where high-scoring detection boxes range from η ≥ 0.4, and low-scoring detection boxes range from 0.2 ≤ η < 0.4. Because the MTA strategy utilizes some low-scoring detection boxes, retaining them leads to an increase in false positives (FP), thus decreasing the IDF1 score. Compared to the first row of data, the MT increased by 3.2% after adopting the MTA strategy, indicating a significant reduction in missing trajectories. MT represents the percentage of at least 80% of the target's trajectory being correctly tracked during video tracking.

[0106] The last row of data shows the ablation experiment results after adding the AM, DFL, and MTA modules simultaneously. Compared with the first row of data, MOTA and IDF1 improved by 2.1% and 4.3% respectively, and IDs decreased from 528 to 333. Therefore, it can be proved that the model described in this disclosure can effectively improve the performance of multi-target tracking in complex scenarios.

[0107] Furthermore, this disclosure also selected video scenes from the MOT17 test set to conduct a qualitative analysis of the model described herein, and compared its target tracking performance with the benchmark algorithm CenterTrack. Figures 5(a) and 5(b) respectively show partial visualization results of the benchmark algorithm CenterTrack and the model described herein in the MOT17-04 video sequence. From left to right, these are before occlusion, during occlusion, and after occlusion, with the frame number in the lower right corner. In distant scenes, targets are easily affected by local occlusion. As can be seen from Figure 5(a), targets 103 and 128 at the arrow exhibit IDs after being occluded, switching to a new trajectory identifier. However, as shown in Figure 5(b), the benchmark algorithm CenterTrack, after introducing the DFL network, AM matrix, and MTA strategy, can maintain the original target ID in occluded scenes, improving the tracking performance of multi-target trackers.

[0108] Figures 6(a) and 6(b) show the visualization results of the benchmark algorithm CenterTrack and the model described in this disclosure in MOT17-09, respectively. In close-range scenes, targets are easily affected by severe occlusion. As shown in Figure 6(a), target 21 at the arrow generates a new ID value of 29 after experiencing complete occlusion; while as shown in Figure 6(b), the model described in this disclosure can achieve re-association after occlusion. That is, after complete occlusion, when the target reappears, appearance features can be extracted through ReID and cosine distance matching with the trajectory to cope with strong occlusion scenes, thus improving the tracking performance of the tracker.

[0109] Figures 7(a) and 7(b) show the visualization results of the baseline algorithm CenterTrack and the model described in this disclosure on MOT17-11, respectively. In some scenarios, the target's detection box score decreases as the occlusion level increases, and low-scoring detection boxes are usually filtered out by the detector, resulting in missing trajectories. As shown in Figure 7(a), the target under the baseline algorithm CenterTrack is not detected due to occlusion and therefore cannot be tracked; while as shown in Figure 7(b), the model described in this disclosure retains some low-scoring detection boxes in the MTA strategy part, thus enabling continuous tracking of these targets, reducing the phenomenon of missing trajectories, and thereby improving the tracking performance of the tracker.

[0110] To further verify the effectiveness of the model described in this disclosure, six advanced MOT algorithms were selected: CTRracker, JDE, CenterTrack, QuasiDense, TransTrack, and MOTR. Comparative analysis was conducted on the MOT17 and MOT20 datasets, and the results are shown in Table 2.

[0111] Table 2

[0112]

[0113] As shown in Table 2, the model described in this disclosure achieved MOTA and IDF1 scores of 68.2% and 68.5% respectively on the MOT17 dataset, and 52.7% and 48.2% respectively on the MOT20 dataset, achieving the best tracking results compared to the other 6 algorithms.

[0114] On MOT20, compared with the benchmark algorithm CenterTrack, the model described in this disclosure improves the MOTA index by 1.4%, the IDF1 index by 7.9%, reduces IDS from 7731 to 3043, increases FP from 10080 to 13403, and reduces FN from 281757 to 274419. These results demonstrate that the model described in this disclosure can effectively address the problems of insufficient target feature extraction, identity switching, and trajectory loss in dense scenes.

[0115] In another embodiment, this disclosure also provides a multi-target tracking device based on bi-branch feature enhancement and multi-level trajectory association, the device comprising:

[0116] The acquisition module is used to acquire an input image containing multiple targets to be tracked;

[0117] The model building and training module is used to build and train a multi-object tracking model to obtain a well-trained multi-object tracking model.

[0118] The multi-target tracking model utilizes a dual-branch feature learning network to alleviate the excessive competition between detection and tracking tasks, and introduces correlation matrix AM prediction to obtain a more accurate offset vector to reduce the number of identity switching of the target to be tracked in the input image.

[0119] The tracking module is used to input the input image into the trained multi-target tracking model to achieve simultaneous tracking of multiple targets in the input image.

[0120] In another embodiment, the model building and training module includes:

[0121] The sub-module is used to divide the dataset for model training into training and testing sets;

[0122] The training submodule is used to train the model using the training set;

[0123] The test submodule is used to test the trained model using a test set.

[0124] In another embodiment, this disclosure also provides a computer storage medium storing computer-executable instructions for performing any of the methods described above.

[0125] In another embodiment, this disclosure also provides an electronic device, including:

[0126] Memory, processor, and computer programs stored in memory and executable on the processor, wherein,

[0127] When the processor executes the program, it implements the method described in any of the preceding methods.

[0128] The applicant has provided a detailed description of the embodiments of this disclosure in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred examples of this disclosure and are not limited to the specific embodiments described above. The detailed description is intended to help readers better understand the spirit of this disclosure and is not intended to limit the scope of protection of this disclosure. On the contrary, any improvements or modifications made based on the inventive spirit of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multi-target tracking method based on bi-branch feature enhancement and multi-level trajectory association, characterized in that, The method includes the following steps: S100: Acquire an input image containing multiple targets to be tracked; S200: Construct and train a multi-target tracking model to obtain a well-trained multi-target tracking model; The multi-target tracking model utilizes a dual-branch feature learning network to alleviate the excessive competition between detection and tracking tasks, and introduces correlation matrix AM prediction to obtain a more accurate offset vector to reduce the number of identity switching of the target to be tracked in the input image. The multi-target tracking model includes: an input layer, a feature extraction layer, a feature enhancement layer, a parallel detection and tracking layer, a correlation layer, and an output layer; The feature enhancement layer employs a dual-branch feature learning network (DFL), which first obtains shared features from the backbone network. To reduce the computational cost of feature map matrix operations, it is first necessary to analyze the shared features. Pooling operations are performed, and different pooling methods are required for different tasks. Average pooling yields features that are more sensitive to background information and is used for learning detection features in the detection layer; max pooling yields features that are more sensitive to texture information and is used for learning tracking features in the tracking layer; shared features... After pooling, two types of features containing local information are obtained, and these are used for feature detection. and tracking features Secondly, , Each feature map is encoded using a 3×3 convolutional layer to generate features for detection and tracking. and And reshaped into a size of Two-dimensional tensor and Then, for and and its corresponding transpose tensor and Perform matrix multiplication and normalize using the softmax function to calculate the task-specific response map. The calculation method is as follows: ， Where · represents the dot product of two vectors. , and They represent The , and OK, for superior The value of the position represents the first position in the feature map. The first channel and the first The correlation between the channels is given by C, which represents the number of feature channels, and C=64. Next, and , and Perform matrix multiplication separately, where, This represents the transpose operation, used to learn the correlations between different tasks, and then normalized to obtain the correlation response map between tasks. The calculation method is as follows: ， in, Indicates the first task of task 1 The first channel is for Task 2. The correlation between the channels This indicates the first task of task 2. The first channel is for Task 1. The correlation between the channels is shown in (k, h), which represents different tensor combinations, and (1, 2) represents tensors M1 and M2. The larger the value, the greater the degree to which the feature information of that channel is of common interest to both tasks; Finally, through a trainable parameter By fusing the specificity response map and the correlation response map, a feature-enhanced response map is obtained. The calculation method is as follows: ， in, Indicates trainable parameters, Representation of response graph Trainable parameters, Representation of response graph Trainable parameters; Enhance response maps of features corresponding to different tasks and The augmented features for each task are obtained by matrix multiplication with the reshaped input features. These augmented features are then reshaped to resemble the input features. Three-dimensional tensors of the same shape, and with The features are fused to prevent information loss, ultimately yielding two features for detecting branches and tracking branch inputs. An AM matrix is ​​introduced into the tracking layer. The AM matrix utilizes extracted features to construct a similarity relationship between two frames, thereby predicting a more accurate offset vector and reducing the number of identity switching events of the target being tracked in the input image. The AM matrix is ​​represented as follows: ,pass and The matrix is ​​obtained by performing matrix multiplication on the corresponding transpose tensor. , representing an image and The similarity between them, of which ,matrix The calculation method is as follows: ， in, It is the transpose symbol. Indicates the first Frame image target point With the Frame image target point Feature similarity between them, for the first Frame target center point From the matrix The corresponding two-dimensional correlation matrix is ​​obtained from it. , indicating the target and the Feature similarity among all points in a frame image; Next, through Find the offset vector, and apply max pooling in both the horizontal and vertical directions. The pooling kernels are respectively and This yields matrices in two directions. and ,Will and After normalization using the softmax function, two vectors are obtained. and ,in, and These respectively indicate that the target appears in the th position. The probabilities of horizontal and vertical positions on the frame; offset templates in the two directions are defined based on the output image resolution, respectively. and This represents the offset value at which the target actually appears in other locations, calculated as follows: ， ， in, This indicates the downsampling factor, set to 4. and They represent the target at the 1st and 2nd positions, respectively. Position appearing on the frame and location The offset; the final tracking offset is obtained by the dot product of the actual target position's offset value and its offset probability at its corresponding position: ， The horizontal and vertical position offsets are learned through two channels respectively, and the final tracked offset is obtained as follows: This is used for subsequent trajectory association; S300: Input the input image into the trained multi-target tracking model to achieve simultaneous tracking of multiple targets in the input image.

2. The method according to claim 1, characterized in that, In step S200, the multi-target tracking model is trained using the following method: S201: Obtain the dataset and divide it into training and test sets; S202: Set training parameters, train the model using the training set, and the model training is complete when the set number of training rounds is reached; S203: Test the trained model using the test set. During the testing process, multi-object tracking accuracy and... The score is used as an evaluation metric to assess the model. When the tracking accuracy reaches 66.1%, and The score reached 64.2%, and the model test was passed.

3. The method according to claim 2, characterized in that, In step S203, the multi-target tracking accuracy is expressed as follows: ， Here, FN represents false negative, FP represents false positive, IDS represents identity switching count, and GT stands for Ground Truth, which represents the number of targets labeled in the scene.

4. The method according to claim 2, characterized in that, In step S203, the IDF1 score is represented as: ， in, It is a true positive ID, indicating the number of correctly assigned targets in the entire video; It is a false negative ID, indicating the number of targets that were missed in the entire video; It is a false positive ID, representing the number of targets that were incorrectly assigned in the entire video.

5. A multi-target tracking device based on bi-branch feature enhancement and multi-level trajectory association, characterized in that, The device includes: The acquisition module is used to acquire an input image containing multiple targets to be tracked; The model building and training module is used to build and train a multi-object tracking model to obtain a well-trained multi-object tracking model. The multi-target tracking model utilizes a dual-branch feature learning network to alleviate the excessive competition between detection and tracking tasks, and introduces correlation matrix AM prediction to obtain a more accurate offset vector to reduce the number of identity switching of the target to be tracked in the input image. The multi-target tracking model includes: an input layer, a feature extraction layer, a feature enhancement layer, a parallel detection and tracking layer, a correlation layer, and an output layer; The feature enhancement layer employs a dual-branch feature learning network (DFL), which first obtains shared features from the backbone network. To reduce the computational cost of feature map matrix operations, it is first necessary to analyze the shared features. Pooling operations are performed, and different pooling methods are required for different tasks. Average pooling yields features that are more sensitive to background information and is used for learning detection features in the detection layer; max pooling yields features that are more sensitive to texture information and is used for learning tracking features in the tracking layer; shared features... After pooling, two types of features containing local information are obtained, and these are used for feature detection. and tracking features Secondly, , Each feature map is encoded using a 3×3 convolutional layer to generate features for detection and tracking. and And reshaped into a size of Two-dimensional tensor and Then, for and and its corresponding transpose tensor and Perform matrix multiplication and normalize using the softmax function to calculate the task-specific response map. The calculation method is as follows: ， Where · represents the dot product of two vectors. , and They represent The , and OK, for superior The value of the position represents the first position in the feature map. The first channel and the first The correlation between the channels is given by C, which represents the number of feature channels, and C=64. Next, and , and Perform matrix multiplication separately, where, This represents the transpose operation, used to learn the correlations between different tasks, and then normalized to obtain the correlation response map between tasks. The calculation method is as follows: ， in, Indicates the first task of task 1 The first channel is for Task 2. The correlation between the channels This indicates the first task of task 2. The first channel is for Task 1. The correlation between the channels is shown in (k, h), which represents different tensor combinations, and (1, 2) represents tensors M1 and M2. The larger the value, the greater the degree to which the feature information of that channel is of common interest to both tasks; Finally, through a trainable parameter By fusing the specificity response map and the correlation response map, a feature-enhanced response map is obtained. The calculation method is as follows: ， in, Indicates trainable parameters, Representation of response graph Trainable parameters, Representation of response graph Trainable parameters; Enhance response maps of features corresponding to different tasks and The augmented features for each task are obtained by matrix multiplication with the reshaped input features. These augmented features are then reshaped to resemble the input features. Three-dimensional tensors of the same shape, and with The features are fused to prevent information loss, ultimately yielding two features for detecting branches and tracking branch inputs. An AM matrix is ​​introduced into the tracking layer. The AM matrix utilizes extracted features to construct a similarity relationship between two frames, thereby predicting a more accurate offset vector and reducing the number of identity switching events of the target being tracked in the input image. The AM matrix is ​​represented as follows: ,pass and The matrix is ​​obtained by performing matrix multiplication on the corresponding transpose tensor. , representing an image and The similarity between them, of which ,matrix The calculation method is as follows: ， in, It is the transpose symbol. Indicates the first Frame image target point With the Frame image target point Feature similarity between them, for the first Frame target center point From the matrix The corresponding two-dimensional correlation matrix is ​​obtained from it. , indicating the target and the Feature similarity among all points in a frame image; Next, through Find the offset vector, and apply max pooling in both the horizontal and vertical directions. The pooling kernels are respectively and This yields matrices in two directions. and ,Will and After normalization using the softmax function, two vectors are obtained. and ,in, and These respectively indicate that the target appears in the th position. The probabilities of horizontal and vertical positions on the frame; offset templates in the two directions are defined based on the output image resolution, respectively. and This represents the offset value at which the target actually appears in other locations, calculated as follows: ， ， in, This indicates the downsampling factor, set to 4. and They represent the target at the 1st and 2nd positions, respectively. Position appearing on the frame and location The offset; the final tracking offset is obtained by the dot product of the actual target position's offset value and its offset probability at its corresponding position: ， The horizontal and vertical position offsets are learned through two channels respectively, and the final tracked offset is obtained as follows: This is used for subsequent trajectory association; The tracking module is used to input the input image into the trained multi-target tracking model to achieve simultaneous tracking of multiple targets in the input image.

6. The apparatus according to claim 5, characterized in that, The model building and training module includes: The sub-module is used to divide the dataset for model training into training and testing sets; The training submodule is used to train the model using the training set; The test submodule is used to test the trained model using a test set.

7. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions for performing the method described in any one of claims 1 to 4.

8. An electronic device, characterized in that, include: Memory, processor, and computer programs stored in memory and executable on the processor, wherein, When the processor executes the program, it implements the method according to any one of claims 1 to 4.

Images