Multi-target tracking method based on joint detection
By using PVT and Guided Anchoring structures in the multi-objective tracking model, the two-way softmax function is optimized, and the cosine similarity and IoU distance are fused, QDTrack's problems in capturing remote features and positioning accuracy are solved, significantly improving detection accuracy and tracking performance.
Patent Information
- Application Number
- CN202410975506.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-19
AI Technical Summary
The existing multi-objective tracking model QDTrack is difficult to effectively capture when capturing interactions between targets or key features spanning a larger spatial range within the target, resulting in insufficient detection accuracy and tracking performance.
Pyramid Vision Transformer (PVT) is used as the backbone network, and guided anchor box generation is combined in the RPN structure, and target matching is carried out through the optimized bidirectional softmax function.
Improve the model's remote modeling ability and feature map information positioning ability, reduce the number of missed and false detections, improve the tracking performance, especially maintain good continuous tracking ability under occlusion, and achieve excellent MOTA, MT and ML Indicator results.
Smart Images

Figure CN118918143A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of target tracking, and in particular relates to a multi-target tracking method based on joint detection. Background Art
[0002] Driven by social needs and technological progress, multi-target tracking technology has made significant progress, and the development of multi-target tracking algorithms has matured. QDTrack is a two-stage tracking model based on Faster R-CNN, which densely samples hundreds of region proposals on a pair of images for comparative learning through quasi-dense similarity learning. This tracking model does not require additional motion priors. In the process of tracking and matching, it only calculates similarity through a simple bidirectional softmax function, and achieves efficient tracking through a simple matching method.
[0003] However, the original QDTrack uses ResNet as the feature extraction network. The interactions between objects or the key features within the objects that span a large spatial range may be difficult to capture through local convolution operations. How to optimize the QDTrack model has become an urgent problem to be solved. Summary of the invention
[0004] In order to solve the above technical problems, the present invention proposes a multi-target tracking method based on joint detection to solve the problems existing in the above prior art.
[0005] To achieve the above object, the present invention provides a multi-target tracking method based on joint detection, comprising:
[0006] Use the PVT model as the backbone network to extract image features;
[0007] Positioning and identifying target features in the image based on the GA-RPN structure to obtain a candidate target region;
[0008] The bidirectional softmax function is optimized based on cosine similarity and IoU distance, and the targets in the candidate target area are matched by the optimized bidirectional softmax function to generate a multi-target tracking result.
[0009] Preferably, the process of extracting image features using the PVT model as a backbone network includes:
[0010] The PVT model controls the scale of the feature map through a progressive shrinkage strategy;
[0011] For the i-th stage, the feature map F from the previous stage input i-1 (H i-1 ×Wi-1 ×C i-1 ) is divided into patches, and then flatten each patch and map it to C i dimensional feature embedding, and then after linear mapping, the feature size can be regarded as
[0012] Preferably, the PVT model adopts a spatial scaling attention mechanism;
[0013] The expression of the spatial scaling attention mechanism is:
[0014]
[0015] Among them, Concat() is the concatenation operation in Transformer. is the weight parameter, N i is the number of attention heads in the i-th stage, so the dimension of each head is LSR() performs a linear dimensionality reduction operation on the spatial dimension of the input sequence.
[0016] Preferably, the features of the target in the image are located and identified based on the GA-RPN structure, and the process of obtaining the candidate target area also includes: constructing a quasi-dense similarity learning model based on a joint learning model of sparse ID loss, sampling the surrounding area of the candidate target area based on the quasi-dense similarity learning model, and obtaining positive and negative samples of the information area.
[0017] Preferably, the expression for predicting the optimal shape of each position by the GA-RPN structure is:
[0018] w=σ·s·e dw ;
[0019] h=σ·s·e dh ;
[0020] where w and h are the optimal shapes at each location, i.e., the shapes that result in the highest coverage with the nearest true label bounding box, s is the stride, and σ is the scaling factor.
[0021] Preferably, the GA-RPN structure is further used to add a 1×1 convolution to the output of the shape prediction branch to predict the offset domain, and then map the offset into a 3×3 convolution to obtain an adaptive feature map.
[0022] Preferably, the target in the candidate target area is matched by the optimized bidirectional softmax function, and the process of generating the multi-target tracking result includes: constructing an IoU threshold β iou , when the IoU threshold of the candidate target is greater than βiou When , cosine similarity will be given priority for matching, and the remaining value is less than β iou The candidate target with detection confidence higher than β obj , then consider matching the detected target with the existing trajectory, if the matching score is greater than β m , then the match is determined, and the unmatched candidate target is determined if the detection confidence is higher than β new , a new trajectory is created.
[0023] Preferably, the loss function of the overall network is expressed as:
[0024] L=L det +γ 1 L embed +γ 2 L aux ;
[0025] Among them, L det is the original FasterR-CNN loss function, including RPN loss, category loss and regression loss, L aux Auxiliary loss.
[0026] Compared with the prior art, the present invention has the following advantages and technical effects:
[0027] The present invention studies the tracking model QDTrack using quasi-dense similarity learning, and optimizes the model in three aspects. First, the present invention replaces the ResNet50 backbone network in the original tracking model with the Pyramid Vision Transformer to improve the remote modeling capability of the model. Secondly, the present invention uses the GA-RPN with position guidance to improve the model's ability to locate information in the feature map. Experiments have shown that these two improved methods effectively improve the detection accuracy of the model, reduce the number of missed detections and false detections, and thus improve the tracking performance of the model. Finally, the present invention proposes a bidirectional softmax matching that integrates cosine similarity to optimize the original matching method, and selects the optimal parameter value through experiments. Compared with the mainstream algorithm, the improved model achieves the best results in MOTA, MT and ML indicators, and has better continuous tracking capabilities when the tracking target faces occlusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0029] Figure 1 This is a diagram of the optimized model structure of an embodiment of the present invention;
[0030] Figure 2 A schematic diagram of the structure of a PVT network module according to an embodiment of the present invention;
[0031] Figure 3 Schematic diagram of the GA-RPN network structure of an embodiment of the present invention;
[0032] Figure 4 This is a schematic diagram of comparing the MOTA indicators of each sequence of the MOT16 test set according to an embodiment of the present invention;
[0033] Figure 5 Schematic diagram of MOTA index comparison of each sequence of the MOT17 test set according to an embodiment of the present invention. DETAILED DESCRIPTION
[0034] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0035] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0036] Embodiment 1
[0037] This embodiment provides a multi-target tracking method based on joint detection, including:
[0038] The original QDTrack uses ResNet as the feature extraction network. It may be difficult to capture the key features of the interaction between objects or the large spatial range within the object through local convolution operations. Therefore, this embodiment uses Pyramid Vision Transformer as the backbone network to enhance the long-range modeling ability of the network, and introduces Guided Anchoring to the RPN result to guide the generation of anchor boxes. The optimized network structure is as follows: Figure 1 As shown;
[0039] Transformer-based detection models have been widely used in computer vision tasks, and they have achieved advanced performance as backbone networks. However, most models only propose attention mechanisms similar to Transformer, and rarely use non-convolutional Transformers for feature extraction. For dense prediction tasks, the long-range modeling capabilities of Transformer can better extract feature information and provide better support for subsequent processing. Pyramid Vision Transformer (PVT) overcomes the problems of low output resolution and high computational cost. The network can achieve feature scaling and channel deepening as the number of layers increases, just like traditional convolutional neural networks, thereby constructing a feature pyramid structure to generate multi-scale feature maps.
[0040] like Figure 2 As shown in Figure 2, similar to the CNN backbone network, PVT consists of four stages, each of which controls the scale of the feature map through a progressive shrinking strategy. For the i-th stage, the feature map F from the previous stage input i-1 (H i-1 ×W i-1 ×C i-1 ) is divided into patches, and then flatten each patch and map it to C i After linear mapping, the feature size can be regarded as The size of the feature map is reduced by P compared to the previous stage. i times.
[0041] Each stage contains L i Transformer encoders, each consisting of an attention layer and a feed-forward layer. The traditional Transformer uses a multi-head attention mechanism (MHA), which greatly increases the computational cost. PVT uses a spatial scaling attention mechanism (SRA) to replace MHA. Similar to MHA, SRA receives a query Q, a key K, and a value V as input and outputs a refined feature. The difference is that SRA reduces the spatial scale of K and V before the attention operation. The specific operation is shown in equations (1)-(2):
[0042]
[0043] Among them, Concat() is the concatenation operation in Transformer. is the weight parameter. iis the number of attention heads in the i-th stage, so the dimension of each head is LSR() is a linear dimensionality reduction operation on the spatial dimension of the input sequence, as shown in formula (3):
[0044] LSR(x)=Norm(Pooling(x,P)W S ) (3)
[0045] in, W represents the length of the input sequence, P represents the size of the attention layer reduction at each stage, and the default value is 7. S is to reduce the input sequence to C i dimensional linear projection operation. Similar to the attention calculation of the original Transformer, the final calculation is shown in formula (4):
[0046]
[0047] By adopting the PVT structure, the improved network obtains more local continuity of feature images, the input variable resolution is more flexible, and has the same linear complexity as CNN.
[0048] Anchors are an important part of target detection models. Most detectors rely on generating dense anchors. A set of anchors of predefined scales are generated by uniformly sampling in the image. However, this method is very inefficient. This embodiment optimizes the RPN structure in the original tracking model and introduces the Guided Anchoring structure to guide the generation of anchors.
[0049] GA-RPN uses semantic features to guide the positioning of anchor boxes. This method also predicts the possible locations of the center of the object of interest and the scales of different locations. Figure 3 As shown in the figure, the module is divided into two parts: anchor generation and feature adaptation. Anchor generation is divided into position prediction and shape prediction. In the position prediction branch, the feature map is convolved by 1×1 to obtain the score map of each position, and then converted into the corresponding probability value by the sigmoid function. By selecting the probability threshold, most of the background area can be filtered out.
[0050] After determining the possible locations of the objects, the next step is to determine the shape of the possible objects at each location through the shape prediction branch. Unlike traditional bounding box regression, the shape prediction branch does not change the position of the anchor. Specifically, for a given feature map, this branch will predict the best shapes w and h for each location, that is, the shapes that may result in the highest coverage with the nearest true label bounding box. However, from practical experience, directly predicting the length and width is unstable because the range of values is large. Simplified predictions can be made using equations (5) and (6).
[0051] w=σ·s·e dw (5)
[0052] h=σ·s·e dh (6)
[0053] Where s is the stride and σ is the scale factor (set to 8 by default). The shape prediction branch outputs dw and dh, which are then mapped to w and h. This transformation maps the output values to the interval [-1,1], making it easier to learn stably. In the shape prediction branch, the conversion is achieved by a 1×1 convolution to produce a dual-channel map containing dw and dh.
[0054] In the traditional RPN network, the position of the anchor is uniform across the entire feature map, with the same shape and scale at each position, so the feature map can learn consistent representations. However, with the position and shape prediction branches, the size of the anchor is not fixed. Larger anchors should encode larger regional content, and smaller anchors should encode smaller regional content. Therefore, the network can be guided to achieve adaptive feature conversion based on the prediction of the anchor shape. A 1×1 convolution is added to the output of the shape prediction branch to predict the offset domain, and then the offset is mapped in a 3×3 convolution to obtain an adaptive feature map.
[0055] The original tracking model uses a bidirectional softmax function to calculate the similarity for tracking and matching. This embodiment further optimizes the matching process on this basis. By introducing the cosine similarity function, a more accurate association can be achieved in combination with IoU. Although the use of appearance similarity is a reliable indicator in most scenarios, for dense target objects, the matching accuracy can be improved by combining the IoU distance. Under the conditions of blurred images and some fast-moving targets, the appearance similarity is not reliable. If the search and match are performed within a certain IoU distance, the interference of similar targets can be effectively avoided. Therefore, this embodiment re-optimizes and proposes a new association matching method.
[0056] Cosine Similarity is a measure of the similarity between two vectors. It is usually used to calculate the similarity in the fields of text, images, user preferences, etc. Cosine similarity measures the cosine value of the angle between two vectors. The closer the value is to 1, the higher the similarity is, and the closer the value is to -1, the lower the similarity is. For two vectors a and b, the calculation formula of cosine similarity is:
[0057]
[0058] Cosine similarity focuses on the direction of the vector rather than its length, and is therefore insensitive to changes in scale and contrast in the image. This means that when the images are scaled or have varying brightness, cosine similarity is better able to capture the similarities between images.
[0059] In trajectory matching, this embodiment introduces the IoU threshold β iou , when the IoU threshold of the candidate target is greater than β iou When , cosine similarity will be given priority for matching. The remaining is less than β iou The candidate target with detection confidence higher than β obj , then consider matching the detected target with the existing trajectory, if the matching score is greater than β m , then the match is determined. If the detection confidence of the unmatched candidate target is higher than β new , a new track is created. The remaining unprocessed candidate targets are placed in the background. The activated track is the target in the previous frame that has completed the match with the current candidate target. The unmatched track is in an inactive state. If the track is not activated within K frames, it will be removed and no longer considered for matching. In general, K defaults to 30. The optimized tracking and matching process pseudo code is shown below.
[0060]
[0061] The configuration information of the hardware and software used in the experiment of this embodiment is shown in Table 1.
[0062] Table 1
[0063]
[0064] To ensure fairness, the experiment in this embodiment follows the parameter settings of the original QDTrack model training. 128 RoIs are selected from the key frames as training samples, and 256 RoIs are selected from the reference frame with a positive-negative ratio of 1 as comparison targets. RoIs are sampled using IoU-balanced sampling. Four convolutional layers and one fully connected layer are used as Embedding Head for feature extraction, and the size of the feature dimension is 256. The experiment in this embodiment uses AdamW as the optimizer, with an initial learning rate of 1e-4, a weight decay coefficient of 1e-4, and a batch size of 2. Warm-up training is performed in a linear increase manner in the first 1000 iterations. A total of 12 epochs are trained. After the 8th epoch and the 11th epoch, the learning rate is reduced by 0.1, respectively.
[0065] The original scale of the images was used for training and inference in the experiments. No other data augmentation methods were used except for random horizontal flipping. The experiments were trained using pre-trained weights on the COCO dataset. When performing online joint object detection and tracking, a new track is initialized if the detection confidence is greater than 0.8. The background is kept for only one frame, and objects are associated only if they are classified into the same category.
[0066] In addition to detection loss and similarity loss, model training also introduces L2 loss as auxiliary training, as shown in formula (8).
[0067]
[0068] If the two samples match positively, c is 1, otherwise it is 0. Auxiliary loss L aux The purpose is to limit the size of the logarithm in similarity learning, rather than to improve performance. Therefore, the loss of the entire network is shown in formula (9):
[0069] L=L det +γ 1 L embed +γ 2 L aux (9)
[0070] Among them, L det is the original Faster R-CNN loss function, which includes RPN loss, category loss, and regression loss. To maintain fairness, the experiment refers to the parameter setting of the original QDTrack training and changes γ 1 and γ 2 The weights of are set to 0.25 and 0.1 respectively.
[0071] In addition to conducting experiments on the MOT16 and MOT17 datasets, this embodiment also uses the CrowdHuman dataset for training.
[0072] The CrowdHuman dataset is a large-scale dataset for pedestrian detection. The dataset focuses on crowds in dense scenes, with complex occlusions, diverse postures, and images from different perspectives. The dataset contains about 15K images and 340K pedestrian instances. This embodiment uses only the training set in the CrowdHuman dataset as a supplementary dataset for training based on the original MOT dataset.
[0073] In order to verify the effectiveness of the improved tracking model in this embodiment, an ablation experiment was conducted on the MOT dataset. Since the test set of MOT needs to be submitted for evaluation on the official website, and is limited by the submission time and number of times, this embodiment divides half of the MOT17 training set into a training set and the other half into a validation set in the ablation experiment to verify the tracking performance of the model. This embodiment mainly conducted three experiments, including ablation experiments of each module, matching effect experiments of different similarity functions, and optimized tracking matching performance experiments under different thresholds.
[0074] In this embodiment, PVT and GA-RPN modules are introduced on the original tracking model QDTrack to verify the detection and tracking performance of the model. As shown in Table 2, after replacing the backbone network of QDTrack with PVT, the MOTA and IDF1 indicators of the model are improved by 3% and 2.8%, the number of identity switching IDs is reduced by 63, and the AP and Recall are improved by 2.1% and 1.3% respectively. It is verified that PVT without CNN structure has stronger feature extraction ability as the backbone network. After replacing the original RPN with GA-RPN, the MOTA and IDF1 indicators of the model are improved by 2.6% and 1.2%, the number of identity switching IDs is reduced by 132 times, and the AP and Recall are improved by 2.3% and 1.9% respectively. It shows that GA-RPN can improve the detection accuracy of the model by optimizing the anchor position through the feature map, thereby improving the overall tracking performance. When the PVT and GA-RPN modules are introduced at the same time, the performance of the model is further improved. Compared with the original model, the MOTA and IDF1 indicators increased by 3.3% and 3.8%, the number of identity switching IDs decreased by 177 times, and the AP and Recall increased by 2.4% and 1.8% respectively. Although the Recall of the model decreased by 0.1%, the overall tracking performance did not decrease, especially the number of identity switching during the tracking process was significantly reduced. This shows that while improving the model's feature extraction capabilities, the introduction of more directional anchor positioning has better distinguishability for the features of the detected target, making it easier to distinguish different identities when tracking and matching similarity calculations, effectively reducing the number of identity switching.
[0075] Table 2
[0076]
[0077] When performing similarity matching, calculating similarity through different similarity functions will lead to different matching results. In order to verify the influence of various similarity functions on the tracking results, this embodiment selects softmax function, cosine function and bidirectional softmax function for comparative experiments. As shown in Table 3, the bidirectional softmax function can effectively reduce the number of identity switching IDs, and verifies that the bidirectional softmax function can better calculate the similarity relationship between the detection object and the candidate target, and then accurately find the corresponding target during matching. When R50-RPN is used as the backbone network of the detection model, different similarity functions have only a 0.2% difference in MOTA and IDF1 indicators. However, when PVT-GA-RPN is used as the backbone network of the detection model, the bidirectional softmax function makes the MOTA and IDF1 indicators reach the highest, which is 0.4% and 0.7% higher than the softmax function, and 0.3% and 1.5% higher than the cosine function. It is verified that the bidirectional softmax function can better utilize the feature information extracted by the PVT-GA-RPN network for similarity matching.
[0078] Table 3
[0079]
[0080] In the previous experiments, it can be observed that the use of different similarity functions for matching has certain advantages in different aspects. For this reason, this embodiment proposes a two-way softmax matching that integrates cosine similarity. On the basis of the two-way softmax function, IoU distance and cosine similarity are introduced. The similarity between the detected object and the candidate target is preferentially calculated within a certain IoU distance, and then the remaining objects are matched through two-way softmax. As shown in Table 4, this embodiment selects different IoU distance thresholds to verify the effect of tracking matching. It can be seen that the matching algorithm proposed in this embodiment has achieved further improvement on both the original model and the improved model. When β iou When the value is 0.8, the best tracking effect is achieved in all indicators. Combined with the experimental results in Table 3, the original tracking model reduces the number of identity switching IDs by 74 times after adopting the new matching method, and the MOTA and IDF1 indicators are improved by 0.7% and 0.5%. After the improved tracking model adopts the new matching method, the number of identity switching IDs is reduced by 22 times, and the MOTA and IDF1 indicators are improved by 0.1% and 0.3%.
[0081] Table 4
[0082]
[0083] This example first compares the MOTA index of the algorithm before and after the improvement on the MOT16 and MOT17 data sets. Figure 4 As shown in the figure, the MOTA index of each video sequence in the MOT16 test set is shown. It can be seen that the tracking performance of the improved algorithm has obvious performance improvements in other video sequences except that it has not changed significantly in the MOT16-03 video sequence.
[0084] Figure 5 The MOT17 test set results in show similar results. The tracking performance of the improved algorithm on the MOT17-01, MOT17-06, MOT17-07, and MOT17-08 sequences is significantly improved. The tracking performance on other video sequences remains unchanged. This verifies that the improved algorithm in this embodiment has a certain generalization ability and is better than the original QDTrack model.
[0085] The tracking model proposed in this embodiment is compared with the mainstream multi-target tracking model on the MOT16 and MOT17 benchmarks. This embodiment selects two types of tracking methods for comparison, including detection-based tracking models: DeepSort, TAP, POI, TubeTK, CTracker, and joint detection-based tracking models: JDE, Tracktor, CenterTrack, FairMOT, GSDT, Swin_JDE, QDTrack. As shown in Tables 5 and 6, * indicates a detection-based tracking model, and the indicators of some methods are empty, indicating that the specific values cannot be found in the relevant papers or MOT official website.
[0086] In the MOT16 data, the improved algorithm of this embodiment is 3.4 points higher than the original QDTrack in the MOTA index. Although there is a decrease in the IDF1 and HOTA indicators, the MT index is improved by 25, reflecting that the number of continuous tracking trajectories of the model has increased, and the corresponding ML index has decreased by 53. The IDs index of the target identity switching during the tracking process has decreased by 70, indicating that the continuous tracking ability of the improved model has been improved to a certain extent. In addition, the FP and FN indicators of the improved model have decreased by 19.3% and 9.6% respectively, indicating that the model has greatly reduced the number of false detections and has better detection accuracy. The Tracktor algorithm has certain similarities with the algorithm proposed in this embodiment in terms of results, and both use Faster-RCNN as the detector. However, the algorithm of this embodiment uses quasi-dense similarity learning to learn instance similarity from dense image pairs, and has stronger discrimination ability than Tracktor, which learns the similarity of a single image pair. Therefore, the algorithm proposed in this embodiment has a significant performance improvement compared to Tracktor. The TAP algorithm has great advantages in IDF1 and IDs indicators, but its comprehensive performance indicator MOTA is 8.4 points lower than the algorithm proposed in this embodiment. These differences are reflected in the high false detection and missed detection of TAP. This also shows that the performance of the detection model has a great impact on the performance of the tracking model.
[0087] Table 5
[0088]
[0089] Table 6
[0090]
[0091] In the MOT17 data, the improved algorithm of this embodiment has no change in the IDF1 index compared with the original QDTrack, but has certain improvements in other performance indicators. Among them, the MOTA index increased by 4.3 points, the HOTA index increased by 0.8 points, and the IDs index decreased by 222. The MT index increased by 141, reaching the best, and the corresponding ML index decreased by 240, further verifying that the improved model has a stronger continuous tracking ability for the target. The number of false detections and missed detections decreased by 11.5% and 14.4% respectively, further verifying that the improved model has better detection performance. Similar to the MOT16 data set, Tracktor has better results in the number of false detections and IDs, but its MOTA index is still the lowest. Swin_JDE adopts the idea of the JDE algorithm, which uses the Transformer with stronger detection capabilities as a detector, so the performance is greatly improved compared with JDE. The algorithm proposed in this embodiment also uses Transformer as a feature extraction network to improve the detection performance of the model. In addition, this embodiment proposes to integrate cosine similarity bidirectional softmax matching to improve the data association ability of the model, which is better than Swin_JDE in MT and ML indicators, so the overall performance of the model is in the lead.
[0092] In order to further verify the effectiveness of the method proposed in this embodiment in improving the performance of the tracking model, this section shows the tracking results of some sequences in the MOT17 test set. In the 614th frame image of this embodiment, the pedestrian with ID 71 marked by the red arrow is selected as the research target. In the image of the 633rd frame, it can be seen that the pedestrian is blocked by the street light, and QDTrack cannot continue to track the target in the occlusion. On the contrary, the improved algorithm can still continue to track the target when it is blocked by obstacles. Therefore, after the target completely appears on the screen in the 650th frame, the improved algorithm still maintains the original ID. However, QDTrack switches the ID at this time, and the ID becomes 177, and the target is judged as a new target.
[0093] At the same time, a pedestrian with ID 59 was selected as the research target. It can be seen that the target is about to intersect with the pedestrian in front. In the image of the 145th frame, the target is completely blocked by the pedestrian, and the algorithm cannot continuously track the target. However, the target reappears in the image of the 150th frame, and the improved algorithm correctly determines that the target ID is 59 and continuously tracks the target. On the contrary, QDTrack determines the target as a newly appeared target. Through the above analysis, it is verified that the improved algorithm has a strong continuous tracking capability. After the target blocked by obstacles or pedestrians reappears, the number of identity ID switching can be greatly reduced.
[0094] Although the improved model has achieved good results on the MOT dataset, quasi-dense similarity learning has great potential. For this reason, this section chooses to conduct further analysis on the more challenging DanceTrack dataset. The DanceTrack dataset contains more than 100K image frames, which is almost 10 times the MOT17 dataset. This data mainly collects the recognition of collective dance, which pays more attention to the motion discrimination ability of the tracking model. Therefore, the DanceTrack dataset brings two challenges: first, the appearance of the characters in the dataset is extremely similar, and it is difficult for a simple ReID model to distinguish them; second, the body posture movement amplitude of the characters in the dataset varies greatly, and people frequently intersect and occlude each other.
[0095] The DanceTrack dataset uses HOTA as the main evaluation indicator because the MOTA indicator focuses more on the detection quality of the model. The DanceTrack dataset has fewer targets than the MOT dataset and focuses more on the association ability of the model. The AssA and IDF1 indicators are used to measure the association performance, and the DetA and MOTA indicators are used to measure the detection quality. The experiment refers to the configuration of the MOT17 dataset training to train the DanceTrack dataset. The test set results were submitted on the official website of DanceTrack for evaluation and compared with the mainstream tracking models. The test results are shown in Table 7.
[0096] Table 7
[0097]
[0098] It can be found that the improved algorithm has improved all indicators compared with QDTrack, and the HOTA indicator has increased by 3 points. ByteTrack is leading in MOTA and IDF1 indicators because it makes the detection model and appearance discrimination model independent. Although ByteTrack has good detection capabilities based on YOLOX as a detector, its HOTA and AssA indicators do not achieve ideal results when facing targets with similar appearances through the target features extracted by the ReID model. The ReID algorithm alone is not the optimal solution for the multi-target tracking model. Compared with sparse feature learning, quasi-dense similarity learning can better learn the differences between the target object and the surrounding environment.
[0099] The experiment selected the sequence of the DanceTrack test set and compared it with the tracking effect of the QDTrack algorithm. In the 178th frame of the image, the ID of one target in QDTrack changed, while the ID of the improved algorithm did not change. Then, in the 300th frame of the image, the IDs of four targets in QDTrack changed, while the ID of the improved algorithm remained unchanged. This shows that the improved algorithm has better target discrimination ability in complex motion scenes.
[0100] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A multi-target tracking method based on joint detection, characterized in that: The following steps are involved: Use the PVT model as the backbone network to extract image features; Positioning and identifying target features in the image based on the GA-RPN structure to obtain a candidate target region; The bidirectional softmax function is optimized based on cosine similarity and IoU distance, and the targets in the candidate target area are matched by the optimized bidirectional softmax function to generate a multi-target tracking result.
2. The multi-target tracking method based on joint detection according to claim 1, characterized in that: The process of using the PVT model as the backbone network to extract image features includes: The PVT model controls the scale of the feature map through a progressive shrinkage strategy; For the i-th stage, the feature map F from the previous stage input i-1 (H i-1 ×W i-1 ×C i-1 ) is divided into patches, and then flatten each patch and map it to C i dimensional feature embedding, and then after linear mapping, the feature size can be regarded as 3. The multi-target tracking method based on joint detection according to claim 1, characterized in that: The PVT model adopts a spatial scaling attention mechanism; The expression of the spatial scaling attention mechanism is: SRA(Q,K,V)=Concat(head0,head1,...,head Ni )W O ; Among them, Concat() is the concatenation operation in Transformer. is the weight parameter, N i is the number of attention heads in the i-th stage, so the dimension of each head is LSR() performs a linear dimensionality reduction operation on the spatial dimension of the input sequence.
4. The multi-target tracking method based on joint detection according to claim 1, characterized in that: The features of the target in the image are located and identified based on the GA-RPN structure, and the process of obtaining the candidate target area also includes: constructing a quasi-dense similarity learning model based on the joint learning model of sparse ID loss, sampling the surrounding area of the candidate target area based on the quasi-dense similarity learning model, and obtaining positive and negative samples of the information area.
5. The multi-target tracking method based on joint detection according to claim 1, characterized in that: The GA-RPN structure predicts the expression of the optimal shape for each position: w=σ·s·e dw ; h=σ·s·e dh ; where w and h are the optimal shapes at each location, i.e., the shapes that result in the highest coverage with the nearest true label bounding box, s is the stride, and σ is the scaling factor.
6. The multi-target tracking method based on joint detection according to claim 1, characterized in that: The GA-RPN structure is also used to add a 1×1 convolution to the output of the shape prediction branch to predict the offset domain, and then map the offset into a 3×3 convolution to obtain an adaptive feature map.
7. The multi-target tracking method based on joint detection according to claim 1, characterized in that: The target in the candidate target area is matched by the optimized bidirectional softmax function, and the process of generating multi-target tracking results includes: constructing an IoU threshold β iou , when the IoU threshold of the candidate target is greater than β iou When , cosine similarity will be given priority for matching, and the remaining value is less than β iou The candidate target with detection confidence higher than β obj , then consider matching the detected target with the existing trajectory, if the matching score is greater than β m , then the match is determined, and the unmatched candidate target is determined if the detection confidence is higher than β new , a new trajectory is created.
8. The multi-target tracking method based on joint detection according to claim 1, characterized in that: The expression of the loss function of the overall network is: L=L det +γ1L embed +γ2L aux ; Among them, L det is the original FasterR-CNN loss function, including RPN loss, category loss and regression loss, L aux Auxiliary loss.
Citation Information
Patent Citations
Improved high-precision end-to-end vehicle multi-target tracking method
CN113963168A
Transform-based online update target tracking method and system
CN114998601A
Multi-target tracking and segmentation system and method
CN115880330A
Fish multi-target tracking method, system and equipment for industrial aquaculture
CN116721132A
Method, device and system for detecting target object in video data
CN116994181A
