Appearance feature guided multi-target tracking method and system

Abstract

The invention discloses a multi-target tracking method and system guided by appearance features, and aims to solve the problems that when appearance information is utilized in an existing method, candidate box generation cannot be optimized by utilizing distribution information of the appearance features, and limitation exists in target authenticity evaluation only depending on detection confidence. According to the method, appearance embedding features are fully mined by using an instance aggregation module, global target saliency distribution under different scenes is generated, and thus the target sensing ability of an appearance model is expanded. A spatial enhancement module is utilized to introduce target saliency distribution into a tracking framework, a high-quality candidate frame is generated for a tracker, and an association process is optimized. By integrating the two modules into a single-stage multi-target tracking framework, the sensing ability of the framework to the target position and state can be improved, a more reliable candidate frame and a more reliable association result are generated, and the performance and robustness of multi-target tracking are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and deep learning, and relates to a multi-target tracking method and system based on appearance feature perception, which is suitable for unmanned aerial vehicle target tracking application scenarios. BACKGROUND

[0002] Multi-target tracking is an important task in the field of computer vision, involving the detection and tracking of multiple targets in video sequences. Today, it plays a key role in scenarios captured by unmanned aerial vehicles, including aerial monitoring, target search, and post-disaster assessment.

[0003] With the rapid development of object detection, detection-based tracking methods have become the most popular multi-object tracking paradigm, and are divided into two-stage and one-stage multi-object tracking frameworks according to the way of appearance feature extraction. For two-stage multi-object tracking, the detector is first used to obtain the candidate boxes, then the appearance model is used to generate the feature embedding of the target in the box, and finally the identity association is performed using these feature embeddings to connect the candidate boxes with existing trajectories. Deepsort first introduced a tracker using an appearance model and designed a cascade matching strategy to introduce appearance information into the matching process (Wojke N, Bewley A, Paulus D. Simple online and realtime tracking with a deep association metric[C] / / 2017 IEEE international conference on image processing (ICIP). IEEE, 2017: 3645-3649.). In addition, Strongsort refines the feature update strategy based on Deepsort, improving the robustness of the tracker (Du Y, Zhao Z, Song Y, et al. Strongsort: Made deepsort great again[J]. IEEE Transactions on Multimedia, 2023, 25: 8725-8737.). Although two-stage methods are effective, they have high computational overhead because the appearance model needs to perform forward inference on each candidate box separately. To improve computational efficiency, one-stage multi-object tracking integrates detection and appearance embedding extraction into a unified framework, attracting a lot of research interest. By redesigning the prediction head of the detector, one-stage trackers can generate detection results and appearance features simultaneously. Thanks to this advantage, JDE introduces an appearance model into the detector, achieving real-time multi-object tracking (Wang Z, Zheng L, Liu Y, et al. Towards real-time multi-object tracking[C] / / European conference on computer vision. Cham: Springer International Publishing, 2020: 107-122.).Similarly, FairMOT balances the proportion of detection and re-identification tasks, improving the accuracy of re-identification (Zhang Y, Wang C, Wang X, et al. Fairmot: On the fairness of detection and re-identification in multiple object tracking [J]. International journal of computer vision, 2021, 129: 3069-3087.). Further, CSMOT alleviates the task conflict problem based on the JDE framework, making target detection and embedding correlation benefit each other (Liang C, Zhang Z, Zhou X, et al. Rethinking the competition between detection and reid in multiobject tracking [J]. IEEE Transactions on Image Processing, 2022, 31: 3182-3196.).

[0004] Although the current tracker with a re-identification model has achieved good results, there are still the following two problems: Due to the different levels of representation, the multi-target tracking framework cannot utilize the distribution information of the target contained in the re-identification feature, which limits the detection performance and produces more false detections in complex scenes, affecting the generation of candidate frames.

[0005] In the detection-to-tracking framework, after obtaining the detection frame, only high-confidence detections are selected as real targets to form tracking trajectories, and low-confidence frames are treated as false positives or processed with special strategies. However, when the target is blurred or occluded, the detection score cannot correctly reflect the authenticity of the target, resulting in reduced association effect. SUMMARY

[0006] The present application aims to overcome the shortcomings of the prior art and proposes a global perception appearance-guided multi-target tracking method, named GAMOT. The instance aggregation module is used to fully exploit the appearance embedding features to generate global target saliency distribution in different scenes, thereby expanding the target perception ability of the appearance model. The hierarchical spatial enhancement module is used to introduce the target saliency distribution into the tracking framework to generate high-quality candidate frames for the tracker and optimize the association process. The above modules are efficiently integrated into a single-stage multi-target tracking framework, and finally achieve excellent tracking performance.

[0007] The technical scheme adopted by the present application is: we use YOLOX as an object detector to generate candidate bounding boxes. In order to generate appearance features suitable for subsequent processing, we use contrastive learning to train the appearance model. It has been proved that this method can effectively generate global pixel-level appearance information while retaining instance-level features. On this basis, we use pixel-level appearance embedding to constrain the appearance model and generate stable global representation. Specifically, we use the proposed instance aggregation module to decode the appearance features generated by the re-identification model. The decoder is trained by the pseudo-label created when building the appearance model, so that it can extract stable features from noisy appearance information. This method can effectively extract semantic information contained in the re-identification features and generate target saliency distribution. In addition, we also design a spatial enhancement module to integrate the target saliency distribution into the multi-target tracking process in a hierarchical manner to optimize the tracking effect. At the global level, we provide fine-grained target perception information for the detection module to help the detector better focus on the target area and improve the detection accuracy. At the instance level, we introduce the target saliency distribution into the association process to evaluate the quality of the candidate box, and then perform hierarchical matching on the candidate box. This method avoids the interference of low-quality boxes on high-quality boxes, thereby improving the association efficiency. Finally, we integrate these improvements into a single-stage multi-target tracking framework and design a new tracker suitable for complex scenes. The combination of instance aggregation and spatial enhancement ensures robust and effective feature extraction, which helps to improve the detection and tracking performance. Finally, the tracker is trained on the Visdrone dataset, and multi-target tracking is performed based on the trained tracker. The method includes the following steps: Step 1, frame the video taken by the unmanned aerial vehicle to obtain a plurality of continuous frame images, and the obtained continuous frame images are used as target images for subsequent processing; Step 2, construct an improved target detector, the improved detector includes a backbone network, an instance aggregation module, a spatial enhancement module, and a detection head, and outputs a set of target detection boxes; Step 3, obtain a target tracking box, use all target detection boxes and target tracking boxes as input, calculate motion matching degree and feature matching degree by using an optimized hierarchical matching strategy to determine the most matched target tracking box in the historical trajectory for each target detection box in the current frame, after hierarchical matching, the tracker determines the tracking result, thereby realizing multi-target tracking; Step 4, train the improved target detector, calculate the gap between the saliency distribution output by the instance aggregation module and the pseudo-label, which is used to provide high-quality candidate boxes for the tracker.

[0008] Further, in step 2, the target image is processed through the backbone network to obtain a plurality of feature maps of different resolutions for the detection task, and then is divided into two branches. One branch uses a deep learning re-identification model to extract a high-dimensional feature vector capable of representing the unique appearance of the target, i.e., a re-identification feature, from the feature map, and then inputs the obtained re-identification feature vector into an instance aggregation module; The other branch uses a spatial enhancement module to enhance the multi-scale detection capability of the model by adjusting the multi-scale feature map. After the spatial enhancement module, the feature maps of different levels use the anchor-free mechanism of YOLOX to assign different levels of feature maps to predict different scales of targets, and through independent convolution branches, target classification prediction, bounding box regression prediction and confidence prediction are performed respectively, and the prediction results are integrated to obtain a set of target detection boxes.

[0009] Further, the processing process of the instance aggregation module is as follows: First, based on the existing labels of the detection task, the feature vectors at each position of the current re-identification feature are traversed, and the target center point is used to represent the instance. The cosine similarity is used to measure the correlation between features, and the pixel-level distinction is realized on the feature map to generate the spatial distribution prediction of the target in the current frame. The above process is repeated for each target, and finally the global saliency distribution of all targets in the frame is obtained:

[0010] Among them, represents the target index, represents the center feature of the th target, represents the overall appearance feature, i.e., the re-identification feature; then, the saliency distribution information is subjected to a hard mask operation. The saliency of the pixel position in the target region of the global saliency distribution map is set to 0 if it is less than a set threshold, and is set to 1 if it is greater than the set threshold. The saliency of the position not in the target region is set to 0. The resulting hard mask result is used as a pseudo-label to supervise a decoder to extract the target saliency distribution map from the appearance feature as the global spatial representation of the target.

[0011] Further, the processing process of the spatial enhancement module is as follows: For the feature map of each scale corresponding to the resolution, first, a 3x3 convolution operation is performed to align the feature state of the current scale to generate a scale-adaptive perception map. For features of different scales, the feature map of different resolutions is adapted through downsampling operation, and then 3x3 convolution is further used to extract key features to effectively guide multi-scale target detection. This process is represented as:

[0012] Among them, a saliency distribution representing an example aggregation module output, is a down-sampled perception map, represents a down-sampling operation, and scale_factor represents a down-sampling factor, is a target perception map after convolution processing, and represent a weight and a bias, respectively, represents a batch normalization layer, represents an activation function; The target perception map after convolution processing is multiplied by the original feature map, and the original feature information is fused to form a final perception feature by using a skip connection strategy, and then the perception-enhanced feature map is obtained by processing through a channel attention mechanism The process is represented as:

[0013]

[0014] where F represents an original feature map, represents a target perception map, and CA represents a channel attention mechanism; the processing process of the channel attention mechanism is:

[0015]

[0016]

[0017] where represents a fully connected neural network, represents an activation function, represents an average value pooling, represents a maximum value pooling.

[0018] Further, in step 3, before matching, the coordinates of the detection boxes in the previous frames are corrected by using a camera motion compensation technique to offset the influence of camera movement, and a Kalman filter is combined to predict the state of the target in the current frame based on historical detection information, so as to obtain the target tracking box.

[0019] Further, the specific implementation of step 3 includes the following sub-steps: Step 3.1, calculate the re-identification perception score, the calculation method is as follows:

[0020] wherein, represents a re-identification perception score, is obtained by calculating the maximum value of saliency in each target detection box region, wherein i and j represent the horizontal and vertical coordinates of the pixels in the target box, respectively, representative target detection confidence, hyper-parameters; Step 3.2, hierarchical matching is performed with the re-identification perception score of the target as a reference, target detection boxes with a re-identification perception score greater than a threshold are divided into the first stage of matching, and the rest are divided into the second stage of matching; The first stage of matching designs two measurement methods, motion matching degree and feature matching degree, to perform one-to-one matching between the current effective trajectory and the currently detected target; The second stage of matching is performed on unmatched trajectories and target detection boxes with a re-identification perception score less than a threshold; the matching is realized by only considering the intersection over union (IoU) between the tracking box obtained by camera motion compensation and Kalman filtering prediction and the current candidate detection box; an IoU threshold is set, and if the IoU between the two is greater than the threshold, it is considered that they are successfully matched; the Hungarian algorithm is applied again to perform accurate pairing, and for the successfully matched trajectory, the template is updated to adapt to the changes in appearance; Among the candidate boxes that are not successfully matched in the two stages, the candidate boxes with a re-identification perception score greater than a certain set value are initialized as new trajectories and embedded with new identity templates, and conversely, the unmatched trajectories are marked as inactive; if a trajectory cannot be matched for a certain number of consecutive frames, the trajectory will be terminated; however, if an inactive trajectory finds a match before being terminated, it will be reactivated.

[0021] Further, the motion matching degree uses the classic intersection over union to measure the degree of spatial overlap between the detection box and the tracking box; the feature matching degree evaluates the appearance similarity between the two re-identification features by calculating the cosine similarity between them; to integrate these two complementary information, their linear weighted sum is used as the final correlation measure, and the final correlation measure similarity is input into the Hungarian algorithm to determine the most matched target tracking box in the historical trajectory for each target detection box in the current frame, thereby realizing stable and continuous tracking of the target.

[0022] Further, during training, the detection loss , the re-identification loss and the spatial perception loss are combined in a weighted sum manner to construct a joint loss function, and the joint loss function is defined as:

[0023] where , and are hyper-parameters that control the relative contribution of each task; The detection loss is defined as the weighted sum of the classification loss, the bounding box regression loss, and the target existence loss, and each part is controlled by a hyper-parameter. For the re-identification task, the model is trained using contrastive learning to distinguish similar objects based on feature similarity, and the loss function is formulated as: wherein represents the contrastive loss, is the feature embedding of the i-th sample, which is obtained by extracting the feature embedding at the center position coordinate of the target frame of the i-th sample on the re-identification feature; the loss minimizes the distance between the positive samples , while maximizing the distance between the negative samples , and the temperature parameter is a constant for controlling the influence of sample distance on gradient; In terms of spatial perception map training, the mean square error loss is adopted, and its formula is: wherein represents the spatial perception loss, N represents the total number of samples, and are the predicted value and pseudo label of the i-th sample, respectively.

[0024] Further, the detection loss is represented as: wherein α, β and γ are hyperparameters for balancing the contributions of each loss term.

[0025] For the bounding box regression, the intersection over union (IoU) loss is adopted, which is defined as:

[0026] wherein represents the bounding box regression loss, is the predicted bounding box, is the real bounding box; In terms of distinguishing foreground and background, the binary cross-entropy loss is adopted. In terms of classification, the binary cross-entropy loss is also adopted as the classification loss .

[0027] The application also provides a global perception appearance-guided multi-target tracking system, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the global perception appearance-guided multi-target tracking method according to the above technical solution when executing the program.

[0028] Compared with the prior art, the advantages and beneficial effects of the present application: the present application proposes a global perception appearance guided multi-object tracking method. The correlation between appearance embeddings is used to guide the generation of global target saliency distribution of appearance model. In addition, a spatial enhancement module is proposed to introduce saliency distribution into the detection process to improve the detection effect and use the re-identification perception score to comprehensively evaluate the authenticity of the target. These modules are introduced into the single-stage multi-object tracking framework to build a high-performance tracker. Experimental results show that the introduction of these modules significantly improves the tracking effect of the tracker. BRIEF DESCRIPTION OF DRAWINGS Figure 1 is an appearance feature guided multi-object tracking framework structure diagram.

[0029] Figure 2 is a structure diagram of a spatial enhancement module.

[0030] Figure 3 is a comparison of tracking results of different tracking methods.

[0031] Figure 4 is the tracking result under the scene containing numerous small targets. DETAILED DESCRIPTION

[0032] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the examples described herein are only used to explain the present application and do not limit the present application.

[0033] The present application mainly aims at the application requirement of high-precision unmanned aerial vehicle multi-object tracking, proposes a global perception appearance guided multi-object tracking method, named GAMOT. The instance aggregation module is used to fully exploit the appearance embedding features to generate global target saliency distribution in different scenes, thereby expanding the target perception ability of the appearance model. The hierarchical spatial enhancement module is used to introduce the target saliency distribution into the tracking framework to generate high-quality candidate boxes for the tracker and optimize the association process. The above modules are efficiently integrated into the single-stage multi-object tracking framework, and finally the excellent tracking performance is realized. The tracking results before and after introducing the above modules are compared as shown in Figure 1 The method specifically comprises the following steps: Step 1, frame the video shot by the unmanned aerial vehicle to obtain a plurality of continuous frame images, and take the obtained continuous frame images as target images for processing.

[0034] Step 2, build the improved target detector. We use YOLOX as the object detector to generate candidate bounding boxes. The improved detector includes an instance aggregation module and a spatial enhancement module, which can embed the saliency distribution map into the detector, helping the detector better identify targets. In this step, the target image is input, and the set of target detection boxes is output.

[0035] Step 3, use the optimized hierarchical matching strategy for data association. The proposed matching strategy can more accurately evaluate target quality, thus better completing data association. Before matching, the camera motion compensation technique is used to correct the coordinates of the detection boxes in the previous frames to offset the influence of camera movement, and the Kalman filter is combined to predict the state of the target in the current frame based on these historical detection information, to obtain the target tracking box. All target detection boxes and target tracking boxes are input, and the proposed matching strategy is used to calculate the motion matching degree and the feature matching degree. Based on these matching degrees, a cost matrix is generated and output. Each element C(i,j) in the cost matrix represents the cost of matching the ith target detection box with the jth target tracking box. The cost matrix is input into the Hungarian algorithm to determine which target tracking box in the historical trajectory should be optimally matched with each detection box in the current frame, thus realizing stable and continuous tracking of the target.

[0036] Step 4, network training. The instance aggregation module and the hierarchical spatial enhancement module are integrated into the detector for joint training. The MSE loss function is used to calculate the gap between the instance aggregation result output by the instance aggregation module, i.e. the saliency distribution and the pseudo label, to provide high-quality candidate boxes for the tracker.

[0037] Further, the specific implementation of step 2 includes the following sub-steps, Step 2.1, the extracted target image is processed through the backbone network to obtain multiple feature maps of different resolutions for the detection task.

[0038] Step 2.2, as Figure 1As shown, after obtaining the feature map of the detection target, we use a deep learning re-identification model (based on convolutional neural network CNN) to extract a high-dimensional feature vector from the feature map that can represent the unique appearance of the target. The model aims to learn a feature space so that the features of the same identity are as close as possible, and the features of different identities are as far apart as possible. The obtained re-identification features are input into the instance aggregation module. In the instance aggregation module, first, based on the labels of the existing detection task, the feature vectors at each position of the current re-identification features are traversed, and the target center point is used to represent the instance. The cosine similarity is used to measure the correlation between features, and pixel-level differentiation is achieved on the feature map to generate the spatial distribution prediction of the target in the current frame. This process is repeated for each target, and finally the global saliency distribution of all targets in the frame is obtained. Due to the instability of appearance features, multiple responses may be generated for the same target. If the response matrix is directly superimposed and summed, it will cause the phenomenon of false overlap of response values - both the recognition of some targets is weakened, and the noise interference is amplified. To solve this problem, this method discards the simple aggregation strategy of different target saliency distributions, and instead uses cross-channel max-pooling operation. This operation effectively suppresses noise interference by retaining the most relevant response value, while ensuring the uniformity of the response distribution. The complete algorithm flow can be expressed as:

[0039] wherein, denotes the target index, denotes the center feature of the th target, denotes the overall appearance feature, i.e. the re-identification feature. Through this method, we convert the appearance information from instance-level representation to global-level representation, and obtain the stable spatial distribution of the target. Subsequently, we perform a hard mask operation on these saliency distribution information, setting the saliency of the pixel positions in the target area of the global saliency distribution map that are less than the set threshold to 0, and the saliency of the positions that are greater than the set threshold to 1, and the saliency of the positions that are not in the target area to 0. In the experiment, the threshold is set to 0.35. The resulting hard mask result is used as a pseudo-label to supervise a decoder to extract the target saliency distribution map from the appearance feature as the global spatial representation of the target for optimization of subsequent steps.

[0040] Step 2.3, as shown in Figure 2As shown, a spatial enhancement module is proposed. The module enhances the multi-scale detection capability of the model by adjusting the multi-scale feature maps. At each scale, the target perception map is first aligned with the feature state of the current scale through a 3x3 convolution operation to generate a scale-adaptive perception map. For features of different scales, the target perception map is adapted to different resolution feature maps through downsampling operation. Specifically, the perception map is downsampled to match the low-resolution feature map, and then a 3x3 convolution is further used to extract key features, effectively guiding multi-scale object detection. As shown in (a) of FIG. 1, Figure 2 The process can be represented as:

[0041] wherein represents the saliency distribution output by the instance aggregation module, is the target perception map after downsampling, represents the downsampling operation, and scale_factor represents the downsampling factor, is the perception map after convolution processing, and represent the weight and bias, respectively, represents a batch normalization layer, represents an activation function. The process ensures the effective fusion of the target perception map on different scale feature maps. The perception map can promote the extraction of key features from the feature layer while maximizing the suppression of background noise interference. Subsequently, the feature map generated by the detector is added after being aligned with the perception map shape after a 3x3 convolution operation, and then a 3x3 convolution operation is performed for feature fusion. In order to preserve key original information during feature optimization, we use a skip connection strategy to fuse the original feature information with the optimized feature map to generate a perception-enhanced feature map.

[0042] The target perception map after convolution processing is multiplied by the original feature map, and a skip connection strategy is used to fuse the original feature information to form the final perception feature, which is then processed by a channel attention mechanism to obtain a perception-enhanced feature map The process is represented as:

[0043]

[0044] wherein F represents the original feature map, represents the target perception map, and CA represents the channel attention mechanism; as shown in (b) of FIG. 1, Figure 2 The processing process of the channel attention mechanism is as follows:

[0045]

[0046]

[0047] wherein represents a fully connected neural network, represents an activation function. represents average value pooling, represents maximum value pooling.

[0048] The main advantage of this design is that it can accurately extract multi-scale target region details using global fine-grained perception information, significantly reducing background noise interference. The introduction of the target perception map enables the model to adaptively focus on target regions in a multi-scale environment, avoiding background interference. Finally, the channel attention mechanism enables the model to learn the weights of different channels, further enhancing the feature expression ability of the target region. The generated features are concatenated with the original detection branch generated features in the channel dimension, thereby completing the global spatial enhancement.

[0049] Step 2.4, for different levels of feature maps after the spatial enhancement module, use the anchor-free mechanism of YOLOX to assign different levels of feature maps to predict targets of different scales. Through independent convolution branches, target classification prediction, bounding box regression prediction, and confidence prediction are performed respectively, and the results of these predictions are combined to obtain a set of target detection boxes.

[0050] Further, the specific implementation of step 3 includes the following sub-steps, Step 3.1, use the saliency distribution map output by the instance aggregation module to perform instance-level optimization on the target detection boxes. For this purpose, we design a "re-identification perception score" that evaluates target quality based on perception information. A high score indicates reliable target quality, while a low score indicates a lack of highly similar objects, which may mean false detection or a newly appearing target. These low-score targets are not suitable for high-quality matching and should be divided into the next lower level to avoid affecting the matching performance of high-quality boxes in the first level. The calculation method of the re-identification perception score is as follows:

[0051] wherein, represents the re-identification perception score, is obtained by calculating the maximum value of the saliency in each target box region, represents the target detection confidence, is a hyperparameter, set to 0.6 in the experiment.

[0052] Step 3.2, hierarchical matching is performed with the target re-identification perception score as the reference. Target bounding boxes with a re-identification perception score greater than 0.7 are divided into the first stage of matching, and the rest are divided into the second stage of matching. Before matching, camera motion compensation techniques are used to correct the coordinates of the detection boxes in the previous frames to offset the effects of camera movement, and a Kalman filter is used to predict the state (such as position and size) of the target in the current frame based on historical detection information, to obtain the target tracking box. (Liu K, Jin S, Fu Z, et al. Uncertainty-aware unsupervised multi-object tracking [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 9996-10005.). The first stage of matching designs two types of metrics: motion matching degree and feature matching degree. The motion matching degree uses the classic intersection over union (IoU) to measure the spatial overlap between the detection box and the tracking box; the feature matching degree evaluates the appearance similarity by calculating the cosine similarity between the two appearance features. To integrate these two complementary information, we use their linear weighted sum as the final association metric. This metric is directly used in the subsequent data association stage. The final association similarity is input into the Hungarian algorithm to determine which target tracking box in the historical trajectory should be optimally matched with the current detection box, thereby achieving stable and continuous tracking of the target. Unmatched trajectories and target detection boxes with a re-identification perception score less than 0.7 will enter the second stage of matching. In this stage, considering that these targets have poor visual effects, their appearance features may not be sufficient to support high-confidence re-identification matching, we eliminate the dependence on ReID scores. The specific implementation is as follows: only the intersection over union (IoU) between the tracking box obtained through camera motion compensation and Kalman filtering prediction and the current candidate detection box is used for matching. We set an IoU threshold, if the IoU between the two is greater than this threshold, then they are considered to be matched successfully. Finally, a similarity result is obtained. This approach prioritizes the continuity of the target in space. This adjustment aims to improve the likelihood of finding potential matches. Here, the Hungarian algorithm is again applied for precise pairing. For successfully matched trajectories, the template will be updated to adapt to changes in appearance. Among the candidate boxes that are not successfully matched in both stages, candidate boxes with a re-identification score greater than 0.8 are initialized as new trajectories and embedded with new identity templates. Conversely, unmatched trajectories are marked as "inactive". If a trajectory fails to find a match for 15 consecutive frames, the trajectory will be terminated. However, if an "inactive" trajectory finds a match before termination, it will be reactivated.After hierarchical matching, the tracker determines the tracking results, thereby realizing multi-target tracking.

[0053] Further, the following loss function is used in step 4 as a guide for network optimization, During the overall model training process, the decoder in the instance aggregation module is initially optimized only by the spatial perception loss. When joint training is performed, we combine the detection loss, the re-identification loss and the spatial perception loss by weighted summation to construct a joint loss function. The joint loss function is defined as:

[0054] where , and are hyperparameters that control the relative contribution of each task. By jointly optimizing the target detection, re-identification and spatial perception tasks, the model can improve the overall performance through multi-task learning To jointly optimize the target detection and re-identification tasks, we minimize the weighted sum of the detection loss (composed of classification loss and bounding box regression loss) and the re-identification loss. Specifically, the real labels of the input image are represented as a set Bi , Bi ( x ( i ), y ( i ), w ( i ), h ( i ), c ( i ))}. Here, x ( i , y ( i )) represent the center coordinates of the bounding box, w ( i , h ( i )) represent the width and height of the bounding box, c ( i ) represents the identity index of the i th pedestrian. C represents the total number of pedestrians in the training data set, and represents the number of different individuals in the data set. For each position x , y in the detection result image at different resolutions, we convert it into a 5-dimensional vector t =( x’ , y’ , w’ , h’ ,p ),in( x’ , y’ , w’ , h’ ) are the predicted bounding box coordinates, p is the probability that an object exists at that location. In the classification task, we use the binary cross entropy loss with logits (denoted as BCELoss) to improve the accuracy of the model in the object detection task.

[0055] For bounding box regression, we adopt the intersection-over-union (IoU) loss, which is defined as:

[0056] in represents the regression loss, is the predicted bounding box, is the true bounding box. This loss function measures the similarity between the predicted box and the true box by comparing their overlapping areas, thereby reducing the difference between the predicted box and the true box and enhancing the model's positioning ability. The larger the IoU value, the higher the overlap between the predicted box and the true box, and the smaller the regression loss. In terms of distinguishing foreground and background, we use binary cross entropy (BCE) loss We incorporate the target existence probability score into the total detection loss. In terms of classification, we also use binary cross entropy (BCE) loss as the classification loss The total detection loss is defined as the weighted sum of the classification loss, bounding box regression loss, and object existence loss, each of which is controlled by hyperparameters. Expressed as: in α 、 β and γ is a hyperparameter used to balance the contribution of each loss term. In this formula, α The value of is set to 5, and β and γ Both are set to 1 to ensure that the relative importance of classification loss and regression loss is properly reflected in the model optimization process.

[0057] For the re-identification task, we use contrastive learning to train the model to distinguish similar objects based on feature similarity. The loss function is formulated as: in represents the contrast loss, It is i feature embedding of samples (by extracting the re-identification features on the i The feature embedding of the target box center coordinates of the samples is obtained). This loss minimizes the positive sample (the same pedestrian) distance, while maximizing the negative samples Distance between (different pedestrians). Temperature parameter is set to 0.07, which is used to control the influence of sample distance on the gradient, and improve the accuracy of the re-identification task.

[0058] In the training of the spatial-aware map, we adopt the mean square error (MSE) loss, whose formula is: wherein represents the awareness loss, N represents the total number of samples, and are the predicted value and pseudo label respectively. The loss measures the error between the predicted spatial-aware map and the true value, and the model's awareness of the spatial distribution is enhanced by minimizing the loss.

[0059] Based on the above steps, the multi-target tracking result is obtained. In order to compare with other methods, we select two advanced multi-target tracking algorithms to compare with our method on the images taken by the unmanned aerial vehicle, and the results are shown in the attached Figure 3 and Figure 4 .

[0060] In order to quantitatively evaluate the multi-target tracking result, we use MOTA and IDF1 as evaluation indexes of tracking performance. The quantitative comparison results on the public Visdrone test data set are as follows: Table 1 Quantitative comparison results

[0061] The quantitative index results show that the multi-target tracking result obtained by the method proposed in the present application is better than that of the existing method, and the target can be tracked with high quality.

[0062] On the other hand, the embodiment of the present application also provides a global-aware appearance-guided multi-target tracking system, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the global-aware appearance-guided multi-target tracking method according to the above technical solution when executing the program.

[0063] It should be understood that parts not described in detail in the specification are all prior art.

[0064] It should be understood that the above description of the embodiments is more detailed, and therefore should not be considered as a limitation on the scope of patent protection of the present application. Ordinary skilled in the art can make substitutions or modifications without departing from the scope of protection claimed by the present application, which all fall within the scope of protection of the present application. The scope of protection claimed by the present application should be subject to the appended claims.

Claims

1. A global appearance-guided multi-target tracking method, characterized in that: The steps include: Step 1: The video captured by the drone is divided into frames to obtain a number of continuous frame images, and the obtained continuous frame images are used as target images for subsequent processing; Step 2: Build an improved object detector. The improved detector includes a backbone network, an instance aggregation module, a spatial enhancement module, and a detection head, and outputs a set of object detection boxes. Step 3: Get the target tracking frame. Take all target detection frames and target tracking frames as input, and use the optimized hierarchical matching strategy to calculate the motion matching degree and feature matching degree to determine the target detection frame of the current frame and the target tracking frame that best matches the historical trajectory. After hierarchical matching, the tracker determines the tracking result, thereby realizing multi-target tracking. In step 4, the improved target detector is trained to calculate the gap between the saliency distribution output by the instance aggregation module and the pseudo label to provide high-quality candidate boxes to the tracker.

2. The method for multi-target tracking guided by global appearance perception according to claim 1, characterized in that: In step 2, the target image is processed by the backbone network to obtain multiple feature maps of different resolutions for the detection task. This is then divided into two branches. One branch uses a deep learning re-identification model to extract high-dimensional feature vectors that can represent the unique appearance of the target from the feature map, namely re-identification features. The obtained re-identification feature vectors are then input into the instance aggregation module. The other branch uses the spatial enhancement module to enhance the multi-scale detection capability of the model by adjusting the multi-scale feature maps; After the spatial enhancement module, feature maps of different levels are assigned with predictions of targets of different scales using YOLOX's anchor-free mechanism. Target classification prediction, bounding box regression prediction, and confidence prediction are performed separately through independent convolution branches. These prediction results are combined to obtain a set of target detection frames.

3. The global appearance-guided multi-target tracking method according to claim 1, wherein: The processing process of the instance aggregation module is as follows: First, based on the existing detection task labels, we traverse the feature vectors of each position of the current re-identified feature and characterize the instance with the target center point. We use cosine similarity to measure the correlation between features and achieve pixel-level differentiation on the feature map, thereby generating a prediction of the target's spatial distribution in the current frame. We repeat the above process for each target and finally obtain the global saliency distribution of all targets in the frame: in, represents the target index, Indicates the The central feature of a target, Represents the overall appearance features, namely the re-identification features; subsequently, these saliency distribution information are hard-masked, and the saliency of the pixel positions within the target area of ​​the global saliency distribution map that are less than the set threshold is set to 0, the saliency of the positions greater than the set threshold is set to 1, and the saliency of the positions not in the target area is set to 0. The generated hard mask results are used as pseudo-labels to supervise a decoder to extract the target saliency distribution map from the appearance features as the global spatial representation of the target.

4. The method for multi-target tracking guided by global appearance perception according to claim 1, characterized in that: The processing of the spatial enhancement module is as follows: For the feature map of the corresponding resolution at each scale, we first align it with the feature state of the current scale through a 3×3 convolution operation to generate a scale-adaptive perception map. For features of different scales, we adapt the feature maps of different resolutions through downsampling operations, and then further extract key features through 3×3 convolution to effectively guide multi-scale object detection. The process is expressed as: in, represents the saliency distribution output by the instance aggregation module, is the downsampled perceptual map, Represents the downsampling operation, scale_factor represents the downsampling factor, is the target perception map after convolution processing, and denote weight and bias respectively, represents the batch normalization layer, represents the activation function; The target perception map after convolution processing is multiplied with the original feature map, and the jump connection strategy is used to fuse the original feature information to form the final perception feature, which is then processed through the channel attention mechanism to obtain the perception-enhanced feature map. , the process is expressed as: Where F represents the original feature map, represents the target perception map, CA represents the channel attention mechanism; the processing process of the channel attention mechanism is: in represents a fully connected neural network, represents the activation function, represents average pooling, Represents maximum pooling.

5. The method for multi-target tracking guided by global appearance perception according to claim 1, characterized in that: In step 3, before matching, the coordinates of the detection frames of the previous frames are corrected using camera motion compensation technology to offset the impact of camera movement, and the Kalman filter is combined with historical detection information to predict the state of the target in the current frame to obtain the target tracking frame.

6. The method for multi-target tracking guided by global appearance perception according to claim 1, characterized in that: The specific implementation of step 3 includes the following sub-steps: Step 3.1: Calculate the re-identification perception score. The calculation method is as follows: in, represents the re-identification perception score, It is obtained by calculating the maximum significance value in each target detection box area, where i and j represent the horizontal and vertical coordinates of the pixels in the target box respectively. represents the target detection confidence, is a hyperparameter; Step 3.2: Perform hierarchical matching based on the target's re-identification perception score. Target detection frames with re-identification perception scores greater than the threshold are assigned to the first-stage matching, and the rest are assigned to the second-stage matching. The first stage of matching designs two metrics: motion matching and feature matching, and performs a one-to-one match between the current valid trajectory and the currently detected target; Unmatched tracks and target detection frames with re-identification perception scores less than a threshold will enter the second stage of matching. This is specifically implemented as follows: matching is performed based solely on the intersection over union (IoU) between the tracking frame obtained through camera motion compensation and Kalman filter prediction and the current candidate detection frame. An IoU threshold is set. If the IoU between the two is greater than this threshold, they are considered to be matched successfully, and the Hungarian algorithm is applied again for accurate pairing. For successfully matched tracks, their templates are updated to adapt to changes in appearance. Among the candidate frames that were not successfully matched in both stages, the candidate frames whose re-identification perception score is greater than a set value are initialized as new tracks and embedded in new identity templates. Conversely, unmatched tracks are marked as inactive. If a track fails to find a match for several consecutive frames, the track will be terminated. However, if an inactive track finds a match before termination, it will be reactivated.

7. The method for multi-target tracking guided by global appearance perception according to claim 6, characterized in that: Motion matching uses the classic intersection-over-union (IoU) to measure the degree of spatial overlap between the detection frame and the tracking frame; The feature matching degree evaluates the appearance similarity by calculating the cosine similarity between the two re-identified features; In order to integrate these two complementary information, their linear weighted sum is used as the final association metric. The final association metric similarity is input into the Hungarian algorithm to determine the target tracking frame that best matches the target detection frame in the current frame and the historical trajectory, thereby achieving stable and continuous tracking of the target.

8. The global appearance-guided multi-target tracking method according to claim 1, wherein: During training, the detection loss is summed up in a weighted manner , re-identification loss and spatial perception loss Combined to construct a joint loss function, the joint loss function Defined as: in 、 and It is a hyperparameter that controls the relative contribution of each task; The detection loss is defined as the weighted sum of the classification loss, bounding box regression loss, and object existence loss, each of which is controlled by hyperparameters; For the re-identification task, a contrastive learning training model is used to distinguish similar objects based on feature similarity. The loss function is formulated as: in represents the contrast loss, is the feature embedding of the i-th sample, which is obtained by extracting the feature embedding of the center coordinates of the target box of the i-th sample on the re-identification feature; this loss minimizes the positive sample The distance between them, while maximizing the negative samples The distance between the temperature parameters is a constant used to control the effect of sample distance on the gradient; In the training of spatial perception maps, the mean square error loss is adopted, and its formula is: in represents the spatial perception loss, N represents the total number of samples, and are the predicted value and pseudo label of the i-th sample respectively.

9. The method for multi-target tracking guided by global appearance perception according to claim 8, characterized in that: Detection loss Expressed as: Among them, α, β, and γ are hyperparameters used to balance the contribution of each loss term. For bounding box regression, the intersection-over-union (IoU) loss is used, which is defined as: in represents the bounding box regression loss, is the predicted bounding box, is the ground-truth bounding box; In terms of distinguishing foreground and background, binary cross entropy loss is used ; In terms of classification, binary cross entropy loss is also used as the classification loss .

10. A global appearance-guided multi-target tracking system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multi-target tracking method guided by global perception appearance is implemented as described in any one of claims 1 to 9.

Images