Multi-target tracking method based on multi-scale deformable attention mechanism

Abstract

The application discloses a kind of multi-target tracking methods based on multi-scale deformable attention mechanism, belong to video monitoring field.Firstly, the detection frame with multi-scale feature is extracted to video;Then the pedestrian is selected in the detection frame of video frame;Position information and appearance feature are encoded, and multi-scale robust feature is extracted through spatial attention module and channel attention module;Secondly, the robust feature of detection frame and tracking frame obtained is constructed into matching cost body C by Transform decoder;On the basis of fully using low confidence and high confidence detection frame, secondary matching association is carried out using matching cost body C, the loss between predicted value and real tracking value is calculated, the whole network is trained using loss value to obtain final parameter, so as to obtain the tracking result of pedestrian multi-target tracking video.It is simple in steps, convenient to use, and can efficiently and quickly track pedestrians in large-scale natural image dataset.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-target tracking method based on a multi-scale deformable attention mechanism, which is suitable for multi-target tracking of pedestrians in a large-scale natural image dataset and belongs to the technical field of image processing. BACKGROUND

[0002] With the development and extension of deep learning, deep learning models have made certain progress in various fields, and the field of computer vision is no exception. In recent years, computer basic tasks such as image classification, target detection and semantic segmentation have developed rapidly, and the multi-target tracking field has also been valued. As a major application of computer vision tasks, pedestrian multi-target tracking has been valued by relevant professionals, and more and more efforts have been put into it. The method of pedestrian multi-target tracking is mainly divided into two categories: one is a non-deep learning multi-target tracking method, and the other is a multi-target tracking method combined with deep learning.

[0003] The basic process of the traditional image non-deep learning multi-target method includes four parts: image preprocessing, bottom feature extraction, feature encoding, feature convergence and classifier classification. However, this method has high dependence on manual feature extraction. Manual feature extraction is time-consuming and requires relevant professional knowledge. In the era of big data, it is impossible to efficiently and accurately complete image classification by relying solely on manual feature extraction. At this time, deep learning models are very useful. The basic idea is to learn hierarchical feature representation in a supervised or unsupervised manner to complete the feature description of images from the bottom to the top. Deep learning models can automatically extract image features through layer-by-layer training and can learn powerful feature representations. Therefore, the multi-target tracking method based on deep learning models can achieve good results when tracking pedestrians. Existing deep learning models are mainly based on convolutional neural networks (CNN), and widely used models include AlexNet, VGGNet and residual network (ResNet).

[0004] The existing multi-target tracking method based on deep learning still has the following problems:

[0005] (1) Unable to handle high occlusion and dense crowd environment: Some methods can handle simple scenarios, such as low pedestrian flow, low crowd density and less occlusion. However, existing multi-target tracking methods cannot handle high occlusion and dense crowd environments. The reason is that multi-scale features are not considered comprehensively, and only existing detectors or association methods are used. These local sub-modules do not fully utilize image-level and instance-level multi-scale features. This will result in insufficient robustness of image and pedestrian features. At the same time, during matching, because the bounding box scales of different frames are different, it also interferes with matching.

[0006] (2) Detection target and tracking sequence are often manually designed matching process, and in the design of matching cost body, the proportion of appearance model and motion model needs to be adjusted according to the data set. In the design of multi-target tracking process, often only simple information is focused on, so the same method is not ideal for different data sets.

[0007] (3) In a simple matching process, in order to avoid background and false detection, low score detection box (low confidence box) is often directly discarded, but these low score detection boxes are often occluded tracking targets. Many methods do not make good use of low confidence boxes, so the tracking result is limited to the generated detection box. SUMMARY

[0008] The purpose of the application is to overcome the shortcomings of the prior art, and a multi-target tracking method based on a deformable attention mechanism is proposed, which has simple steps, good tracking effect, and can realize multi-target prediction and tracking of pedestrians in large-scale natural image data sets.

[0009] Technical scheme: In order to achieve the purpose of the application, the application provides a multi-target tracking method based on a multi-scale deformable attention mechanism, which uses a multi-target tracking model to track pedestrians of various scales in natural images. The multi-target tracking model includes a detection module for extracting multi-scale features, a multi-scale appearance feature extraction module based on a deformable attention mechanism, a semantic information fusion module, a Transformer encoder and a decoder connected in sequence. The semantic information fusion module includes a normalization layer LN, a multi-layer perception MLP, a spatial attention module SAM, and a channel attention module CAM connected in sequence.

[0010] The steps are as follows:

[0011] For multi-target tracking video containing pedestrians of various scales, the tracking video is decomposed into a set of video frames, and a detection module for extracting multi-scale features is used to set a detection box with multi-scale features for extracting pedestrian information;

[0012] For the pedestrians selected by the detection box in the video frame, the deformable attention mechanism is used to extract multi-scale feature appearance;

[0013] The extracted position information and appearance features are encoded, and the multi-scale robust features are extracted through the spatial attention module and the channel attention module;

[0014] The robust features of the detection box and the tracking box in the adjacent frame which have been successfully matched with the detection box are obtained through the Transformer decoder, and the matching cost body C of the detection box and the tracking box is constructed;

[0015] By using the matching cost volume C, on the basis of fully using the low-confidence and high-confidence detection boxes, the detection boxes are matched and associated with the tracking box sequence for a second time to obtain a final matching result;

[0016] The matching result is taken as a prediction value, then a loss value between the prediction value of the matching result and a real tracking value in the training set of the data set is calculated, a multi-target tracking model is trained according to the obtained loss value to obtain final parameters, and the above steps except training are repeated using the multi-target tracking model to complete multi-target tracking of the video, and finally a tracking result of the pedestrian multi-target tracking video is obtained.

[0017] The specific steps are as follows:

[0018] S1, for a given sequence video, two frames of images continuous in time are selected, and a multi-scale detector based on Resnet is used to output multiple boundary boxes and categories in the frame images and where i is the number of boundary boxes in a frame; the boundary box B set {B|(x, y, w, h, c, p)} is a detection box set, wherein x is the horizontal coordinate of the center point, y is the vertical coordinate of the center point, w is the width of the boundary box, h is the height of the boundary box, c is the category, and p is the detection confidence;

[0019] S2, the detection boxes in the frame are obtained and Then, the images in the regions of these detection boxes are input into the re-ID network, and the re-ID network performs attention weighting on the different positions of the detection boxes through self-attention, and autonomously learns the features of the unoccluded parts of the positive samples. These features are the most representative appearance feature representations. Therefore, even in the face of partially occluded environments, the most robust appearance feature representations can still be obtained, and robust pedestrian feature representations F' are obtained.

[0020] S3, the position information of the detection boxes in the frame image is encoded by a multilayer perceptron, and finally F" is obtained by inputting the robust appearance representation F' into an adaptive cost network, the appearance feature and information position attention fusion are completed, and then the channel self-attention and spatial self-attention are performed to obtain the feature F used for the final feature matching. c ;

[0021] S4, initialize the detection frame obtained in S1 as the tracking frame, that is, assign a unique identity to each tracking frame, save the obtained appearance features using a memory module formed by a memory, set a threshold T, save the position information and appearance features of the tracking frame from t-T to t-1 frames, then obtain the feature Txd through the Transformer self-attention encoder, change the feature Txd obtained by the current frame, that is, the t-th frame, through the self-encoder, and then obtain the matching cost body C through the Transformer decoder;

[0022] S5, using the Hungarian matching algorithm to perform the first round of matching according to the cost body C, obtaining the matching pair P[d, t] of the detection frame and the tracking frame, the detection frame UD that is not matched, and the tracking frame UT that is not matched, wherein d in P[d, t] represents the matched detection frame, and t represents the matched tracking sequence; then, the detection frame that is not matched and the tracking frame that is not matched are repeatedly subjected to step S4 to obtain a new cost body C', and then the Hungarian algorithm is used to perform the second round of matching according to the new cost body C', and finally the last tracking frame T is obtained after careful screening of the low-confidence frame and the high-confidence detection frame i＝t ={trk|trk∈B};

[0023] S6, obtaining the last tracking frame T of S5 i＝t ={trk|trk∈B}, calculating the loss of the predicted tracking frame and the true value, regarding the whole process as a classification process, using cross-entropy loss to repeatedly train the multi-target tracking model through S1-S5 until a predetermined number of training rounds is reached, at which time the obtained network parameters are used as the final multi-target tracking model parameters; after obtaining the final training result of S5, S1-S5 are performed frame by frame, and finally the tracking frame of the whole video is obtained In this way, the pedestrian multi-target tracking is completed.

[0024] Further, in step S1, the specific calculation method of the multi-scale feature detection frame is as follows:

[0025] S11, for a given pedestrian multi-target tracking video, a sequence video containing N color pictures is set as where H is the height of the video picture, W is the width of the video picture, N is the total number of video pictures, two consecutive frames of images, t-1 frame image I t-1 and t frame image I t , t∈N, and the first frame and the second frame are taken as examples;

[0026] S12, the first frame image and the second frame image are input into the ResNet network for convolution processing, and the FPN feature pyramid is used on the basis of the ResNet network to construct multi-scale pedestrian detection image features Both the first and second frames yield feature maps at four scales. The features from the first frame are denoted as F1, and the features from the second frame as F2. Specifically, the feature maps obtained are from stages C2 to C5 of layers 2-5 of the FPN feature pyramid in the ResNet network, where C2 represents a downsampling of 2. 2 C3 represents downsampling 2 3 Similarly, the feature maps at these three scales are convolved with a 1×1 stride of 1 to obtain features at four scales. These are then upsampled and added to feature maps of the same scale to obtain feature maps at stages M2-M4. The M5 feature map is obtained directly from the C5 feature map through a 1×1 convolution. M2-M4 are then convolved with another 1×1 convolution to obtain the final P2-P4 feature maps. The P5 feature map is simply M5. Although C1 has the highest resolution, it has too few semantic features and is therefore not used. The final feature maps are... The scales, from smallest to largest, are as follows: Then, these feature maps are predicted, and the resulting bounding boxes are mapped back onto the source-sized image to finally obtain the location and classification of pedestrians, which is a series of bounding boxes {B|(x, y, w, h, c, p)}.

[0027] Furthermore, in step S2, the specific calculation method for the multi-scale appearance features of the detection box is as follows:

[0028] S21. For the first frame of two consecutive frames, I1∈R H×w×3 The image I2∈R of the second frame H×w×3 Take the detection boxes {B1|(x, y, w, h, c, p)} and {B2|(x, y, w, h, c, p)}, and then... Cut into small rectangular pieces Where N is the total number of blocks into which the detection frame is divided, N = h × w / P 2 Then, the data is fed into a deformable Transformer encoder for feature transformation. The specific feature transformation includes the following steps:

[0029] S21.1 For the small rectangular block features r of the input detection box image, the number of channels is changed from 3 to D through a linear embedding transformation. At this point, the initial input features of the small rectangular block are obtained, denoted as x. i The formula is as follows;

[0030]

[0031] S21.2 Due to the parallelism of the Transformer encoder, the initial input features x i Add position encoding e i Finally, the feature representation f of the entire detection box is obtained, as shown in the following formula:

[0032]

[0033] S21.3 input the obtained entire detection frame feature f into the multi-scale deformable attention for processing, and then perform displacement window-based transformer encoder processing, f i-1 output by the transformer encoder of the i-1 layer, and then added to itself after normalization layer LN and multi-scale deformable self-attention based obtained After normalization layer LN and multi-layer perceptron MLP processing, add to itself, and finally obtain window-based output f i , the formula is as follows.

[0034]

[0035]

[0036] wherein LN represents a normalization layer, MLP represents a multi-layer perceptron; MSDA represents deformable self-attention based, and variable self-attention calculation of formula (4) is performed between blocks; formula (4) is the formula of the deformable self-attention mechanism, z q represents a query vector; p q represents the position of the query vector; x is the input feature; M is the number of attention heads; W k and W v are the projection matrices of key and value respectively; C is the number of channels; Δp mqk is the relative position offset in the position encoding; φ represents the softmax function; W m represents that the result after applying attention on the value is linearly transformed to obtain the output result of different heads.

[0037] Further, the appearance feature and information position attention fusion method in step S3 is as follows:

[0038] S31, first, the position information of the detection frame and the tracking frame is encoded using a multi-layer perceptron to obtain Fp, and then added to the robust pedestrian appearance feature F' obtained in S2 to obtain the final output feature F'', the calculation formula is as follows:

[0039] F p = MLP ([x, y, w, h]) (5)

[0040] F″=F′⊕F p (6)

[0041] wherein MLP is a multi-layer perceptron, x is a horizontal coordinate of a center point of a detection frame, y is a vertical coordinate of the center point, w is a width of the detection frame, h is a height of the detection frame, and represents element-wise addition;

[0042] S32, output the feature F" to a spatial attention module SAM to obtain a spatial attention score M s Then, F" is multiplied by the spatial attention score M s , and then enters a normalization layer to be added to F" to obtain a final feature map F s .

[0043] The specific operation of the spatial attention module is as follows: first, the spatial attention module SAM is used to perform global pooling GP on the feature map F" in the channel dimension, thereby compressing the channel dimension, and then average pooling AP is performed on the feature map F" in the channel dimension, thereby compressing the channel dimension; the two feature maps after compression of the channel dimension are spliced in the channel dimension to obtain a feature map of HxWx2, and then a 7x7 convolution operation is performed on the spliced HxWx2 feature map and then an activation function is performed, thereby obtaining a feature after the spatial attention module, and the formula is as follows:

[0044]

[0045]

[0046] M s = σ (conυ (concat (Aυgpool (F), Aυg max (F))) ) (9)

[0047]

[0048] wherein F w，h in formula (7) represents each pixel value on the feature map, represents an average value in the channel direction, represents a maximum value in the channel direction, concat(·) represents splicing of feature values in the channel direction, σ(·) represents an activation function, a sigmoid function is adopted, and the function value is mapped to 0-1, M s represents an attention score, represents element-wise multiplication, and F s represents a final feature map.

[0049] S33, the final feature map F s is sent to a channel attention module CAM to obtain a channel attention score M c , and then F s is multiplied by the channel attention score M c , and then enters a normalization layer to be added to Fs The final feature map F is obtained by adding c ;

[0050] The specific operation of the channel attention module CAM is as follows: first, global pooling GP is performed on the feature map F in the spatial dimension, so as to compress the spatial size, and then average pooling AP is performed on the feature map F in the spatial dimension, so as to compress the spatial size; then, the two 1x1xc feature maps are put into the MLP with shared parameters, the features in the channel dimension are learned, the importance of each channel is obtained, the two channel attention scores are added, the 1x1xc feature map is obtained, and then the activation function σ(·) is passed, so as to obtain the feature after passing through the channel attention module, and the formula is as follows: s s

[0051]

[0052]

[0053] M c (F s )=σ(MLP(AυgPool(F s ))+MLP(Maxpool(F s ))) (13)

[0054]

[0055] Wherein, F x，y，c in formula (11) represents each pixel value of the feature map represents the average value of the spatial dimension WH in each layer channel c, represents the maximum value of the spatial dimension WH in each layer channel c; MLP represents a multi-layer perceptron with shared parameters, σ(·) represents an activation function, a sigmoid function is adopted to map the function value to 0-1, M c represents an attention score, represents element-wise multiplication, F c represents the final feature map.

[0056] Further, the specific method of step S4 is as follows:

[0057] S41, the final feature map F c obtained in S3 is put into the memory module, the memory module is a queue structure, and the F c features are saved in sequence, a threshold T is set, the F c from t-T to t-1 frames is saved, when the tth frame is processed, the features F c from t-T to t-1 in the memory module are used, after the image of the tth frame is processed, the feature F​​c into the queue. At the same time, delete the F c ; Then get the feature Txd through the Transformer self-attention encoder, and get the matching cost body C after the feature Txd obtained by the current frame, i.e., the t-th frame, through the Transformer decoder after the change.

[0058] S42, the specific steps of changing the feature Txd obtained by the current frame, i.e., the t-th frame, through the self-encoder are as follows: in the t-th frame, the features from t-T to t-1 are put into the Transformer self-encoder to obtain the feature Txd, and for the tracking sequence, there are N tracking sequences, the features of the N tracking sequences are put into the Transformer self-encoder, then the features are spliced in the first dimension to obtain a feature map of (N×T)×d, and for the detection sequence, because there is only one frame of detection box, assuming that the detection number is M, in order to align the dimension with the tracking sequence, the detection sequence xN is obtained, that is, N×M×d detection boxes. c

[0059] The specific process of the tracking sequence into the Transformer self-encoder is as follows: first, the Txd feature map is projected through the projection matrix W q , W s and W υ to obtain new features Q, K and V; then the transpose of K is multiplied by Q, and then divided by the square root of the dimension of the new feature, so that the scale consistency is guaranteed, and the attention score is obtained at the same time. Finally, the attention score is scaled to 0-1, and then the attention score is multiplied by the V matrix. Then the feature is put into the feedforward neural network FFN, and finally the layer normalization is performed to obtain the final feature (N×T)×d, and the formula is as follows:

[0060] Q=FW q , K=FW k , y=FW υ (15)

[0061]

[0062]

[0063] where W q , W k and W υ are projection matrices that map the dimension of Txd to the dimension of Txd', Kt is the transpose matrix of K, the softmax function takes the exponential of each value and then normalizes it, so that the relationship between different values can be established, and the score can also be scaled to 0-1, denotes element-wise multiplication;​

[0064] S43, put the detection frame feature N x M x d obtained in S42 and the tracking frame feature (N x T) x d into the Transformer decoder, pass through multi-head self-attention and multi-head cross-attention, and finally obtain an assignment matrix NxM through layer normalization and residual connection, wherein the rows and columns in the assignment matrix are one-hot vectors, because one detection target can correspond to only one tracking target and one tracking target can correspond to only one detection target, the unit element of the assignment matrix is a probability value of 0-1, and the assignment matrix is a matching cost body C.

[0065] Further, the specific method of generating a matching result in step S5 is as follows:

[0066] S51, after obtaining the matching cost body C, Hungarian matching algorithm is used to match according to the cost body C to obtain a matching pair P[d, t] of the detection frame and the tracking frame, a detection frame UD1 that is not matched, and a tracking frame UT1 that is not matched, wherein d in P[d, t] represents a matched detection frame, and t represents a matched tracking sequence;

[0067] S52, the detection frame with a p value less than 0.5 in S1 is defined as a low-confidence detection frame D2, and the tracking frame UT that is not matched is again subjected to steps S2, S3 and S4 to obtain a cost body C', and then Hungarian algorithm is used to match according to the cost body C' to obtain a matching pair P[d2, ut] of the low-confidence detection frame D2 and the tracking frame UT that is not matched, a detection frame UD2 that is not matched, and a tracking frame UT2 that is not matched, wherein d2 in P[d2, ut1] represents a matched low-confidence detection frame, and ut1 represents a matched tracking sequence; the tracking frame UT2 that is not matched is deleted from T t-1 , it is judged that these pedestrian targets have left the field of view or are blocked by other pedestrians or backgrounds, and therefore UT2 is put into T lost ; the deletion condition is that when the unit in T lost is stored for more than ε frames, T lost and UT2 are deleted at the same time; the detection frame UD2 that is not matched is compared with a threshold value ε, the detection frame UD2 that is not matched and is greater than the threshold value ε is initialized as a new tracking sequence T new , and the detection frame UD2 that is not matched and is less than the threshold value ε is regarded as a non-target foreground or background and is therefore directly deleted; after careful screening of the low-confidence frame and the high-confidence detection frame, the final tracking frame T i＝t ={trk|trk∈B} is obtained.

[0068] Further, the specific method of using cross-entropy loss to repeatedly train the multi-target tracking model in step S6 is as follows:

[0069] S61, in each training phase, after obtaining the last tracking result, the predicted result T = {trk1, trk2…trk n} and the loss of the true value T truth = {trk1, trk2…trk n} are calculated, the whole process can be regarded as a classification process, the cross-entropy loss is used to train the whole network until the pre-set training round is reached, at this time, the obtained network parameters are the last model parameters. In the inference stage, after obtaining the result according to S5, S1, S2, S3, S4 and S5 are performed frame by frame, and finally the result is obtained, which is the tracking frame of the whole video

[0070] Advantages:

[0071] The application utilizes the self-attention of the deformable attention mechanism to extract the global features of the image, and pays more attention to the relationship between the target position and the positions of other objects in the image. Meanwhile, the deformable attention mechanism can make the attention not be the same at all positions, but start from the nearby sampling and assign different weights, so that the network training can accelerate the convergence. In addition, it can also effectively reduce the computational cost, has a larger receptive field than the ordinary convolutional neural network, and has stronger model representation ability. Meanwhile, the cross-attention in the attention mechanism can naturally complete the pairing of detection and tracking, avoid post-processing operation, and make the pedestrian multi-target tracking system end-to-end. Meanwhile, the network adopts an anchor-free detection method, which can avoid obtaining heterogeneous features when extracting target object features. The attention uses multi-scale features, which can effectively target pedestrians of various scales in natural images. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 It is the network structure diagram of the multi-target tracking method based on the multi-scale deformable attention mechanism. DETAILED DESCRIPTION

[0073] The embodiments of the application will be further described below with reference to the accompanying drawings.

[0074] As Figure 1As shown, the application discloses a multi-target tracking method based on a deformable attention mechanism, various scale pedestrians in natural images are tracked by using a multi-target tracking model, the multi-target tracking model comprises sequentially connected detection modules for extracting multi-scale features, a multi-scale appearance feature extraction module of the deformable attention mechanism, a semantic information fusion module, a Transformer encoder and a decoder; wherein the semantic information fusion module comprises sequentially connected normalization layers LN, a multi-layer perceptron MLP, a spatial attention module SAM, a channel attention module CAM and a normalization layer LN based on a deformable self-attention MSDA;

[0075] Specifically, the following steps are included:

[0076] S1, for a given pedestrian multi-target tracking video, represent a sequence of videos, the sequence of videos contains N color pictures, wherein H is the height of the video picture, W is the width of the video picture, N is the total number of video pictures, t-1 frame images R t-1 and t frame images R t are selected in turn, and the images are output by a multi-scale detector based on Resnet, and the boundary box and the class in the image are output wherein i is the number of boundary boxes in a frame.{B|(x, y, w, h, c, p)}, x is the horizontal coordinate of the center point, y is the vertical coordinate of the center point, w is the width of the boundary box, h is the height of the boundary box, c is the class, and p is the detection confidence;

[0077] S11, for a given sequence of videos select I1 R H×W×3 and I2 R H×W×3 Two pictures are required to be continuous in time;

[0078] S12, the first frame image and the second frame image are input into the ResNet network for convolution processing, and a multi-scale pedestrian detection image feature is constructed by using a FPN feature pyramid based on ResNet The first frame and the second frame image will obtain four scale feature maps, the first frame image feature is denoted as F1 and the second frame image feature is denoted as F2, and the specific method is as follows: the feature maps obtained in C2-C5 stages of ResNet are taken, the three scale feature maps are subjected to 1x1 convolution with a step of 1 to obtain four scale features, then the same scale feature maps are added after being up-sampled to obtain M2-M4 stage feature maps, and M5 feature map is directly obtained by 1x1 convolution on C5 feature map, M2-M4 are subjected to 1x1 convolution again to obtain final P2-P4 feature maps, and P5 feature map is M5. C1 has the highest resolution but too few semantic features, so C1 is not used. Finally, the feature maps The scales from small to large are in order: Then the feature maps are predicted, and the obtained bounding box is mapped back to the source size image to finally obtain the positioning and classification of the pedestrian, that is, a series of bounding boxes {B|(x, y, w, h, c, p)};

[0079] Further, the specific calculation method of appearance feature extraction on the detection frame in step S2 is as follows:

[0080] S2, obtain the detection frame in the frame After that, the images containing the detection frame are input into the re-ID network, and the self-attention in the re-id network is used to perform attention weighting on different positions of the detection frame, and the features of the unoccluded part of the positive sample are learned autonomously. These features are the most representative appearance feature representations. In this way, even in the face of partially occluded environment, the most robust appearance feature representation can still be obtained. Robust pedestrian feature representation F' is obtained.

[0081] S21, for the above obtained first frame image I1∈R H×W×3 and the second frame image I2∈R H×W×3 , take the detection frame {B1|(x, y, w, h, c, p)} and {B2|(x, y, w, h, c, p)} in it, cut the detection frame into small rectangular blocks where N is the total number of blocks N = h x w / P that the detection frame is cut into 2 Then send it into the deformable Transformer for feature transformation. The specific feature transformation includes the following steps:

[0082] (1) For the small rectangular block features r of the input detection frame image, the channel number is changed from 3 to D through linear embedding transformation. At this time, the initial input feature of the small rectangular block is obtained, denoted as x i , the formula is as follows;

[0083]

[0084] (2) Due to the parallelism of the Transformer, the fragmented small rectangular blocks are directly input, and the positional order relationship between the blocks cannot be obtained. Therefore, the obtained x i is added with position encoding e i , and finally the feature representation f of the entire detection frame is obtained, and the formula is as follows:

[0085]

[0086] (3) The obtained f is input into the multi-scale deformable attention for processing, and is processed into the transformer based on the displacement window. fi-1 After normalization layer LN and multi-scale deformable self-attention, add to itself to obtain After normalization layer LN and multi-layer perceptron MLP processing, add to itself, finally obtain window-based output f i , the formula is as follows;

[0087]

[0088]

[0089] Wherein, LN represents the normalization layer, MLP represents the multi-layer perceptron; MSDA represents the deformable self-attention based on the formula (4) between blocks and blocks; formula (4) is the formula of the deformable self-attention mechanism, z q represents the query vector; p q represents the position of the query vector; x is the input feature; M is the number of attention heads; W k and W v are the projection matrix of key and value respectively; C is the number of channels; Δp mqk is the relative position offset in the position coding; φ represents the sofimax function; W m represents the linear transformation of the result after the attention is applied on the value to obtain the output result of different heads;

[0090] S3, the position information of the detection frame is encoded by the multi-layer perceptron, and finally sent into the adaptive cost network together with the robust appearance representation F' to obtain F'', after the channel self-attention and the spatial self-attention, the final feature representation F c is obtained, this feature F c will be used for the final feature matching;

[0091] S31, first, the position information of the detection frame and the tracking frame is encoded by the multi-layer perceptron to obtain F p , then, the robust appearance feature F' obtained by S2 is added and merged to obtain the final output feature F'', the calculation formula is as follows:

[0092] F p = MLP ([x, y, w, h]) (5)

[0093] F″ = F′ ⊕ F p (6)

[0094] Wherein, MLP is the multi-layer perceptron, x is the horizontal coordinate of the center point of the detection frame, y is the vertical coordinate of the center point, w is the width of the detection frame, and h is the height of the detection frame. Wherein, ⊕ represents element-level addition; ​

[0095] S32, after S31, F" is obtained, F" is sent to the spatial attention module, and the spatial attention score M is obtained s , then F" is multiplied by the spatial attention score M s , and then enters the normalization layer, and F" is added to obtain the final feature map F s .

[0096] The specific operation of the spatial attention module is: first, the feature map F" is globally pooled in the channel dimension, thereby compressing the channel dimension, and then the feature map F" is averaged pooled in the channel dimension, thereby compressing the channel dimension; the two feature maps after compressing the channel dimension are spliced in the channel dimension to obtain a HxWx2 feature map, and then the spliced HxWx2 feature map is subjected to a 7x7 convolution operation and then an activation function, thereby obtaining a feature after the spatial attention module, the formula is as follows:

[0097]

[0098]

[0099] M s = σ (conυ (concat (Aυgpool (F), Aυg max (F))) ) (9)

[0100]

[0101] Wherein F w，h in formula (7) represents each pixel value on the feature map, represents the average value in the channel direction, represents the maximum value in the channel direction, concat(·) represents the concatenation of feature values in the channel direction, σ(·) represents an activation function, and a sigmoid function is adopted to map the function value to 0-1, M s represents the attention score, represents element-wise multiplication, F s represents the final feature map.

[0102] s33, F s obtained after S32, F s is sent to the channel attention module, and the channel attention score M c is obtained, then F s is multiplied by the channel attention score M c , and then enters the normalization layer, and F s is added to obtain the final feature map F c .

[0103] The specific operation of the channel attention module is as follows: First, the feature map F... s Perform global pooling (GP) along the spatial dimension to compress the spatial size, and then apply it to the feature map F. s Average pooling (AP) is performed along the spatial dimension to compress the spatial size. Then, two 1×1×C feature maps are fed into a shared-parameter MLP to learn channel-dimensional features and obtain the importance of each channel. The attention scores of the two channels are then added together to obtain a 1×1×c feature map, which is then passed through an activation function to obtain the features after the channel attention module. The formula is as follows:

[0104]

[0105]

[0106] M c (F s )=(MLP(AυgPool(F s ))+MLP(Maxpool(F s (13)

[0107]

[0108] In formula (11), F x，y，c Represents the pixel value of the feature map This means taking the average value of the spatial dimension WH on each channel c. To represent taking the maximum value of spatial dimension WH on each channel c, MLP stands for Multilayer Perceptron with shared parameters, σ(·) represents the activation function, using the sigmoid function to map the function value between 0 and 1, and Mc represents the attention score. F represents element-wise multiplication. c This represents the final feature map.

[0109] S4. Save the obtained appearance features, set a threshold T, save the position information and appearance features of the tracking box from frame tT to frame t-1, and then obtain the features Txd through the Transformer self-attention encoder. After passing through S2 and S3, the current frame, i.e. the t-th frame, obtain the features d. Transform these features and obtain the matching cost body C through the Transformer decoder.

[0110] S41. Obtain F from S3 c Feature maps are stored in a memory module, which consists of a queue. c The features are saved sequentially, and a threshold T is set to select the F features from frame tT to frame t-1. cSave it, when processing the t-th frame, then use the t-T to t-1 features F in the memory module c , after processing the t-th frame of image, the t-th frame of feature F c is pressed into the queue. At the same time, delete the Fc of t-T frame. Then get the feature Txd through the Transformer self-attention encoder, get the feature d after the t-th frame through S2 and S3, and get the matching cost body C after transforming these features through the Transformer decoder.

[0111] S42, in the t-th frame, the features from t-T to t-1 are put into the Transformer self-encoder to get the feature Txd, for tracking sequence, assuming there are N tracking sequences, put the features F c of N tracking sequences into the Transformer self-encoder, then concatenate these features in the first dimension to get the feature map of (N x T) x d, for detection sequence, because there is only one frame of detection box, assuming the number of detection is M, in order to align the dimension with the tracking sequence, get NxM queries for detection sequence xN.

[0112] The specific process of tracking sequence into Transformer self-encoder is: first, the Txd feature map is projected through projection matrix W q , W k and W υ to get new features Q, K and V. Then transpose K and multiply it with Q, and then divide by the square root of the new feature dimension, which can guarantee the scale consistency, and at the same time get the attention score. Finally, scale the attention score to 0-1, and then multiply the attention score with V matrix. Then put the feature into feedforward neural network FFN, and finally get the final feature (N x T) x d through layer normalization, the formula is as follows:

[0113] Q = FW q , K = FW k , y = FW υ (15)

[0114]

[0115]

[0116] Where W q , W k and W υ are projection matrices that map Txd dimension to Txd' dimension, Kt is the transpose matrix of K, and softmax function takes the exponential of each value and then normalizes it, which can establish the relationship between different values, and also scale the score to 0-1, denotes element-wise multiplication.

[0117] S43, put the detection frame feature N x M x d obtained in S42 and the tracking frame feature (N x T) x d into the Transformer decoder, pass through multi-head self-attention and multi-head cross-attention, and finally obtain an assignment matrix of N x M through layer normalization and residual connection, wherein the rows and columns in the assignment matrix are one-hot vectors, because one detection target can correspond to only one tracking target and one tracking target can correspond to only one detection target, the unit element of the assignment matrix is a probability value of 0-1, and the assignment matrix is the matching cost body C.

[0118] S5, after obtaining the matching cost body C, a first round of matching is performed on the cost body C by using the Hungarian matching algorithm to obtain a matching pair P[d, t] of detection and tracking, a detection frame UD that is not matched, and a tracking frame UT that is matched, wherein d in P[d, t] represents a detection frame that is matched, and t represents a tracking sequence that is matched. Then, S4 is performed again according to the detection frame that is not matched and the tracking frame that is not matched to obtain a cost body C', and a second round of matching is performed on the cost body C' by using the Hungarian algorithm to obtain a final tracking frame T i＝t = {trk | trk e B};

[0119] S51, after obtaining the matching cost body C through S4, matching is performed on the cost body C by using the Hungarian matching algorithm to obtain a matching pair P[d, t] of detection and tracking frames, a detection frame UD1 that is not matched, and a tracking frame UT1 that is not matched, wherein d in P[d, t] represents a detection frame that is matched, and t represents a tracking sequence that is matched.

[0120] S52, then the low-confidence detection frame in S1 and the tracking frame UT that is not matched are subjected to the steps of S2, S3 and S4 again, a cost body C' is obtained, and matching is performed again on the cost body C' by using the Hungarian algorithm to obtain a matching pair P[d2, ut] of the low-confidence detection frame D2 and the tracking frame UT that is not matched, a detection frame UD2 that is not matched, and a tracking frame UT2 that is not matched, wherein d2 in P[d2, ut] i ] represents a low-confidence detection frame that is matched, and ut1 represents a tracking sequence that is matched. The tracking frame UT2 that is not matched is deleted from T t-1 , because these pedestrian targets may leave the field of view, but may also be blocked by other pedestrians or the background, so the UT2 is put into T lost , and T lost and UT2 are deleted at the same time when the unit in T lost is stored for more than e frames. The detection frame UD2 that is not matched is compared with the threshold e, and the detection frame UD2 that is not matched and is greater than the threshold e is initialized as a new tracking sequence T newUnmatched detection boxes UD2 less than the threshold value epsilon are regarded as non-target foreground or background, and thus are directly deleted. After careful screening of low-confidence boxes and high-confidence detection boxes, the final tracking box T is obtained i＝t = {trk | trk e B}.

[0121] S6, in the training stage, after obtaining the final tracking result, the loss of the prediction result and the true value is calculated, the whole process can be regarded as a classification process, the cross-entropy loss is used to train the whole network, until the pre-set training round is reached, at this time the obtained network parameter is the final model parameter. In the inference stage, after obtaining the result according to S5, S1, S2, S3, S4 and S5 are performed frame by frame, and finally the tracking box of the whole video is obtained , so the pedestrian multi-target tracking is completed;

[0122] S61, in the training stage, after obtaining the final tracking result, the loss of the prediction result T = {trk1, trk2…trk n} and the true value T truth = {trk1, trk2…trk n} is calculated, the whole process can be regarded as a classification process, the cross-entropy loss is used to train the whole network, until the pre-set training round is reached, at this time the obtained network parameter is the final model parameter. In the inference stage, after obtaining the result according to S5, S1, S2, S3, S4 and S5 are performed frame by frame, and finally the tracking box of the whole video is obtained

[0123] The data set of the application will be further described below:

[0124] Experimental data set:

[0125] The experiment of the application uses the MOT17 and MOT20 data sets in the MOTChallenge.

[0126] MOTChallenge consists of MOT15, MOT16 (MOT17) and MOT20. MOT15 contains 22 video sequences, 11 for training and 11 for testing, MOT15 dataset collects older datasets in the past. MOT16 dataset is a new version of video, video and annotation are newly made, compared with MOT15, MOT16 is more challenging because the video has higher pedestrian density. MOT16 contains 14 video sequences, 7 for training and 7 for testing, annotated with the public detector DPMv5] MOT17 still uses the video of MOT16 dataset, but is annotated with more accurate ground truth boxes, and three different detectors are used for each video, Faster-RCNN, DPM and SDP. Trackers need to work with detectors to prove their robustness. MOT20 contains 8 videos, 4 for training and 4 for testing, with an average of 245 pedestrians per frame in the most crowded video. MOTChallenge series is divided into public detection track and private detection track, because many existing algorithms are TBD paradigm, detection quality seriously affects tracking quality. Therefore, the public track requires the use of official detectors for multi-object tracking, which can more fairly compare algorithms. While the private detection track can use its own designed detector or its own designed multi-object tracking system, the purpose is to compare the best multi-object tracking algorithm, according to s1, the method uses the private detection track.

[0127] The above describes in detail a multi-target tracking method based on a multi-scale deformable attention mechanism provided by the embodiments of the present application. For those skilled in the art, according to the idea of the embodiments of the present application, the specific implementation and application range will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A multi-target tracking method based on a multi-scale deformable attention mechanism, characterized in that, A multi-object tracking model is used to track pedestrians at various scales in natural images. The multi-object tracking model includes a detection module for extracting multi-scale features, a multi-scale appearance feature extraction module based on deformable attention mechanism, a semantic information fusion module, a Transformer encoder, and a decoder. The semantic information fusion module includes a normalized layer LN, a deformable self-attention based MSDA, a normalized layer LN, a multilayer perceptron MLP, a spatial attention module SAM, and a channel attention module CAM. The steps are as follows: For multi-target tracking videos containing pedestrians at various scales, the tracking video is decomposed into a set of video frames, and a detection module for extracting multi-scale features is used to set detection boxes with multi-scale features to extract pedestrian information. For pedestrians selected by the detection box in the video frame, a deformable attention mechanism is used to extract multi-scale features and appearance. The extracted location information and appearance features are encoded, and robust features at multiple scales are extracted through spatial attention module and channel attention module; The robust features of the obtained detection box and the tracking box in the adjacent frame that has successfully matched the detection box are used by the Transformer decoder to construct the matching cost body C between the detection box and the tracking box. Using the matching cost body C, and based on the full use of low-confidence and high-confidence detection boxes, a secondary matching association is performed on the sequence of detection boxes and tracking boxes to obtain the final matching result; The matching result is used as the predicted value. Then, the loss value between the predicted value of the matching result and the real tracking value in the training set of the dataset is calculated. The multi-object tracking model is trained based on the obtained loss value to obtain the final parameters. The multi-object tracking model is then used to repeat the above steps except for training to complete the multi-object tracking of the video, and finally the tracking result of the pedestrian multi-object tracking video is obtained.

2. The multi-target tracking method based on a multi-scale deformable attention mechanism according to claim 1, characterized in that, The specific steps are as follows: S1. For a given video sequence, select two temporally consecutive frames, pass them through a ResNet-based multi-scale detector, and output multiple bounding boxes and categories for each frame. and Where i is the number of bounding boxes in a frame; the bounding box set B {B|(x, y, w, h, c, p)} is the detection box set, where x is the x-coordinate of the center point, y is the y-coordinate of the center point, w is the width of the bounding box, h is the height of the bounding box, c is the class, and p is the detection confidence. S2, Obtain the detection box within the frame. and Then, the images within these detection box areas are input into the re-ID network. The re-ID network uses self-attention to apply attention weights to different positions of the detection box and autonomously learns the features of the unoccluded parts of the positive samples. These features are the most representative appearance feature representations. In this way, even in the face of partially occluded environments, the most robust appearance feature representations can still be obtained, resulting in a robust pedestrian feature representation F′. S3. The position information of the detection boxes in the frame image is encoded using a multilayer perceptron. Finally, it is combined with the robust appearance representation F′ and input into the adaptive cost network to obtain F″, completing the fusion of appearance features and information location attention. Then, after channel self-attention and spatial self-attention, the feature F is obtained for the final feature matching. c ; S4. Initialize the detection box obtained in the first frame of S1 into a tracking box, that is, assign a unique identity to each tracking box, save the obtained appearance features using a memory module composed of memory, set a threshold T, save the position information and appearance features of the tracking boxes from frame tT to t-1, and then obtain the feature Txd through the Transformer self-attention encoder. After the feature Txd obtained by the autoencoder in the current frame, i.e. the t-th frame, is transformed, it is then passed through the Transformer decoder to obtain the matching cost body C. S5. Using the Hungarian matching algorithm, perform the first round of matching based on the cost body C. This yields matching pairs P[d, t] between detection boxes and tracking boxes, unmatched detection boxes UD, and unmatched tracking boxes UT, where d in P[d, t] represents the matched detection box and t represents the matched tracking sequence. Then, repeat step S4 with the unmatched detection boxes and unmatched tracking boxes to obtain a new cost body C′. Next, use the Hungarian algorithm to perform the second round of matching based on the new cost body C′. After carefully screening low-confidence boxes and high-confidence detection boxes, the final tracking box T is obtained. i＝t ={trk|trk∈B}; S6, obtain the final tracking bounding box T from S5. i＝t After setting {trk|trk∈B}, calculate the loss between the predicted tracking box and the ground truth value, and treat the whole process as a classification process. Use cross-entropy loss to repeat S1-S5 to train the multi-object tracking model until the preset number of training rounds is reached. The network parameters obtained at this time are used as the final multi-object tracking model parameters.

3. The multi-target tracking method based on a multi-scale deformable attention mechanism according to claim 2, characterized in that, In step S1, the specific calculation method for multi-scale feature detection box extraction is as follows: S11. For a given pedestrian multi-target tracking video, suppose a video sequence containing N color images is... Where H is the height of the video image, W is the width of the video image, and N is the total number of video images. Two consecutive frames are selected: frame t-1, image I. t-1 and the image I of frame t t , t∈N, the following uses the first and second frame images as examples; S12. Input the first and second frame images together into a ResNet-based network for convolution processing. Based on the ResNet network, use the FPN feature pyramid to construct multi-scale pedestrian detection image features. Both the first and second frames yield feature maps at four scales. The features from the first frame are denoted as F1, and the features from the second frame as F2. Specifically, the feature maps obtained are from stages C2 to C5 of layers 2-5 of the FPN feature pyramid in the ResNet network, where C2 represents a downsampling of 2. 2 C3 represents downsampling 2 3 Similarly, the feature maps at these three scales are convolved with a 1×1 stride of 1 to obtain features at four scales. These are then upsampled and added to feature maps of the same scale to obtain feature maps at stages M2-M4. The M5 feature map is obtained directly from the C5 feature map through a 1×1 convolution. M2-M4 are then convolved with another 1×1 convolution to obtain the final P2-P4 feature maps. The P5 feature map is simply M5. C1 corresponds to the bottom layer of the pyramid; therefore, although it has the highest resolution, it has too few semantic features and is not used. The final feature map is... The scales, from smallest to largest, are as follows: Then, these feature maps are predicted, and the resulting bounding boxes are mapped back onto the source-sized image to finally obtain the location and classification of pedestrians, which is a series of bounding boxes {B|(x, y, w, h, c, p)}.

4. The multi-target tracking method based on a multi-scale deformable attention mechanism according to claim 1, characterized in that, In step S2, the specific calculation method for the multi-scale appearance features of the detection box is as follows: S21. For the first frame of two consecutive frames, I1∈R H×W×3 The image I2∈R of the second frame H×W×3 Take the detection boxes {B1|(x, y, w, h, c, p)} and {B2|(x, y, w, h, c, p)}, and then... Cut into small rectangular pieces Where N is the total number of blocks into which the detection frame is divided, N = h × w / P 2 Then, the data is fed into a deformable Transformer encoder for feature transformation. The specific feature transformation includes the following steps: S21.1 For the small rectangular block features r of the input detection box image, the number of channels is changed from 3 to D through a linear embedding transformation. At this point, the initial input features of the small rectangular block are obtained, denoted as x. i The formula is as follows; S21.2 Due to the parallelism of the Transformer encoder, the initial input features x i Add position encoding e i Finally, the feature representation f of the entire detection box is obtained, as shown in the following formula: S21.3 inputs the obtained features f of the entire detection box into a multi-scale deformable attention system for processing, and then performs processing using a transformer encoder based on a displacement window. i-1 The output of the transformer encoder at layer i-1 is then passed through a normalization layer (LN) and a multi-scale deformable self-attention layer, and then added to itself to obtain the final product. The result After processing by a normalization layer (LN) and a multilayer perceptron (MLP), the result is added to itself to obtain the window-based output f. i The formula is as follows; Where LN represents the normalization layer, MLP represents the multilayer perceptron; MSDA represents deformable self-attention computation based on deformable self-attention, performed between blocks according to formula (4); formula (4) is the formula for the deformable self-attention mechanism, z q Represents the query vector; p q The query vector is represented by its position; x is the input feature; M is the number of attention heads; W k and W v These are the projection matrices for the key and value, respectively; C is the number of channels; Δp mqk It represents the relative position offset in positional encoding; φ represents the softmax function; W m This indicates that a linear transformation is performed on the result after attention is applied to the value to obtain the output results of different heads.

5. A multi-target tracking method based on a multi-scale deformable attention mechanism according to claim 2, characterized in that, The specific method for fusing appearance features and information location attention in step S3 is as follows: S31. First, the position information of the detection box and the tracking box is encoded using a multilayer perceptron to obtain E. p Then, it is added proportionally to the robust pedestrian appearance feature F′ obtained from S2 and combined to obtain the final output feature F″, calculated as follows: F p =MLP([x,y,w,h]) (5) Where MLP stands for Multilayer Perceptron, x is the x-coordinate of the center point of the detection box, y is the y-coordinate of the center point, w is the width of the detection box, and h is the height of the detection box. This indicates element-wise addition. S32. Send the output feature F″ to the spatial attention module SAM to obtain the spatial attention score M. s Then F″ is compared with the spatial attention score M. s Multiply, then enter the normalization layer, and add to F″ to obtain the final feature map F. s ; The specific operation of the spatial attention module is as follows: First, the spatial attention module (SAM) is used to perform global pooling (GP) on the feature map F″ along the channel dimension, thereby compressing the channel dimension. Then, average pooling (AP) is performed on the feature map F″ along the channel dimension, thereby compressing the channel dimension. The two feature maps after compression are then concatenated along the channel dimension to obtain an H×W×2 feature map. Then, a 7x7 convolution operation is performed on the concatenated H×W×2 feature map, followed by an activation function, to obtain the features processed by the spatial attention module. The formula is as follows: M s =σ(conv(concat(Avgpool(F),Avgmax(F)))) (9) In formula (7), F w，h This represents the value of each pixel on the feature map. This indicates that the average value is taken along the channel direction. This indicates taking the maximum value along the channel direction; concat(·) concatenates the feature values ​​along the channel direction; σ(·) represents the activation function, using the sigmoid function to map the function value to the range of 0-1; M s This represents the attention score. F represents element-wise multiplication. s This represents the final feature map; S33, Transfer the final feature map F s The channel attention score M is obtained by sending the data to the channel attention module (CAM). c Then F s With channel attention score M c Multiply, then enter the normalization layer, and with F s The summation yields the final feature map F. c ; The specific operation of the Channel Attention Module (CAM) is as follows: First, the feature map F... s Perform global pooling (GP) along the spatial dimension to compress the spatial size, and then apply it to the feature map F. s Perform average pooling of AP in the spatial dimension to compress the spatial size; Then, the two 1×1×c feature maps are fed into a shared-parameter MLP to learn channel-dimensional features and obtain the importance of each channel. The attention scores of the two channels are then added together to obtain a 1×1×c feature map, which is then passed through the activation function σ(·) to obtain the features after the channel attention module. The formula is as follows: M c (F s )<σ(MLP(AvgPool(F s ))+MLP(Maxpoll(F s ))) (13) In formula (11), F x，y，c Represents the value of each pixel in the feature map This means taking the average value of the spatial dimension WH on each channel c. To represent taking the maximum value of spatial dimension WH on each channel c; MLP represents a multilayer perceptron with shared parameters, σ(·) represents the activation function, using the sigmoid function to map the function value between 0 and 1, M c This represents the attention score. F represents element-wise multiplication. c This represents the final feature map.

6. A multi-target tracking method based on a multi-scale deformable attention mechanism according to claim 5, characterized in that, The specific method for step S4 is as follows: S41. The final feature map F obtained in S3 is... c Place it into the memory module, which is a queue structure, and put F c The features are saved sequentially, a threshold T is set, and the F values ​​from frame tT to t-1 are... c Save it, and when processing frame t, use the features F from tT to t-1 in the memory module. c After processing the image of frame t, the features F of frame t are... c Push it into the queue; simultaneously delete the F of frame tT. c Then, the feature Txd is obtained through the Transformer self-attention encoder. The feature Txd obtained by the autoencoder in the current frame (i.e., the t-th frame) is transformed and then passed through the Transformer decoder to obtain the matching cost body C. S42. The specific steps for transforming the feature Txd obtained from the autoencoder in the current frame (i.e., frame t) are as follows: In frame t, the features from tT to t-1 are fed into the Transformer autoencoder to obtain feature Txd. For the tracking sequence, there are N tracking sequences. The features F of the N tracking sequences are... c The features are then concatenated in the first dimension to obtain a (N×T)×d feature map. For the detection sequence, since there is only one frame of detection boxes, we assume that the number of detections is M. In order to align with the dimension of the tracking sequence, we multiply the detection sequence by N to obtain N×M queries, which are N×M×d detection boxes. The specific process of the tracking sequence entering the Transformer autoencoder is as follows: First, the Txd feature map is projected through the W matrix. q W k and W v We obtain new features Q, K, and V; then we transpose K and multiply it with Q, and divide by the square root of the new feature dimension to ensure scale consistency and obtain the attention score; finally, we scale the attention score to between 0 and 1, and multiply it with the V matrix; then we feed the features into a feedforward neural network FFN, and finally perform layer normalization to obtain the final feature (N×T)×d, as shown in the following formula: Q=FW q ,K=FW k ,V=FW v (15) Among them W q W k and W v Here, Txd is the projection matrix, mapping the Txd dimension to the Txd' dimension. Kt is the transpose of K. The softmax function takes the exponent of each value and then normalizes it, thus establishing a relationship between different values ​​and scaling the scores to the range of 0-1. This indicates element-wise multiplication; S43. The detection box features N×M×d and tracking box features (N×T)×d obtained in S42 are put into the Transformer decoder. After multi-head self-attention and multi-head cross-attention, and after layer normalization and residual connection, the N×M assignment matrix is ​​finally obtained. The rows and columns in the assignment matrix are all one-hot vectors. Because one detection target can only correspond to one tracking target and one tracking target can only correspond to one detection target, the unit element of the assignment matrix is ​​the probability value of 0-1. The assignment matrix is ​​the matching cost body C.

7. A multi-target tracking method based on a multi-scale deformable attention mechanism according to claim 6, characterized in that, The specific method for generating the matching results in step S5 is as follows: S51. After obtaining the matching cost body C, the Hungarian matching algorithm is used to match according to the cost body C to obtain the matching pair P[d, t] of the detection box and the tracking box, the unmatched detection box UD1 and the unmatched tracking box UT1, where d in P[d, t] represents the matched detection box and t represents the matched tracking sequence. S52. Define the detection boxes with p less than 0.5 in S1 as low-confidence detection boxes D2 and unmatched tracking boxes UT. Repeat steps S2, S3, and S4 to obtain the cost body C′. Then, use the Hungarian algorithm to match based on the cost body C′, obtaining the matching pair P[d2, ut] between the low-confidence detection box D2 and the unmatched tracking box UT, the unmatched detection box UD2, and the unmatched tracking box UT2. In P[d2, ut1], d2 represents the matched low-confidence detection box, and ut1 represents the matched tracking sequence. Remove the unmatched tracking box UT2 from T... t-1 The decision to remove these pedestrian targets is based on whether they have left the field of view or are obscured by other pedestrians or the background. Therefore, UT2 is placed into T. lost The condition for deletion is when T lost If a unit in T is stored for more than ∈ frames, T is deleted simultaneously. lost And UT2; compare the unmatched detection boxes UD2 with the threshold ε, and initialize the unmatched detection boxes UD2 with the threshold ε as the new tracking sequence T. new Unmatched detection boxes UD2 with a confidence level less than the threshold ε are considered as non-target foreground or background and are therefore deleted. After carefully screening low-confidence boxes and high-confidence detection boxes, the final tracking box T is obtained. i＝t ={trk|trk∈B}.

8. A multi-target tracking method based on a multi-scale deformable attention mechanism according to claim 7, characterized in that, The specific method for using cross-entropy loss in step S6 to repeat S1-S5 to train the multi-object tracking model is as follows: S61. After obtaining the final tracking result in each training phase, the prediction result T = {trk1, trk2, ..., trk} is calculated. n } and T truth ={trk1, trk2, ..., trk} n The loss for the true values ​​can be viewed as a classification process. Cross-entropy loss is used to train the entire network until a pre-set number of training epochs is reached. The network parameters obtained at this point are the final model parameters. In the inference phase, after obtaining the result from S5, S1, S2, S3, S4, and S5 are performed frame by frame. The final result is the tracking bounding box for the entire video.

Images