A multi-target tracking method with long and short-term target memory

Abstract

The present invention provides a multi-target tracking method with long-term and short-term target memory. Based on the inspiration of the DETR structure, the physical concept of query in DETR is firstly expanded to detection query and tracking query, and the label matching strategy of tracking perception is used to realize the decoupling of detection and tracking tasks; secondly, a memory module is embedded to transfer and update the label matching information between consecutive frames, so as to improve the model's memory ability for long-term and short-term targets, so that the model is sufficient to solve complex time series modeling problems. Under the premise that the first frame image lacks tracking information, the present invention relies on the detection model to complete the positioning of the target and the allocation of the index; for the target detection and tracking of subsequent frames, the memory module can provide effective tracking information, and the confidence entry threshold and the confidence exit threshold can collaboratively screen out high-quality tracking information to complete the spatial modeling of the target trajectory; the present invention significantly improves the multi-target tracking performance in the scenes of irregular motion of the target, motion blur of the target and target occlusion.

Description

Technical Field

[0001] The present invention relates to the general field of image data processing or generation technology, and in particular to a multi-target tracking method with long-term and short-term target memory. Background Art

[0002] The multi-target tracking method aims to solve the problem of predicting the target motion trajectory in the video sequence. The existing methods often adopt the detection-tracking paradigm. First, the target positioning of a single frame is realized through the detection model. Then, based on simple heuristic strategies, such as spatial and appearance similarity, the association between the targets in the previous and next frames is established to predict the target trajectory. Although the above methods have achieved good performance, the heuristic strategies are still not enough to model complex temporal changes, such as irregular motion of the target, motion blur of the target, and occlusion between targets.

[0003] Therefore, in order to improve the versatility of multi-target tracking methods and solve the timing modeling problem of multi-target tracking tasks in different scenarios, the technical field of target tracking methods urgently needs a multi-target tracking method with long and short-term target memory. Summary of the invention

[0004] The present invention aims to solve the versatility of multi-target tracking methods and provides a multi-target tracking method with long-term and short-term target memory. Inspired by the DETR structure, the physical concept of query in DETR is firstly expanded to detection query and tracking query, supplemented by a tracking-aware label matching strategy to achieve the decoupling of detection and tracking tasks; secondly, a memory module is embedded to transfer and update the label matching information between consecutive frames, thereby improving the model's memory ability for long-term and short-term targets, so that the model is sufficient to solve complex time series modeling problems.

[0005] The present invention provides a multi-target tracking method with long-term and short-term target memory, comprising the following steps:

[0006] S1. The images in the video sequence are processed frame by frame by the image preprocessing algorithm of the multi-target tracking model to obtain the preprocessed images:

[0007] Image preprocessing is an image processing process, including image resizing and normalization. The image size is adjusted to meet the model's requirements for the input image size, and normalization improves the model's convergence speed.

[0008] S2, initialization, i=1;

[0009] S3, inputting the i-th frame of the preprocessed image into the feature extraction network of the multi-target tracking model to obtain a multi-scale feature map, fusing the multi-scale feature map with the position information and the feature map level information, and then inputting the multi-scale feature map into the encoder of the multi-target tracking model for multi-scale feature fusion, and then performing confidence prediction and regression prediction. The query includes a detection query and a tracking query. The labels of the detection query and the tracking query are matched through the tracking-aware label matching strategy and used as the supervision information of the multi-target tracking model.

[0010] The feature extraction network aims to obtain multi-scale feature maps of the image and then fuse texture information and semantic information;

[0011] S4, inputting the detection query and the tracking query into the decoder of the multi-target tracking model, and the decoder performs information interaction between the detection query and the tracking query, and between the detection query, the tracking query and the feature maps of different levels;

[0012] A memory module is also embedded in the decoder. The memory module transmits and updates the tracking information between consecutive frames through the confidence entry threshold and the confidence exit threshold, outputs the tracking result of the i-th frame, and provides tracking query information for the next frame of pre-processed image.

[0013] S5, i=i+1, return to step S3, until all pre-processed image target tracking is completed, and a multi-target tracking method with long-term and short-term target memory is completed.

[0014] The multi-target tracking method with long-term and short-term target memory of the present invention is preferably configured such that, in step S1, an image preprocessing algorithm resizes and normalizes the images in the video sequence frame by frame;

[0015] In step S3, the feature extraction network is Resnet50.

[0016] The multi-target tracking method with long and short-term target memory described in the present invention is, as a preferred embodiment, in step S3, the encoder is composed of a deformable convolution module.

[0017] The multi-target tracking method with long-term and short-term target memory of the present invention is preferably configured such that in step S3, the encoder is a Deformable DETR encoder;

[0018] The label matching strategies of tracking perception include consistency matching strategy and Hungarian matching strategy. The consistency matching strategy establishes the temporal, semantic and positional relationship between the targets to be tracked in the video sequence, and the Hungarian matching strategy establishes the relationship between the detection query and the newly appeared target. the connection between;

[0019] The memory module is used to enhance the stability of tracking information about the content.

[0020] In the multi-target tracking method of the long-term and short-term target memory described in the present invention, as a preferred embodiment, in step S3, the label matching strategy of consistency matching is:

[0021] ;

[0022] in, For the The prediction information corresponding to the detection query of the frame matching the target; For the Frame tracking query information, For the Frame tracking query information, , is a union;

[0023] pass and Deliver and update tracking information;

[0024] when When it matches the label information of the i-th frame and the IoU is greater than the consistency matching threshold, Keep the label information of the image after preprocessing the i-th frame consistent, and proceed to step S4;

[0025] when When it does not match the label information of the i-th frame, or the IoU is less than or equal to the consistency matching threshold, the new label information of the i-th frame is , detect the prediction information of the query and Perform Hungarian matching.

[0026] The multi-target tracking method of the long-term and short-term target memory of the present invention is preferably implemented in the Hungarian matching stage as follows:

[0027] ;

[0028] in, For label transfer mode, For the Prediction of frame detection query; For the The new label information of the frame; To establish and The cost matrix between ;

[0029] The Hungarian matching phase is in the search space Find the label transfer method with the minimum cost , after the Hungarian matching is completed, go to step S4;

[0030] The consistency match threshold is 0.5.

[0031] The multi-target tracking method of the present invention with long-term and short-term target memory is preferably configured such that in step S4, the memory module uses the confidence entry threshold to obtain the Filter out the tracking information and assign target IDs, summarize .

[0032] The memory module uses the confidence exit threshold and the continuous K frame conditions to retain the tracking value. A sub-collection of .

[0033] The multi-target tracking method of the present invention with long-term and short-term target memory is preferably configured to retain the prediction confidence corresponding to the tracking query when the prediction confidence corresponding to the tracking query is greater than or equal to the confidence entry threshold. , assign target IDs, and summarize .

[0034] The multi-target tracking method of the present invention with long-term and short-term target memory is preferably configured such that when the prediction confidence corresponding to the tracking query is greater than or equal to the confidence exit threshold, reserve;

[0035] When the prediction confidence corresponding to the tracking query of consecutive K frames is not less than the confidence exit threshold, and the consecutive K frames do not meet the above conditions, it is considered that the tracking target may be blocked, and the tracking is saved. , and follow up later;

[0036] When the prediction confidences corresponding to the tracking queries of K consecutive frames are all less than the confidence exit threshold, and K consecutive frames meet the above conditions, the tracking target is considered lost, and the tracking query is deleted and no longer tracked.

[0037] The multi-target tracking method with long-term and short-term target memory described in the present invention, as a preferred mode, uses the tracking query selected by the collaborative screening of the confidence entry threshold, the confidence exit threshold and the continuous K frame conditions as the memory module network input;

[0038] The memory module includes MHA, FFN and Add&Norm connected in sequence. The output of MHA is also input into Add&Norm. The output of the memory module is .

[0039] The multi-target tracking method of the present invention with long-term and short-term target memory is preferably such that the training set, verification set and test set of the multi-target tracking model are data sets of multi-target tracking in natural scenes; the verification stage and the test stage use memory modules with the same network structure;

[0040] The decoder is the Deformable DETR decoder, which includes N self-attention modules and N cross-attention modules;

[0041] The loss function of the multi-target tracking model is:

[0042] ;

[0043] in, For the The target number of frames; To track losses, To detect losses; and It consists of two parts: bounding box regression loss and classification loss;

[0044] When the multi-target tracking model is trained, verified and tested using the training set, validation set and test set respectively, and the validation result and the test result reach the target value, a trained multi-target tracking model is obtained.

[0045] The present invention has the following advantages:

[0046] The present invention provides a multi-target tracking method with long-term and short-term target memory. The pictures in the input video sequence are sequentially subjected to a preprocessing algorithm to complete the preprocessing of the video sequence. Under the premise that the first frame of the picture lacks effective tracking information, the detection model is relied upon to complete the positioning of the target and the allocation of the index. For the target detection and tracking of subsequent frames, the memory module can provide effective tracking information, and the confidence entry threshold and the confidence exit threshold can collaboratively screen out high-quality tracking information to complete the spatial modeling of the target trajectory. Compared with the previous multi-target tracking methods, the multi-target tracking method with long-term and short-term target memory of the present invention can significantly improve the multi-target tracking performance in the scenes of irregular motion of the target, motion blur of the target, and target occlusion, and has certain practical use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a multi-target tracking method with long and short-term target memory;

[0048] Figure 2 A flowchart of the network structure of a memory module of a multi-target tracking method with long and short-term target memory;

[0049] Figure 3A multi-target tracking model network structure flow chart of a multi-target tracking method with long and short-term target memory;

[0050] Figure 4 Establish a flow chart for a multi-target tracking model of a multi-target tracking method with long and short-term target memory;

[0051] Figure 5 This is a tracking effect diagram of a multi-target tracking method with long and short-term target memory. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Example 1

[0053] A multi-target tracking method with long and short-term target memory, such as Figure 1 As shown, the specific implementation steps are as follows:

[0054] The following steps are involved:

[0055] S1. The images in the video sequence are processed frame by frame by the image preprocessing algorithm of the multi-target tracking model to obtain the preprocessed images:

[0056] S2, initialization, i=1;

[0057] S3. Input the i-th frame of the preprocessed image into the encoder of the Deformable DETR of the multi-target tracking model to obtain a multi-scale feature map. The query includes detection query and tracking query. The complex temporal modeling problem in the video sequence is solved through the tracking-aware label matching strategy, so that the proposed model can perform long-term and short-term target memory. The tracking-aware label matching strategy includes Hungarian matching and consistency matching. Hungarian matching aims to establish a minimum cost between the detection query and the newly appeared target. The consistency matching aims to establish the temporal, semantic and positional relationships between the objects to be tracked in the video sequence;

[0058] S4, input the initialization information (position information) of the detection query and the tracking query into the decoder of Deformable DETR, and embed a memory module in the decoder. The memory module transmits and updates the tracking information between consecutive frames through the confidence entry threshold and the confidence exit threshold, outputs the tracking result of the i-th frame, and provides the tracking query information for the next frame of preprocessed image;

[0059] S5, i=i+1, return to step S3, until all pre-processed image target tracking is completed, and a multi-target tracking method with long-term and short-term target memory is completed.

[0060] Tracking-aware tag matching strategies include:

[0061] 1) Hungarian matching, which establishes the connection between the detection query and the emerging target through the minimum cost. There are two sources:

[0062] (1) No. Label information for the first appearance of a frame;

[0063] (2) No. Frame tracking query With The label information of the frames matches, but the intersection-over-union (IoU) is less than 0.5;

[0064] Hungarian matching can be expressed as:

[0065] (1)

[0066] In the above formula, For the Prediction of frame detection query; For the The new label information of the frame; Establish and The cost matrix between them; the Hungarian matching algorithm aims to search the space Find the label transfer method with the minimum cost ;

[0067] 2) Consistency matching aims to establish the temporal, semantic and positional relationships between the targets to be tracked in the video sequence. The way the tracking query information is updated maintains the temporal relationship of the tracking target, and the tracking query explicitly maintains the content and position information related to the target, so that the model has long-term and short-term memory capabilities and can solve complex temporal modeling problems. Specifically, it can be expressed as follows:

[0068] (2)

[0069] In the above formula, For the The prediction information corresponding to the detection query of the frame matching the target; For the Frame tracking query information, where , since there is no tracking target in the first frame, pass and Deliver and update tracking information; With The matching of frame tag information will adopt a consistent matching strategy. Specifically, and If the frame label information matches and the intersection-over-union (IoU) is greater than 0.5, then Keep consistent with the label information of the i-th frame, is a union;

[0070] First, consistency judgment is performed, and then Hungarian matching is performed; the purpose is to use Hungarian matching to achieve label matching for newly appeared objects based on the label matching results of the tracking query.

[0071] Label information means that the dataset will label each target with an ID, that is, different targets have different IDs. Frame tracking query With The label information of the frame matches, which refers to the Frame tracking query With A consistent set of target IDs for frames.

[0072] The tracking query information update method maintains the temporal relationship of the tracking target, and the tracking query explicitly maintains the content and location information related to the target, so that the model has long-term and short-term memory capabilities and can solve complex temporal modeling problems, first performing consistency judgment and then performing Hungarian matching;

[0073] The memory module uses the Transformer structure to enhance the stability of the content of the tracking information. The network structure diagram of the memory module is as follows: Figure 2 As shown;

[0074] The memory model is embedded in the training phase, verification phase, and test phase. In the training phase, the memory module aims to improve the expression ability of the tracking information; the same network structure is used in the verification phase and the test phase, aiming to search for high-quality tracking information, improve the computational efficiency of the model, and reduce the memory consumption of the model. The network structure is as follows: Figure 2 As shown, using the confidence entry threshold from The tracking information is selected and retained if it is higher than the confidence threshold, otherwise it is deleted. The cooperation of the confidence exit threshold and the continuous K-frame condition is aimed at filtering out invalid tracking information, thereby improving the quality of model information interaction, improving the model's computational efficiency and reducing the model's video memory consumption;

[0075] The embodiment of consistent matching in the inference phase is shown in the figure above. In the inference phase, the query is tracked With the target ID, the confidence exit threshold and continuous K frame conditions are used to retain the target with tracking value. The subset of Select potential tracking targets and assign target IDs, and summarize them into .

[0076] The confidence entry threshold refers to the inference stage, which measures the hyperparameter of whether the detection query becomes the tracking query of the next frame according to the prediction confidence corresponding to the detection query. For example, if the confidence entry threshold is 0.7, then if the prediction confidence corresponding to the detection query is greater than or equal to 0.7, we consider it to be the tracking query of the next frame; otherwise, it cannot be the tracking query of the next frame;

[0077] The confidence exit threshold refers to the hyperparameter of the tracking query in the inference stage, which is measured by the prediction confidence corresponding to the tracking query to determine whether it will become the tracking query of the next frame. For example, if the confidence exit threshold is 0.5, then if the prediction confidence corresponding to the tracking query is greater than or equal to 0.5, we consider it to be the tracking query of the next frame; otherwise, we proceed to the judgment of K consecutive frames;

[0078] The purpose of "cooperation between confidence exit threshold and continuous K-frame condition" is to avoid the loss of the tracking target during the tracking process. For example, if the confidence exit threshold is 0.5, the prediction confidence corresponding to the tracking query is less than 0.5, and the continuous K-frames are the same as above, then it is considered that the tracking target is lost, and the tracking query is deleted and no longer tracked; otherwise, if the continuous K-frames are the same as above, it is considered that the tracking target may be blocked, and the tracking query is saved and tracked later.

[0079] After the confidence level is less than the exit threshold, the continuous K frames are judged.

[0080] The confidence entry threshold, confidence exit threshold and continuous K frame conditions work together to select potential tracking queries for the next frame image. The content information and location information of the tracking query are used as part of the network input.

[0081] MHA (Multi Head Attention), which is the core component of Transformer;

[0082] FFN (Feed-Forward Neural Network) feed-forward neural network;

[0083] Add&Norm refers to residual connection and layer normalization;

[0084] The above architecture enables the Transformer model to first perform a self-attention mechanism, then perform feedforward neural network processing, and perform residual connections and layer normalization; this design enables the Transformer model to effectively process long sequence data.

[0085] The input is the content information and location information of the detection query and the tracking query, and the output is the content information, location information and confidence information of the prediction query corresponding to the detection query and the tracking query.

[0086] The confidence entry threshold, confidence exit threshold and continuous K frame conditions work together to select potential tracking queries for the next frame image. The content information and location information of the tracking query are used as part of the network input.

[0087] The overall structure of the multi-target tracking model is shown in Figure 3 As shown, the specific process can be broken down into:

[0088] (1) The images in the video sequence are sequentially passed through the feature extraction network Resnet50 to obtain multi-scale feature maps;

[0089] Multi-scale feature maps specifically refer to image feature maps of different resolutions (different scales) obtained after the image passes through the convolution layer. The purpose is to improve the model's ability to aggregate high-level semantic information and low-level texture information;

[0090] (2) Inspired by the Deformable DETR structure, deformable convolution is used to improve the ability to fuse information of feature maps of different levels and improve the representation ability of features;

[0091] Multi-scale deformable convolution uses a network structure similar to Transformer, with detection query and tracking query as query, and multi-scale image features as key and value; it aims to associate the image features of interest to the query.

[0092] (3) A two-stage tracking paradigm is adopted. The label matching of the detection query and the tracking query is completed through the collaboration of the Hungarian algorithm and the consistency matching algorithm. The label matching of the tracking query is completely dependent on the consistency matching algorithm. The first stage refers to the label matching of the tracking query completely relying on the consistency matching algorithm. The second stage completes the label matching of the detection query through the Hungarian algorithm.

[0093] (4) Using N consecutive attention modules and cross attention modules as decoders, it provides opportunities for information exchange between queries and between queries and feature maps of different levels, and provides tracking query information for the next frame;

[0094] (5) Establish a loss function based on the label matching results and train the model through the loss;

[0095] Confidence prediction and regression prediction are the model's predictions of target category and location. They are not used as the network structure of the memory module, but are the input information of the memory module. The first stage and the second stage specifically refer to the two-stage tracking paradigm. The first stage refers to the label matching of the tracking query that completely relies on the consistency matching algorithm, and the second stage completes the label matching of the detection query through the Hungarian algorithm.

[0096] The loss function of the tracking task needs to establish a model for the video sequence. The label information of the continuous N-frame video sequence can be represented as a set ,in and Represent the sets of newly appeared targets and already appeared targets respectively; the prediction information of a continuous N-frame video sequence can be represented as a set ,in and They represent the prediction information of the detection query and the tracking query respectively. The loss function can be expressed as:

[0097] (3)

[0098] In the above formula, Indicates that it is The target number of frames; and It consists of two parts: bounding box regression loss and classification loss;

[0099] The bounding box regression loss and classification loss depend on the detection query and tracking query. Based on the collaboration of the confidence entry threshold, confidence exit threshold and continuous K frame conditions, the target is tracked in a video sequence. The target tracking is to accurately predict the category and position of the target to be tracked and to associate the ID.

[0100] like Figure 4 As shown, the method for establishing multi-target tracking includes the following steps: Step 1, establish a data set for multi-target tracking in natural scenes, and divide it into a training set, a validation set, and a test set; Step 2, formulate a label matching strategy for tracking perception; Step 3, build a memory module; Step 4, establish a model; Step 5, establish a loss function; Step 6, use the training set, validation set, and test set to train, verify, and test the model respectively to obtain a multi-target tracking model.

[0101] like Figure 5The figure shows the effect of the multi-target tracking method with long and short-term target memory. Each row represents a video sequence, and the horizontal axis represents the timeline. The multi-target tracking model can realize the tracking of video vehicles, and different targets are represented by boxes of different grayscales.

[0102] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multi-target tracking method with long-term and short-term target memory, characterized in that: The following steps are involved: S1, the pictures in the video sequence are processed frame by frame by the image preprocessing algorithm of the multi-target tracking model to obtain the preprocessed pictures; S2, initialization, i=1; S3, inputting the i-th frame of the preprocessed image into the feature extraction network of the multi-target tracking model to obtain a multi-scale feature map, fusing the multi-scale feature map with position information and feature map level information, and then inputting the multi-scale feature map into the encoder of the multi-target tracking model for multi-scale feature fusion, and then performing confidence prediction and regression prediction, the query includes a detection query and a tracking query, and the label matching of the detection query and the tracking query is performed through a tracking-aware label matching strategy and used as the supervision information of the multi-target tracking model; S4, inputting the detection query and the tracking query into the decoder of the multi-target tracking model, and the decoder performs information exchange between the detection query and the tracking query, and between the detection query, the tracking query and the feature maps of different levels; The decoder is also embedded with a memory module, which transmits and updates the tracking information between consecutive frames through the confidence entry threshold and the confidence exit threshold, outputs the tracking result of the i-th frame, and provides tracking query information for the next frame of pre-processed picture; S5, i=i+1, return to step S3, until all the pre-processed image target tracking is completed, and a multi-target tracking method with long-term and short-term target memory is completed.

2. The multi-target tracking method of long-term and short-term target memory according to claim 1, characterized in that: In step S1, the image preprocessing algorithm resizes and normalizes the images in the video sequence frame by frame; In step S3, the feature extraction network is Resnet50.

3. The multi-target tracking method of long-term and short-term target memory according to claim 1, characterized in that: In step S3, the encoder is composed of a deformable convolution module; The tracking-aware label matching strategy includes a consistency matching strategy and a Hungarian matching strategy. The consistency matching strategy establishes the temporal, semantic and positional relationship between the targets to be tracked in the video sequence, and the Hungarian matching strategy establishes the detection query and the newly appeared target. the connection between; The memory module is used to enhance the stability of the tracking information regarding the content.

4. The multi-target tracking method of long-term and short-term target memory according to claim 3, characterized in that: The encoder is a Deformable DETR encoder; The consistency matching strategy is: ； in, For the The prediction information corresponding to the detection query of the frame matching the target; For the Frame tracking query information, For the Frame tracking query information, , is a union; pass and Deliver and update tracking information; when When it matches the label information of the i-th frame and the IoU is greater than the consistency matching threshold, Keep the label information of the image after preprocessing the i-th frame consistent, and proceed to step S4; when When the label information of the i-th frame does not match, or the IoU is less than or equal to the consistency matching threshold, the new target in the i-th frame is , detect the prediction information of the query and Perform Hungarian matching.

5. The multi-target tracking method of long-term and short-term target memory according to claim 4, characterized in that: The method of the Hungarian matching phase is: ； in, For label transfer mode, For the Prediction of frame detection query; To establish and The cost matrix between ; The Hungarian matching phase searches for Find the label transfer method with the minimum cost , after the Hungarian matching is completed, go to step S4; The consistency matching threshold is 0.

5.

6. The multi-target tracking method of long-term and short-term target memory according to claim 5, characterized in that: In step S4, the memory module uses the confidence entry threshold from Filter out the tracking information and assign target IDs, summarize ; The memory module uses the confidence exit threshold and the continuous K frame condition to synergistically retain the tracking-worthy A sub-collection of .

7. The multi-target tracking method of long-term and short-term target memory according to claim 6, characterized in that: When the prediction confidence corresponding to the tracking query is greater than or equal to the confidence entry threshold, retain , assign target IDs, and summarize .

8. The multi-target tracking method of long-term and short-term target memory according to claim 6, characterized in that: When the prediction confidence corresponding to the tracking query is greater than or equal to the confidence exit threshold, reserve; If the prediction confidence corresponding to the consecutive K-frame tracking query is not less than the confidence exit threshold, it is considered that the tracking target may be blocked and the result is saved. , and follow up later; When the prediction confidences corresponding to the tracking queries of K consecutive frames are all less than the confidence exit threshold, it is considered that the tracking target is lost, and the tracking query is deleted and no tracking is performed subsequently.

9. The multi-target tracking method of long-term and short-term target memory according to claim 8, characterized in that: The tracking query selected by the confidence entry threshold, confidence exit threshold and continuous K-frame conditions is used as the input of the memory module; The memory module includes MHA, FFN and Add&Norm connected in sequence, and the output of MHA is also input to Add&Norm at the same time. .

10. The multi-target tracking method of long-term and short-term target memory according to claim 1, characterized in that: The training set, validation set and test set of the multi-target tracking model are data sets for multi-target tracking in natural scenes; the memory module with the same network structure is used in the validation stage and the test stage; The decoder is a Deformable DETR decoder, and the decoder includes N self-attention modules and N cross-attention modules; The loss function of the multi-target tracking model is: ； in, For the The target number of frames; To track losses, To detect losses; and Both consist of two parts: bounding box regression loss and classification loss; When the multi-target tracking model is trained, verified and tested using the training set, the validation set and the test set respectively, and the validation result and the test result reach the target value, a trained multi-target tracking model is obtained.

Images