A target tracking model training method, a multi-target video labeling method and device

Abstract

The application relates to a target tracking model training method and a multi-target video labeling method and device, and belongs to the technical field of computer vision, which comprises the following steps: a feature extraction module is used for frame-by-frame sampling of a video to be labeled to obtain an i-th frame image; prompt text description is generated according to an input to-be-labeled category; the prompt text description and the i-th frame image are subjected to feature extraction processing to obtain a text feature vector and an image feature vector; the above vectors are input into a target detection module for processing to obtain an i-th detection query vector; an i-1 tracking query vector and the i-th detection query vector are processed by a tracking module to obtain a detection result of the i-th frame image; the above steps are repeated until the last frame of the video to be labeled, so that pre-labeling of the video to be labeled is completed. The multi-target in the video is labeled in a manner that the prompt text is combined with the target, and the problems of missed detection of the target or inaccurate entity category recognition are reduced.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method for training a target tracking model, a method and apparatus for annotating multi-target videos. Background Technology

[0002] As the number of parameters in deep learning models increases, the amount of training data required also grows. Especially in the era of large-scale models, hundreds of millions of parameters require equally hundreds of millions of data points to support them. Currently, most tasks still rely on supervised learning with labeled data for training, making the quality and scale of labeled datasets a bottleneck restricting the development of large-scale models.

[0003] Building a high-quality, large-scale professional domain dataset entirely through manual data annotation is no easy task: training annotators and manual annotation are costly and time-consuming. This is especially true for video annotation tasks, where different sub-scenarios have varying labeling requirements, leading to a heavy workload. While video annotation tools such as CVAT and darklabel have some built-in intelligent functions, they still require manual activation or rely on human cognition, potentially causing omissions of target data. Summary of the Invention

[0004] In view of this, it is necessary to provide a target tracking model training method, a multi-target video annotation method and device to solve the problems of high cost and long time consumption of existing manual annotation; and the fact that existing video annotation tools require human warm-up or rely on human cognition, which may cause the omission of targets.

[0005] To address the above problems, this invention provides a target tracking model training method, comprising: building a target tracking model;

[0006] The video sample dataset of the multi-target tracking task is input into the feature extraction module of the target tracking model, and frame-by-frame sampling is performed to obtain a preset number of original sample image frames.

[0007] The multiple target category labels in the i-th frame of the original sample image are concatenated and spliced ​​together to form the i-th prompt text description;

[0008] The i-th frame image is encoded using a visual feature extractor to obtain the feature vector of the i-th frame image;

[0009] The i-th prompt text description is encoded using a language feature extractor to obtain the text feature vector of the i-th frame;

[0010] The target detection module of the target tracking model processes the feature vector of the i-th frame image and the feature vector of the i-th frame text to obtain the detection query vector of the i-th frame.

[0011] The detection query vector of the (i-1)th frame of the image is input into the tracking module of the target tracking model for processing to obtain the (i-1)th tracking query vector;

[0012] The tracking module of the target tracking model processes the (i-1)th tracking query vector and the i-th frame detection query vector to obtain the detection result of the i-th frame image;

[0013] Repeat the above steps starting from the first frame of the video sample dataset until the last frame of the video sample dataset to obtain the detection results for each frame of the video sample dataset;

[0014] The loss parameters of the target tracking model are calculated based on the predicted bounding box category feature vector and the corresponding category feature vector label in the detection results of each frame of the video sample dataset, as well as the predicted bounding box coordinates and the corresponding coordinate labels in the detection results of each frame of the video sample dataset.

[0015] The target tracking model to be trained is trained according to the loss parameters to obtain the trained target tracking model.

[0016] Optionally, the step of calculating the loss parameters of the target tracking model based on the predicted bounding box category feature vector and corresponding category feature vector label in the detection results of each frame of the video sample dataset, and the predicted bounding box coordinates and corresponding coordinate labels in the detection results of each frame of the video sample dataset, specifically includes:

[0017] Based on the matching results of the detection results and the true results of each frame of image, the classification loss and coordinate regression loss are calculated.

[0018] The loss parameters of the target tracking model are calculated based on the classification loss, the coordinate regression loss, the predicted bounding box category feature vector and the corresponding category feature vector label in the detection results of each frame of the image, and the predicted bounding box coordinates and the corresponding coordinate labels in the detection results of each frame of the image.

[0019] To address the above problems, the present invention also provides a multi-target video annotation method, comprising:

[0020] The video to be labeled and the category to be labeled are input into the trained target tracking model; wherein, the trained target tracking model is trained according to one of the target tracking model training methods in claim 1 or 2;

[0021] The video is sampled frame by frame by the feature extraction module of the trained target tracking model to obtain the original image frames.

[0022] Generate a prompt text description based on the category to be labeled;

[0023] The prompt text description and the i-th frame image from the original image frame are respectively input into the feature extraction module for feature extraction processing to obtain the text feature vector and the image feature vector; where i≥1;

[0024] The text feature vector and the image feature vector are input into the target detection module of the trained target tracking model for processing to obtain the i-th detection query vector.

[0025] The detection query vector of the (i-1)th frame of the image is input into the tracking module of the target tracking model for processing to obtain the (i-1)th tracking query vector;

[0026] The tracking module of the trained target tracking model processes the (i-1)th tracking query vector and the ith detection query vector to obtain the detection result of the ith frame image.

[0027] Starting from the first frame of the video to be labeled, repeat the above steps until the last frame of the video to be labeled to complete the pre-labeling of the video.

[0028] Optionally, the target tracking model includes: the feature extraction module, the target detection module, and the tracking module;

[0029] The feature extraction module includes: a visual feature extractor and a language feature extractor;

[0030] The prompt text description and the i-th frame image from the original image frame are respectively input into the feature extraction module for feature extraction processing to obtain text feature vectors and image feature vectors, specifically including:

[0031] The prompt text description is input into the language feature extractor for language feature extraction to obtain a text feature vector;

[0032] The i-th frame of the original image frame is input into the visual feature extractor for image feature extraction to obtain an image feature vector; wherein the weights of the visual feature extractor and the language feature extractor are fixed values.

[0033] Optionally, the target detection module includes: a detection encoder, a query selection module, and a detection decoder;

[0034] The step of inputting the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain the i-th detection query vector specifically includes:

[0035] The text feature vector and the image feature vector are input into the detection encoder for feature fusion processing to obtain the processed text feature vector and the processed image feature vector;

[0036] The processed text feature vector and the processed image feature vector are input into the query selection module for processing to obtain the first query vector;

[0037] The first query vector is input into the detection decoder for modality alignment processing to obtain the i-th detection query vector.

[0038] Optionally, the detection encoder includes: a first deformable self-attention module, a first self-attention module, an image-text cross-attention module, a text-image cross-attention module, and an encoding feedforward network layer;

[0039] The step of inputting the text feature vector and the image feature vector into the detection encoder for feature fusion processing to obtain the processed text feature vector and the processed image feature vector specifically includes:

[0040] The text feature vector is input into the first self-attention module for processing to obtain the first encoding processing result;

[0041] The image feature vector is input into the first deformable self-attention module for processing to obtain the second encoding processing result;

[0042] The first encoding result is used as K and V, and the second encoding result is used as Q. These are input into the image-text cross-attention module for processing to obtain the processed K, V, and Q.

[0043] The processed K, V, and Q are input into the text-image cross-attention module for further processing to obtain the third encoding result;

[0044] The third encoding processing result is input into the encoding feedforward network layer for processing to obtain the processed text feature vector and the processed image feature vector.

[0045] Optionally, the detection decoder includes: a self-attention layer, an image cross-attention layer, a text cross-attention layer, and a decoding feedforward network layer;

[0046] The step of inputting the first query vector into the detection decoder for modality alignment processing to obtain the i-th detection query vector specifically includes:

[0047] The first query vector is input into the self-attention layer for processing to obtain the first decoding result;

[0048] The processed image feature vectors are used as K and V, and the first decoding result is used as Q. These are then input into the image cross-attention layer for processing to obtain the second decoding result.

[0049] The processed text feature vectors are used as K and V, and the second decoding result is used as Q. These are then input into the text cross-attention layer for processing to obtain the third decoding result.

[0050] The third decoding result is input into the decoding feedforward network layer for processing to obtain the i-th detection query vector.

[0051] Optionally, the tracking module includes: a second self-attention module, a second deformable self-attention module, and a region multilayer perceptron;

[0052] The tracking module of the trained target tracking model processes the (i-1)th tracking query vector and the ith detection query vector of the (i-1)th frame image to obtain the detection result of the ith frame image, specifically including:

[0053] The (i-1)th tracking query vector and the ith detection query vector of the (i-1)th frame image are input into the tracking encoder for splicing processing to obtain the splicing result;

[0054] The stitching result is input into the second self-attention module to obtain the first tracking result;

[0055] The first tracking result and the i-th detection query vector are input into the second deformable self-attention module for processing to obtain the second tracking result;

[0056] The second tracking result is input into the region multilayer perceptron for processing to obtain the detection result of the i-th frame image.

[0057] Optionally, it also includes:

[0058] The trained target tracking model is used to annotate targets in multi-target autonomous driving videos to obtain a video sample dataset.

[0059] The video sample dataset is added to the driving video dataset used to train the autonomous driving model to obtain the updated driving video dataset;

[0060] The autonomous driving model is trained based on the updated driving video dataset to obtain the trained autonomous driving model.

[0061] The present invention also provides a target tracking model training device, comprising: a model building module, a training sampling module, a training stitching module, a visual feature extraction module, a language feature extraction module, a training detection module, a training tracking module, a loop training module, a training calculation module, and a training optimization module;

[0062] The model building module is used to build a target tracking model;

[0063] The training sampling module is used to input the video sample dataset of the multi-target tracking task into the feature extraction module of the target tracking model, perform frame-by-frame sampling, and obtain a preset number of original sample image frames.

[0064] The training stitching module is used to concatenate and stitch multiple target category labels in the i-th frame of the original sample image frame into the i-th prompt text description;

[0065] The visual feature extraction module is used to encode the i-th frame image through a visual feature extractor to obtain the feature vector of the i-th frame image;

[0066] The language feature extraction module is used to encode the i-th prompt text description through a language feature extractor to obtain the text feature vector of the i-th frame.

[0067] The training detection module is used to process the image feature vector and the text feature vector of the i-th frame through the target detection module of the target tracking model to obtain the detection query vector of the i-th frame;

[0068] The training tracking module is used to input the detection query vector of the (i-1)th frame of the (i-1)th frame image into the tracking module of the target tracking model for processing to obtain the (i-1)th tracking query vector; then the tracking module of the target tracking model processes the (i-1)th tracking query vector and the detection query vector of the i-th frame to obtain the detection result of the i-th frame image.

[0069] The loop training module is used to repeat the above steps from the first frame of the video sample dataset until the last frame of the video sample dataset, so as to obtain the detection results of each frame of the video sample dataset.

[0070] The training calculation module is used to calculate the loss parameters of the target tracking model based on the predicted bounding box category feature vector and corresponding category feature vector label in the detection results of each frame of the video sample dataset, and the predicted bounding box coordinates and corresponding coordinate labels in the detection results of each frame of the video sample dataset.

[0071] The training optimization module is used to train the target tracking model to be trained according to the loss parameters, so as to obtain the trained target tracking model.

[0072] By applying the technical solution provided in the embodiments of this application, multimodal fusion and learning are achieved through comparative training between language and vision. This synergistic effect enables the model to learn visual representations of language perception, better guiding object detection and association tasks. Through the model's object detection and tracking modules, complex object features can be better captured and expressed, making it easier for the model's network to learn effective feature representations, thereby improving convergence speed and overall performance. This allows for rapid optimization and improves model iteration efficiency.

[0073] This invention combines prompt text with the target to obtain a target tracking solution with strong scalability and high accuracy. When using the model to annotate multiple targets in a video, the detection results of possible targets in each frame can be output based on the required annotated prompt phrases and video. This reduces the increase in manual correction workload caused by missed target detection or inaccurate entity category recognition, and obtains more high-quality labeled data. Attached Figure Description

[0074] Figure 1 This is a flowchart illustrating the steps of a target tracking model training method provided in an embodiment of the present invention;

[0075] Figure 2 This is a schematic diagram of the structure of a vision-based multi-target tracking model provided in an embodiment of the present invention;

[0076] Figure 3 This is a flowchart of the steps of a multi-target video annotation method provided in an embodiment of the present invention;

[0077] Figure 4 This is a flowchart of the steps of the visual feature extraction and language feature extraction methods provided in the embodiments of the present invention;

[0078] Figure 5 This is a flowchart of the processing steps of the target detection module provided in this embodiment of the invention;

[0079] Figure 6 This is a schematic diagram of the target detection module structure provided in an embodiment of the present invention;

[0080] Figure 7 This is a flowchart of the steps involved in the encoder detection process provided in this embodiment of the invention;

[0081] Figure 8 This is a flowchart of the processing steps of the detection decoder provided in this embodiment of the invention;

[0082] Figure 9 This is a flowchart of the processing steps of the tracking module provided in this embodiment of the invention;

[0083] Figure 10 This is a schematic diagram of the tracking module structure provided in an embodiment of the present invention;

[0084] Figure 11 This is a structural block diagram of a target tracking model training device provided in an embodiment of the present invention. Detailed Implementation

[0085] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0086] Reference Figure 1 The diagram illustrates a flowchart of a target tracking model training method, which can be applied to computer vision, such as... Figure 1 As shown, the specific steps may include:

[0087] Step S101: Build a target tracking model;

[0088] In this embodiment, a target tracking model is built using a feature extraction model, a target detection model, and a tracking module.

[0089] Specifically, the feature extraction model can consist of a dual encoder of ViT-L and BERT, with model weights frozen; the object detection model can be Grounding Dino 1.5 Pro, specifically including an encoder, a query selection module, and a decoder, with weights initialized through pre-training; the tracking module can be the MOTR tracker architecture in MOTRv2, specifically including a decoder that updates the loss parameters based on the real-time training process.

[0090] Step S102: Input the video sample dataset of the multi-target tracking task into the feature extraction module of the target tracking model, perform frame-by-frame sampling, and obtain a preset number of original sample image frames;

[0091] In this embodiment, the feature extraction module includes: a visual feature extractor. and language feature extractor ;

[0092] Step S103: Concatenate multiple target category labels in the i-th frame of the original sample image frame to form the i-th prompt text description;

[0093] In this embodiment, during the training process, each image contains a corresponding category label for the target. The category labels in the images are concatenated to obtain the prompt text description. Each image may contain at least one category label.

[0094] Step S104: Encode the i-th frame image using a visual feature extractor to obtain the feature vector of the i-th frame image;

[0095] In this embodiment, the i-th frame image from the original image frames is input into the visual feature extractor. Image feature extraction is performed to obtain feature vector image encoding. ;

[0096] Step S105: Encode the i-th prompt text description using a language feature extractor to obtain the text feature vector of the i-th frame;

[0097] In this embodiment, the prompt text description is input into a language feature extractor for language feature extraction to obtain the feature vector text encoding. Among them, visual feature extractor and language feature extractor The weights are fixed values.

[0098] Step S106: The target detection module of the target tracking model processes the feature vector of the i-th frame image and the feature vector of the i-th frame text to obtain the detection query vector of the i-th frame;

[0099] In this embodiment, the text feature vector and image feature vectors The input detector encoder performs feature fusion processing to obtain the processed text feature vector. and processed image feature vectors The processed text feature vector and processed image feature vectors The input is processed by the query selection module to obtain the first query vector. The first query vector is then input to the detection decoder for modality alignment to obtain the i-th detection query vector. .

[0100] Step S107: Input the detection query vector of the (i-1)th frame of the (i-1)th frame image into the tracking module of the target tracking model for processing to obtain the (i-1)th tracking query vector;

[0101] In this embodiment, the (i-1)th detection query vector The input is processed in the tracking module of the target tracking model to obtain the (i-1)th tracking query vector. .

[0102] Step S108: The tracking module of the target tracking model processes the (i-1)th tracking query vector and the i-th frame detection query vector to obtain the detection result of the i-th frame image; the tracking module includes: a tracking encoder;

[0103] In this embodiment, the (i-1)th tracking query vector and the i-th detection query vector The data is input into the tracking encoder for splicing processing to obtain the splicing result. ;

[0104] The stitching result is processed by the tracking encoder. The detection results are obtained by performing tracking processing to obtain the detection results of the i-th frame image.

[0105] Step S109: Starting from the first frame of the video sample dataset, repeat the above steps until the last frame of the video sample dataset to obtain the detection results of each frame of the video sample dataset;

[0106] Step S110: Calculate the loss parameters of the target tracking model based on the predicted bounding box category feature vector and corresponding category feature vector label in the detection results of each frame of the video sample dataset, and the predicted bounding box coordinates and corresponding coordinate labels in the detection results of each frame of the video sample dataset.

[0107] Step S111: Train the target tracking model to be trained according to the loss parameters to obtain the trained target tracking model.

[0108] For example, the detection result of the previous frame is used as the tracking query vector. This is input into the current frame for predicting the trajectory of the detected target. It is compared with the detection results of the current frame. The concatenated vectors are input into the decoder as the initial query vector. As shown in Formula 1.

[0109] (Formula 1),

[0110] Aggregated vector The algorithm employs both a self-attention module and a deformable self-attention module. The self-attention module further integrates historically tracked targets and the currently detected target, while the deformable self-attention module recalibrates the target coordinate information to model the target features and trajectory of the query vector. Simultaneously, it relocalizes old targets and determines the iterative changes in target appearance and disappearance. Finally, it outputs the detection results of all current frames, which are used as the tracking vector for the next frame. The detection results are then processed by a region head to output the visual features of the predicted bounding boxes. With coordinate information As shown in Equation 2, the Hungarian matching algorithm is used to match the predicted boxes with the ground truth boxes. For each predicted box, classification loss and coordinate regression loss are calculated, as shown in Equation 3, and parameters are updated. The classification loss uses focal loss to optimize the results of cross-modal contrastive learning fused by dot product, where... The dot product combines visual and linguistic feature vectors. The visual and linguistic feature similarity for annotation is shown in Equation 4; the bounding box uses L1 loss and GIoU loss as losses. To label coordinate information, and Contains the same elements Where x and y are the coordinates of the center point of the rectangle, and w and h are the width and length of the rectangle. L1 loss is used to calculate the absolute error of the coordinates, and GIoU loss is used to calculate the error of the overlapping area of ​​the target boxes.

[0111] (Formula 2),

[0112] (Formula 3),

[0113] (Formula 4),

[0114] in, Visual features of the predicted bounding box representing the detection result. Coordinate information of the detection results express Loss term weights, GIoU loss term weights, The dot product combines visual and linguistic similarity vectors. For the labeled visual and linguistic feature similarity vectors, This is the text feature vector.

[0115] For example, refer to Figure 2As shown, a schematic diagram of a vision-based multi-target tracking model structure is illustrated. Figure 2 As shown, it can specifically include: a target tracking model comprising a dual encoder module, a target detection module, and a tracking module; the dual encoder module includes: a visual feature extractor. and language feature extractor The dual encoder module uses pre-trained ViT-L and BERT as feature extraction models for images and text, respectively. The object detection module includes an encoder and a decoder; it employs the open-set object detection model Grounding Dino 1.5 Pro, initialized with pre-trained weights that are updated continuously during training. The tracking module uses the MOTR tracker architecture from MOTRv2.

[0116] In one example, classification loss and coordinate regression loss are calculated based on the matching results of the detection results and the true results of each frame of the image.

[0117] The loss parameters of the target tracking model are calculated based on the classification loss, the coordinate regression loss, the predicted bounding box category feature vector and the corresponding category feature vector label in the detection results of each frame of the image, and the predicted bounding box coordinates and the corresponding coordinate labels in the detection results of each frame of the image.

[0118] In one example, refer to Figure 3 As shown, a flowchart illustrating the steps of a multi-target video annotation method is presented, such as... Figure 3 As shown, the specific steps may include:

[0119] Step S301: Input the video to be labeled and the category to be labeled into the trained target tracking model; wherein, the trained target tracking model is trained according to one of the target tracking model training methods in any of the above schemes; in this embodiment, the target tracking model is built based on the feature extraction model, the target detection model, and the tracking module. The target tracking model is built using the feature extraction model, the target detection model, and the tracking module.

[0120] Specifically, the feature extraction model can consist of a dual encoder of ViT-L and BERT, with model weights frozen; the object detection model can be Grounding Dino 1.5 Pro, specifically including an encoder, a query selection module, and a decoder, with weights initialized through pre-training; the tracking module can be the MOTR tracker architecture in MOTRv2, specifically including a decoder that updates the loss parameters based on the real-time training process.

[0121] For example, the corresponding query text is the same for all image frames in the video stream.

[0122] Step S302: The video is sampled frame by frame by the feature extraction module of the trained target tracking model to obtain the original image frames; for example, the video is sampled frame by frame. The number of image frames in the video is used to obtain the original image frames. .

[0123] For example, this scheme uses pre-trained ViT-L and BERT as feature extraction models for images and text, respectively, for the visual feature extractor. With language feature extractor The weights in the training process are always frozen and are not updated during training. These weights are trained using a publicly available pre-training method.

[0124] Step S303: Generate a prompt text description based on the category to be labeled;

[0125] In this embodiment, the query category tags are concatenated and strung together to form the prompt text description. The entire video corresponds to the same query text.

[0126] Step S304: Input the prompt text description and the i-th frame image from the original image frame into the feature extraction module for feature extraction processing to obtain the text feature vector and the image feature vector; where i≥1;

[0127] In this embodiment, the i-th frame image from the original image frames is input into the visual feature extractor. Image feature extraction is performed to obtain feature vector image encoding. The prompt text description is input into a language feature extractor to extract language features and obtain the feature vector text encoding. Among them, visual feature extractor and language feature extractor The weights are fixed values.

[0128] For example, for the first Frames, image-text pairs Input dual encoders to generate corresponding modal features.

[0129] Step S305: Input the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain the i-th detection query vector;

[0130] In this embodiment, the text feature vector and image feature vectors The input detector encoder performs feature fusion processing to obtain the processed text feature vector. and processed image feature vectors The processed text feature vector and processed image feature vectors The input is processed by the query selection module to obtain the first query vector. The first query vector is then input to the detection decoder for modality alignment to obtain the i-th detection query vector. .

[0131] For example, image features and text feature encoding The input is an encoder used to fuse image and text features. The encoder converts image vectors... Text feature vectors As input, the features are fused through six layers of deformable-attention and self-attention modules in the encoder, as well as two layers of cross-attention modules (Image-to-text Cross-Attention and Text-to-image Cross-Attention), and then fed into the feedforward network (FFN). Finally, the encoder outputs the enhanced image vector. With text vectors .

[0132] The feature vector input uses a language-guided query selection model. The query selection module uses image vectors... With text vectors As input, output a query vector that is more closely related to the text. Query vector By using a self-attention mechanism, the weight of the query image content is dynamically adjusted based on the prompt text, thereby more accurately locating the object of interest.

[0133] Feature vector input to cross-modal decoder. Query vector. The decoder input is initialized and then passes through a self-attention layer, an image cross-attention layer, and a text cross-attention layer for modality alignment. Finally, it passes through a feedforward network (FFN) to output the detection query vector. Detect query vector By using a cross-modal decoder and a cross-modal attention mechanism, the query vector of S222 is interacted with the intra-frame global, and the selected visual and text features are fused using the cross-modal attention mechanism to ensure the accuracy and consistency of the prediction.

[0134] Step S306: Input the detection query vector of the (i-1)th frame of the (i-1)th frame image into the tracking module of the target tracking model for processing to obtain the (i-1)th tracking query vector;

[0135] In this embodiment, the (i-1)th detection query vector The input is processed in the tracking module of the target tracking model to obtain the (i-1)th tracking query vector. .

[0136] Step S307: The tracking module of the trained target tracking model processes the (i-1)th tracking query vector and the ith detection query vector to obtain the detection result of the ith frame image;

[0137] In this embodiment, the detection result of the previous frame is used as the tracking query vector. This is input into the current frame for predicting the trajectory of the detected target. It is compared with the detection results of the current frame. The concatenated vectors are input into the decoder as the initial query vector. As shown in Formula 1.

[0138] (Formula 1),

[0139] Aggregated vector The algorithm employs both a self-attention module and a deformable self-attention module. The self-attention module further integrates historically tracked targets and the currently detected target, while the deformable self-attention module recalibrates the target coordinate information to model the target features and trajectory of the query vector. Simultaneously, it relocalizes old targets and determines the iterative changes in target appearance and disappearance. Finally, it outputs the detection results of all current frames, which are used as the tracking vector for the next frame. .

[0140] The detection results are used to output the visual features of the predicted bounding boxes through a region head multilayer perceptron. With coordinate information As shown in Formula 2.

[0141] (Formula 2),

[0142] In this embodiment, the tracking module includes: a tracking encoder; and the (i-1)th tracking query vector. and the i-th detection query vector The data is input into the tracking encoder for splicing processing to obtain the splicing result. The stitching result is processed by the tracking encoder. The detection results are obtained by performing tracking processing to obtain the detection results of the i-th frame image.

[0143] Step S308: Starting from the first frame of the video to be labeled, repeat the above steps until the last frame of the video to be labeled, so as to complete the pre-labeling of the video to be labeled.

[0144] In one example, refer to Figure 4 As shown, a flowchart illustrating the steps of visual feature extraction and language feature extraction methods is presented, such as... Figure 4 As shown, the specific steps may include:

[0145] The target tracking model includes: the feature extraction module, the target detection module, and the tracking module;

[0146] The feature extraction module includes: a visual feature extractor and a language feature extractor;

[0147] Step S401: Input the prompt text description into the language feature extractor to extract language features and obtain a text feature vector;

[0148] Step S402: Input the i-th frame image from the original image frame into the visual feature extractor to extract image features and obtain an image feature vector; wherein the weights of the visual feature extractor and the language feature extractor are fixed values.

[0149] For example, data is input to the dual encoder module. The video is sampled frame by frame. The number of image frames in the video is used to obtain the original image frames. .

[0150] The corresponding category label for the target in each image is concatenated and strung together to form a prompt text description. Regarding the video number... Frame, single image Rather than prompt phrases respectively through visual feature extractor Language feature extractor To obtain their respective feature vector image encodings With text encoding .

[0151] In one example, refer to Figure 5 As shown, a flowchart illustrating the processing steps of the target detection module is presented. Figure 5 As shown, the specific steps may include:

[0152] The target detection module includes: a detection encoder, a query selection module, and a detection decoder;

[0153] Step S501: Input the text feature vector and the image feature vector into the detection encoder for feature fusion processing to obtain the processed text feature vector and the processed image feature vector;

[0154] In one example, refer to Figure 6 As shown, a schematic diagram of the target detection module structure is presented, as follows: Figure 6 As shown, it specifically includes: the target detection module structure is as follows Figure 6 As shown, the Self-Attention part is the text dimension processing module, the Deformable-Attention part is the visual dimension processing module, and the white two-layer cross attention module part is the multimodal processing module.

[0155] Feature vector image encoding With text encoding The input is an encoder used to fuse image and text features. The encoder converts image vectors... Text feature vectors As input, the features are fused through six layers of deformable-attention and self-attention modules in the encoder, as well as two layers of cross-attention modules (Image-to-text Cross-Attention and Text-to-image Cross-Attention), and then fed into the feedforward network (FFN). Finally, the encoder outputs the enhanced image vector. With text vectors .

[0156] Step S502: Input the processed text feature vector and the processed image feature vector into the query selection module for processing to obtain the first query vector;

[0157] For example, the feature vector input uses a language-guided query selection model. The query selection module takes image vectors as input. With text vectors As input, output a query vector that is more closely related to the text. .

[0158] Step S503: Input the first query vector into the detection decoder for modality alignment processing to obtain the i-th detection query vector.

[0159] In one example, refer to Figure 6 As shown, a schematic diagram of the target detection module structure is presented, as follows: Figure 6As shown, this specifically includes: a language-guided query selection model using feature vector input. The query selection module uses image vectors... With text vectors As input, output a query vector that is more closely related to the text.

[0160] Query vector Input cross-modal decoder. Query vector. The decoder input is initialized and then passes through a self-attention layer, an image cross-attention layer, and a text cross-attention layer for modality alignment. Finally, it passes through a feedforward network (FFN) to output the detection query vector. .

[0161] This method retains the detector while increasing interaction with historical frames. Trajectory information is transmitted frame by frame, and iterative processing across time is more conducive to feature alignment between frames. It improves target matching and association capabilities while preserving detection performance. It is particularly adaptable to scenes with more complex motion trajectories, such as changes in the speed and direction of a single target, or overlap and occlusion between targets.

[0162] In one example, refer to Figure 7 As shown, a flowchart illustrating the steps of encoder detection is presented, such as... Figure 7 As shown, the specific steps may include:

[0163] The detection encoder includes: a first deformable self-attention module, a first self-attention module, an image-text cross-attention module, a text-image cross-attention module, and an encoding feedforward network layer;

[0164] Step S701: Input the text feature vector into the first self-attention module for processing to obtain the first encoding processing result;

[0165] Step S702: Input the image feature vector into the first deformable self-attention module for processing to obtain the second encoding processing result;

[0166] Step S703: The first encoding processing result is used as K and V, and the second encoding processing result is used as Q. These are input into the image-text cross-attention module for processing to obtain the processed K, V, and Q.

[0167] Step S704: Input the processed K, V and Q into the text-image cross-attention module for processing to obtain the third encoding processing result;

[0168] Step S705: Input the third encoding processing result into the encoding feedforward network layer for processing to obtain the processed text feature vector and the processed image feature vector.

[0169] In one example, refer to Figure 8 As shown, a flowchart illustrating the processing steps of the detection decoder is presented, such as... Figure 8 As shown, the specific steps may include:

[0170] The detection decoder includes: a self-attention layer, an image cross-attention layer, a text cross-attention layer, and a decoding feedforward network layer;

[0171] Step S801: Input the first query vector into the self-attention layer for processing to obtain the first decoding result;

[0172] Step S802: The processed image feature vector is used as K and V, and the first decoding result is used as Q. These are input into the image cross-attention layer for processing to obtain the second decoding result.

[0173] Step S803: The processed text feature vector is used as K and V, and the second decoding result is used as Q. The result is input into the text cross-attention layer for processing to obtain the third decoding result.

[0174] Step S804: Input the third decoding processing result into the decoding feedforward network layer for processing to obtain the i-th detection query vector.

[0175] In one example, refer to Figure 9 As shown, a flowchart illustrating the processing steps of the tracking module is presented. Figure 9 As shown, the specific steps may include:

[0176] The tracking module includes: a second self-attention module, a second deformable self-attention module, and a region multilayer perceptron;

[0177] Step S901: Input the (i-1)th tracking query vector and the ith detection query vector of the (i-1)th frame image into the tracking encoder for splicing processing to obtain the splicing result;

[0178] Step S902: Input the splicing result into the second self-attention module to obtain the first tracking result;

[0179] Step S903: Input the first tracking result and the i-th detection query vector into the second deformable self-attention module for processing to obtain the second tracking result;

[0180] Step S904: Input the second tracking result into the region multilayer perceptron for processing to obtain the detection result of the i-th frame image.

[0181] For example, refer to Figure 10 As shown, a schematic diagram of the tracking module structure is presented, such as... Figure 10 As shown, the specific steps may include:

[0182] The detection result of the previous frame is used as the tracking query vector. This is input into the current frame for predicting the trajectory of the detected target. It is compared with the detection results of the current frame. The concatenated vectors are input into the decoder as the initial query vector. ;

[0183] Aggregated vector The algorithm employs both a self-attention module and a deformable self-attention module. The self-attention module further integrates historically tracked targets and the currently detected target, while the deformable self-attention module recalibrates the target coordinate information to model the target features and trajectory of the query vector. Simultaneously, it relocalizes old targets and determines the iterative changes in target appearance and disappearance. Finally, it outputs the detection results of all current frames, which are used as the tracking vector for the next frame. .

[0184] The detection results are processed by a region head to output the visual features of the predicted bounding boxes. With coordinate information .

[0185] This method decomposes target tracking into two sub-tasks: target detection and tracking. The tracking module utilizes the spatiotemporal localization and feature encoding provided by the detection module for further optimization and training. This information exchange allows the tracking module to focus more on target re-identification, thus mitigating the conflict between the two related tasks. Furthermore, the decoupled sub-tasks can be pre-trained on their respective datasets, or modules can be replaced for application in other scenarios and tasks.

[0186] Multimodal fusion and learning are achieved through training that contrasts language and vision. Visual localization aligns the rich semantic information from textual data with image information; this synergy enables the model to learn visual representations of language perception, better guiding object detection and association tasks.

[0187] In one example, a trained target tracking model is used to annotate multi-target autonomous driving videos to obtain a video sample dataset.

[0188] The video sample dataset is added to the driving video dataset used to train the autonomous driving model to obtain the updated driving video dataset;

[0189] The autonomous driving model is trained based on the updated driving video dataset to obtain the trained autonomous driving model.

[0190] For example, a video pre-annotation method based on multi-object tracking may include:

[0191] A visual positioning-based target tracking model was built, including a dual encoder module, a target detection module, and a tracking module.

[0192] Model training involves training on a dataset for a multi-object tracking task, where each frame of the video contains the object's coordinates and category. Starting from the first frame, image-category text descriptions are combined into text pairs and sequentially input into the object tracking network to ultimately obtain the bounding box and category of the object for each frame.

[0193] Data is input to the dual encoder module. (Video number...) Image-text pairs of frames The inputs are respectively fed into the visual feature extractor. Language feature extractor Obtain image encoding and text encoding ;

[0194] The feature vector is input to the object detection module. The object detection module encodes the image. and text encoding As input, cross-modal image and text features are obtained through the multimodal fusion encoder; these features are then input to the decoder to output a detection vector. ;

[0195] The feature vector is input to the tracking module. The tracking module then detects the vector. Tracking vector from the previous frame As input, the prediction result and tracking vector of the current frame are obtained. ;

[0196] Construct a loss function. Associate the labeled and predicted bounding boxes using a binary Hungarian matching algorithm, and use contrastive loss to measure the localization and classification performance of each target.

[0197] Repeat the above steps until the last frame of the video.

[0198] Model inference includes:

[0199] Data is input to the dual encoder module. The sampled video... Frame image With query text To form image-text pairs Input to the dual encoder module;

[0200] The feature vector is input to the object detection module. The output image is then encoded. and text encoding The detection query vector for each frame is obtained through the object detection model. ;

[0201] The feature vector is input to the tracking module. The detection query vector of the current frame is then used. Tracking query vector from the previous frame Input the data into the tracking model to obtain the tracking query vector for that frame. With the detection bounding box and category;

[0202] Repeat the above steps until the last frame of the video.

[0203] Specifically, for the video stream to be processed, the image sampling interval can be set during model training and inference. This description only applies to scenarios where frame skipping is not set.

[0204] For example, the model inference process includes:

[0205] The video to be labeled and the target category to be labeled are input into the visual localization-based target tracking model that has been trained. For all image frames in the video stream, the corresponding query text is the same.

[0206] Data is input to the dual encoder module.

[0207] Video is sampled frame by frame. The number of image frames in the video is used to obtain the original image frames. .

[0208] The query category tags are concatenated and strung together to form the prompt text description. The entire video corresponds to the same query text.

[0209] For the Frames, image-text pairs Input dual encoders to generate corresponding modal features.

[0210] The feature vector is input to the object detection module. Image features. and text feature encoding The object detection model obtains the detection query vector of the object in the frame. .

[0211] The feature vector is input to the tracking module. The detection query vector of the object in the current frame is used. Tracking query vector of the object in the previous frame The data is input into the tracking model to obtain the detected bounding boxes and categories for that frame, as well as the tracking query vector for relocalization in the next frame. .

[0212] Starting from the first frame of the video, repeat the above steps until the last frame of the video.

[0213] Compared to detection-based tracking methods, this method retains the detector, increases interaction with historical frames, and transmits trajectory information frame by frame. Iteration across time is more conducive to feature alignment between frames. While preserving detection performance, it enhances target matching and association capabilities. It is particularly adaptable to scenes with more complex motion trajectories, such as changes in the speed and direction of a single target, or overlap and occlusion between targets. This results in stable multi-target tracking performance.

[0214] This method decomposes target tracking into two sub-tasks: target detection and tracking. The tracking module utilizes the spatiotemporal localization and feature encoding provided by the detection module for further optimization and training. This information exchange allows the tracking module to focus more on target re-identification, thus mitigating the conflict between the two related tasks. Furthermore, the decoupled sub-tasks can be pre-trained on their respective datasets, or modules can be replaced for application in other scenarios and tasks.

[0215] This method uses model weights trained on tens of millions of datasets for initialization, accelerating training convergence while its excellent generalization ability allows it to be applied to various types and quantities of targets, as well as different scenarios and environments. Furthermore, it can be quickly applied to new tasks and domains even with limited or no training data, effectively expanding to new target categories.

[0216] This approach uses language-guided target tracking tasks and employs multimodal fusion and learning through contrastive training between language and vision. Visual localization aligns the rich semantic information from textual data with image information; this synergy enables the model to learn visual representations of language perception, better guiding target detection and association tasks. This allows the model to maintain strong performance in the current data representation space while achieving good zero-shot and few-shot transfer learning capabilities.

[0217] This approach utilizes the Transformer architecture to process both visual and linguistic data. This allows the entire model to inherit the strong scalability of the Transformer, enabling it to better capture and represent complex features. Furthermore, the consistent structure allows for flexible aggregation of features from both modalities at various stages, resulting in a similar and compact semantic space. This makes it easier for the network to learn effective feature representations, thereby improving convergence speed and overall performance. The Transformer architecture represents the attention architecture of the object detection and tracking modules.

[0218] This solution enhances the data transfer capability of an open-set object detection model through fine-tuning; and improves the understanding of language perception and image features by introducing text modality through visual localization for contrastive learning. Its transfer learning performance allows for easy generalization to more target types after fine-tuning on a small amount of data in practical applications. Furthermore, text prompts enable zero-shot detection and tracking to a certain extent, providing a solution for pre-annotation in various fields.

[0219] In practical use, this solution obtains the category and bounding box of relevant objects for each frame by inputting query text. It employs a multi-object tracking method based on query tracking, enabling it to handle more complex motion trajectories. Furthermore, the tracking model optimizes itself based on prior information provided by the object detection model, synergistically improving the accuracy of object detection and tracking. This reduces the increased workload of manual corrections due to missed object detections or inaccurate entity category identification, and yields more high-quality labeled data. It also lowers the model transfer and training costs across different scenarios, enabling rapid optimization and improving model iteration efficiency.

[0220] In one example, refer to Figure 11 As shown, a structural block diagram of a target tracking model training device according to an embodiment of the present invention is illustrated. Figure 11 As shown, the device may specifically include the following modules:

[0221] Model building module 1001 is used to build target tracking models;

[0222] The training sampling module 1002 is used to input the video sample dataset of the multi-target tracking task into the feature extraction module of the target tracking model, perform frame-by-frame sampling, and obtain a preset number of original sample image frames.

[0223] The training stitching module 1003 is used to concatenate and stitch multiple target category labels in the i-th frame of the original sample image frame into the i-th prompt text description.

[0224] The visual feature extraction module 1004 is used to encode the i-th frame image through a visual feature extractor to obtain the feature vector of the i-th frame image;

[0225] The language feature extraction module 1005 is used to encode the i-th prompt text description through a language feature extractor to obtain the text feature vector of the i-th frame;

[0226] The training detection module 1006 is used to process the feature vector of the i-th frame image and the feature vector of the i-th frame text through the target detection module of the target tracking model to obtain the i-th frame detection query vector;

[0227] The training tracking module 1007 is used to input the detection query vector of the (i-1)th frame of the (i-1)th frame image into the tracking module of the target tracking model for processing to obtain the (i-1)th tracking query vector; and then the tracking module of the target tracking model processes the (i-1)th tracking query vector and the detection query vector of the i-th frame to obtain the detection result of the i-th frame image.

[0228] The loop training module 1008 is used to repeat the above steps from the first frame of the video sample dataset until the last frame of the video sample dataset to obtain the detection results of each frame of the video sample dataset.

[0229] The training calculation module 1009 is used to calculate the loss parameters of the target tracking model based on the predicted bounding box category feature vector and the corresponding category feature vector label in the detection results of each frame of the video sample dataset, and the predicted bounding box coordinates and the corresponding coordinate labels in the detection results of each frame of the video sample dataset.

[0230] The training optimization module 1010 is used to train the target tracking model to be trained according to the loss parameters, so as to obtain the trained target tracking model.

[0231] The target tracking model training device provided in the above embodiments can realize the technical solution described in the above target tracking model training method embodiments. The specific implementation principle of each module or unit can be found in the corresponding content of the above target tracking model training method embodiments, which will not be repeated here.

[0232] In one example, a multi-target video annotation device according to an embodiment of the present invention may specifically include the following modules:

[0233] The input module is used to input the video to be labeled and the category to be labeled into the trained target tracking model; wherein, the trained target tracking model is trained according to any of the above-mentioned target tracking model training methods;

[0234] The sampling module is used to sample the video frame by frame through the feature extraction module of the trained target tracking model to obtain the original image frames;

[0235] The text description generation module is used to generate prompt text descriptions based on the category to be labeled;

[0236] The feature extraction module is used to input the prompt text description and the i-th frame image from the original image frame into the feature extraction module for feature extraction processing to obtain text feature vectors and image feature vectors; where i≥1;

[0237] The detection module is used to input the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing, and obtain the i-th detection query vector;

[0238] The tracking module is used to input the detection query vector of the (i-1)th frame of the (i-1)th frame image into the tracking module of the target tracking model for processing, so as to obtain the (i-1)th tracking query vector;

[0239] The tracking module is used to process the (i-1)th tracking query vector and the ith detection query vector through the tracking module of the trained target tracking model to obtain the detection result of the ith frame image;

[0240] The annotation module is used to repeat the above steps starting from the first frame of the video to be annotated until the last frame of the video to be annotated, so as to complete the pre-annotation of the video to be annotated.

[0241] The multi-target video annotation device provided in the above embodiments can realize the technical solution described in the above multi-target video annotation method embodiments. The specific implementation principle of each module or unit can be found in the corresponding content in the above multi-target video annotation method embodiments, which will not be repeated here.

[0242] The above provides a detailed description of the target tracking model training method, multi-target video annotation method, and apparatus provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for training a target tracking model, characterized in that, include: Build a target tracking model; The video sample dataset of the multi-target tracking task is input into the feature extraction module of the target tracking model, and frame-by-frame sampling is performed to obtain a preset number of original sample image frames. The multiple target category labels in the i-th frame of the original sample image are concatenated and spliced ​​together to form the i-th prompt text description; The i-th frame image is encoded using a visual feature extractor to obtain the feature vector of the i-th frame image; The i-th prompt text description is encoded using a language feature extractor to obtain the text feature vector of the i-th frame; The target detection module of the target tracking model processes the feature vector of the i-th frame image and the feature vector of the i-th frame text to obtain the detection query vector of the i-th frame. The detection query vector of the (i-1)th frame of the image is input into the tracking module of the target tracking model for processing to obtain the (i-1)th tracking query vector; The tracking module of the target tracking model processes the (i-1)th tracking query vector and the i-th frame detection query vector to obtain the detection result of the i-th frame image; Repeat the above steps starting from the first frame of the video sample dataset until the last frame of the video sample dataset to obtain the detection results for each frame of the video sample dataset; The loss parameters of the target tracking model are calculated based on the predicted bounding box category feature vector and the corresponding category feature vector label in the detection results of each frame of the video sample dataset, as well as the predicted bounding box coordinates and the corresponding coordinate labels in the detection results of each frame of the video sample dataset. The target tracking model to be trained is trained according to the loss parameters to obtain the trained target tracking model.

2. The target tracking model training method according to claim 1, characterized in that, The step of calculating the loss parameters of the target tracking model based on the predicted bounding box category feature vector and corresponding category feature vector label in the detection results of each frame of the video sample dataset, and the predicted bounding box coordinates and corresponding coordinate labels in the detection results of each frame of the video sample dataset, specifically includes: Based on the matching results of the detection results and the true results of each frame of image, the classification loss and coordinate regression loss are calculated. The loss parameters of the target tracking model are calculated based on the classification loss, the coordinate regression loss, the predicted bounding box category feature vector and the corresponding category feature vector label in the detection results of each frame of the image, and the predicted bounding box coordinates and the corresponding coordinate labels in the detection results of each frame of the image.

3. A multi-target video annotation method, characterized in that, include: The video to be labeled and the category to be labeled are input into the trained target tracking model; wherein, the trained target tracking model is trained according to one of the target tracking model training methods in claim 1 or 2; The feature extraction module of the trained target tracking model samples the video frame by frame to obtain the original image frames. Generate a prompt text description based on the category to be labeled; The prompt text description and the i-th frame image from the original image frame are respectively input into the feature extraction module for feature extraction processing to obtain the text feature vector and the image feature vector; where i≥1; The text feature vector and the image feature vector are input into the target detection module of the trained target tracking model for processing to obtain the i-th detection query vector. The detection query vector of the (i-1)th frame of the image is input into the tracking module of the target tracking model for processing to obtain the (i-1)th tracking query vector; The tracking module of the trained target tracking model processes the (i-1)th tracking query vector and the ith detection query vector to obtain the detection result of the ith frame image. Starting from the first frame of the video to be labeled, repeat the above steps until the last frame of the video to be labeled to complete the pre-labeling of the video.

4. The multi-target video annotation method according to claim 3, characterized in that, The target tracking model includes: the feature extraction module, the target detection module, and the tracking module; The feature extraction module includes: a visual feature extractor and a language feature extractor; The prompt text description and the i-th frame image from the original image frame are respectively input into the feature extraction module for feature extraction processing to obtain text feature vectors and image feature vectors, specifically including: The prompt text description is input into the language feature extractor for language feature extraction to obtain a text feature vector; The i-th frame of the original image frame is input into the visual feature extractor for image feature extraction to obtain an image feature vector; wherein the weights of the visual feature extractor and the language feature extractor are fixed values.

5. A multi-target video annotation method according to claim 3, characterized in that, The target detection module includes: a detection encoder, a query selection module, and a detection decoder; The step of inputting the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain the i-th detection query vector specifically includes: The text feature vector and the image feature vector are input into the detection encoder for feature fusion processing to obtain the processed text feature vector and the processed image feature vector; The processed text feature vector and the processed image feature vector are input into the query selection module for processing to obtain the first query vector; The first query vector is input into the detection decoder for modality alignment processing to obtain the i-th detection query vector.

6. The multi-target video annotation method according to claim 5, characterized in that, The detection encoder includes: a first deformable self-attention module, a first self-attention module, an image-text cross-attention module, a text-image cross-attention module, and an encoding feedforward network layer; The step of inputting the text feature vector and the image feature vector into the detection encoder for feature fusion processing to obtain the processed text feature vector and the processed image feature vector specifically includes: The text feature vector is input into the first self-attention module for processing to obtain the first encoding processing result; The image feature vector is input into the first deformable self-attention module for processing to obtain the second encoding processing result; The first encoding result is used as K and V, and the second encoding result is used as Q. These are input into the image-text cross-attention module for processing to obtain the processed K, V, and Q. The processed K, V, and Q are input into the text-image cross-attention module for further processing to obtain the third encoding result; The third encoding processing result is input into the encoding feedforward network layer for processing to obtain the processed text feature vector and the processed image feature vector.

7. The multi-target video annotation method according to claim 5, characterized in that, The detection decoder includes: a self-attention layer, an image cross-attention layer, a text cross-attention layer, and a decoding feedforward network layer; The step of inputting the first query vector into the detection decoder for modality alignment processing to obtain the i-th detection query vector specifically includes: The first query vector is input into the self-attention layer for processing to obtain the first decoding result; The processed image feature vectors are used as K and V, and the first decoding result is used as Q. These are then input into the image cross-attention layer for processing to obtain the second decoding result. The processed text feature vectors are used as K and V, and the second decoding result is used as Q. These are then input into the text cross-attention layer for processing to obtain the third decoding result. The third decoding result is input into the decoding feedforward network layer for processing to obtain the i-th detection query vector.

8. A multi-target video annotation method according to claim 3, characterized in that, The tracking module includes: a second self-attention module, a second deformable self-attention module, and a region multilayer perceptron; The tracking module of the trained target tracking model processes the (i-1)th tracking query vector and the ith detection query vector of the (i-1)th frame image to obtain the detection result of the ith frame image, specifically including: The (i-1)th tracking query vector and the ith detection query vector of the (i-1)th frame image are input into the tracking encoder for splicing processing to obtain the splicing result; The stitching result is input into the second self-attention module to obtain the first tracking result; The first tracking result and the i-th detection query vector are input into the second deformable self-attention module for processing to obtain the second tracking result; The second tracking result is input into the region multilayer perceptron for processing to obtain the detection result of the i-th frame image.

9. A multi-target video annotation method according to claim 3, characterized in that, Also includes: The trained target tracking model is used to annotate targets in multi-target autonomous driving videos to obtain a video sample dataset. The video sample dataset is added to the driving video dataset used to train the autonomous driving model to obtain the updated driving video dataset; The autonomous driving model is trained based on the updated driving video dataset to obtain the trained autonomous driving model.

10. A target tracking model training device, characterized in that, include: The module includes a model building module, a training sampling module, a training stitching module, a visual feature extraction module, a language feature extraction module, a training detection module, a training tracking module, a loop training module, a training computation module, and a training optimization module. The model building module is used to build a target tracking model; The training sampling module is used to input the video sample dataset of the multi-target tracking task into the feature extraction module of the target tracking model, perform frame-by-frame sampling, and obtain a preset number of original sample image frames. The training stitching module is used to concatenate and stitch multiple target category labels in the i-th frame of the original sample image frame into the i-th prompt text description; The visual feature extraction module is used to encode the i-th frame image through a visual feature extractor to obtain the feature vector of the i-th frame image; The language feature extraction module is used to encode the i-th prompt text description through a language feature extractor to obtain the text feature vector of the i-th frame. The training detection module is used to process the image feature vector and the text feature vector of the i-th frame through the target detection module of the target tracking model to obtain the detection query vector of the i-th frame; The training tracking module is used to input the detection query vector of the (i-1)th frame of the (i-1)th frame image into the tracking module of the target tracking model for processing, so as to obtain the (i-1)th tracking query vector; The tracking module of the target tracking model then processes the (i-1)th tracking query vector and the i-th frame detection query vector to obtain the detection result of the i-th frame image. The loop training module is used to repeat the above steps from the first frame of the video sample dataset until the last frame of the video sample dataset, so as to obtain the detection result of each frame of the video sample dataset. The training calculation module is used to calculate the loss parameters of the target tracking model based on the predicted bounding box category feature vector and corresponding category feature vector label in the detection results of each frame of the video sample dataset, and the predicted bounding box coordinates and corresponding coordinate labels in the detection results of each frame of the video sample dataset. The training optimization module is used to train the target tracking model to be trained according to the loss parameters, so as to obtain the trained target tracking model.

Images