Target tracking model training method, multi-target video labeling method and device

Abstract

The invention relates to a target tracking model training method and a multi-target video labeling method and device, and belongs to the technical field of computer vision, and the method comprises the steps: carrying out the frame-by-frame sampling of a to-be-labeled video through a feature extraction module, and obtaining an ith frame image; generating prompt text description according to an input to-be-labeled category; performing feature extraction processing on the prompt text description and the ith frame of image to obtain a text feature vector and an image feature vector; inputting the vector into a target detection module for processing to obtain an ith detection query vector; processing the (i-1) th tracking query vector and the ith detection query vector through a tracking module to obtain a detection result of the ith frame of image; and repeating the operation of the steps until the last frame of the video to be labeled, so as to complete the pre-labeling of the video to be labeled. The multiple targets in the video are labeled in a mode of combining the prompt text with the targets, so that the problem of target missing detection or inaccurate entity category recognition is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a target tracking model training method, a multi-target video labeling method and a device. Background Art

[0002] As the number of parameters in deep learning models increases, the amount of training data required is also increasing. Especially in the era of large models, hundreds of millions of parameters require billions of data to support them. The training method for most existing tasks is still supervised learning that requires labeled data, so the quality and scale of labeled data sets have become a bottleneck restricting the development of large models.

[0003] It is not easy to build a high-quality, large-scale professional field dataset entirely through manual data annotation: the training of annotation personnel and manual annotation are costly and time-consuming. Especially for the annotation of video tasks, the label requirements of each sub-scene are different, which brings heavy labeling work. Although video annotation tools such as CVAT and darklabel have built-in some intelligent functions, they require humans to hot start or rely on human cognition, which may cause the omission of targets. Summary of the invention

[0004] In view of this, it is necessary to provide a target tracking model training method, a multi-target video annotation method and a device to solve the problem that the existing manual annotation is high in cost and time-consuming; and the existing video annotation tools require people to hot-start or rely on human cognition, which will cause the problem of missing targets.

[0005] In order to solve the above problems, the present invention provides a target tracking model training method, comprising: building a target tracking model; Inputting the video sample data set of the multi-target tracking task into the feature extraction module of the target tracking model, sampling frame by frame, and obtaining a preset number of original sample image frames; Concatenate multiple target category labels in the i-th frame image of the original sample image frame into an i-th prompt text description; Encode the i-th frame image through the visual feature extractor to obtain the i-th frame image feature vector; Encode the i-th prompt text description by a language feature extractor to obtain an i-th frame text feature vector; Processing the image feature vector of the i-th frame and the text feature vector of the i-th frame through the target detection module of the target tracking model to obtain a detection query vector of the i-th frame; Inputting the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; Processing the i-1th tracking query vector and the i-th frame detection query vector through the tracking module of the target tracking model to obtain a detection result of the i-th frame image; Repeat the above steps from the first frame of the video sample data set to the last frame of the video sample data set to obtain the detection result of each frame image of the video sample data set; Calculate the loss parameter of the target tracking model according to the predicted box category feature vector and the corresponding category feature vector label in the detection result of each frame image of the video sample data set, and the predicted box coordinates and the corresponding coordinate labels in the detection result of each frame image of the video sample data set; The target tracking model to be trained is trained according to the loss parameter to obtain a trained target tracking model.

[0006] Optionally, the calculating the loss parameter of the target tracking model according to the predicted box category feature vector and the corresponding category feature vector label in the detection result of each frame image of the video sample data set, and the predicted box coordinates and the corresponding coordinate labels in the detection result of each frame image of the video sample data set, specifically includes: According to the detection result of each frame of image and the matching result of the real result of each frame of image, a classification loss and a coordinate regression loss are calculated; The loss parameters of the target tracking model are calculated based on the classification loss, the coordinate regression loss, the predicted box category feature vector and the corresponding category feature vector label in the detection results of each frame image, and the predicted box coordinates and the corresponding coordinate labels in the detection results of each frame image.

[0007] In order to solve the above problems, the present invention also provides a multi-target video labeling method, comprising: Inputting the video to be labeled and the category to be labeled into a trained target tracking model; wherein the trained target tracking model is trained according to a target tracking model training method according to claim 1 or 2; The video is sampled frame by frame by using the feature extraction module of the trained target tracking model to obtain original image frames; Generate a prompt text description according to the category to be labeled; Input the prompt text description and the i-th frame image in the original image frame into the feature extraction module for feature extraction processing to obtain a text feature vector and an image feature vector; wherein i≥1; Inputting the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain an i-th detection query vector; Inputting the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; Processing the (i-1)th tracking query vector and the (i)th detection query vector by the tracking module of the trained target tracking model to obtain a detection result of the (i)th frame image; Starting from the first frame of the video to be labeled, the above steps are repeated until the last frame of the video to be labeled, so as to complete the pre-labeling of the video to be labeled.

[0008] Optionally, the target tracking model includes: the feature extraction module, the target detection module and the tracking module; The feature extraction module includes: a visual feature extractor and a language feature extractor; The prompt text description and the i-th frame image in the original image frame are respectively input into the feature extraction module for feature extraction processing to obtain a text feature vector and an image feature vector, specifically including: Inputting the prompt text description into the language feature extractor to extract language features and obtain a text feature vector; The i-th frame image in the original image frame is input into the visual feature extractor to extract image features, so as to obtain an image feature vector; wherein the weight of the visual feature extractor and the weight of the language feature extractor are respectively fixed values.

[0009] Optionally, the target detection module includes: a detection encoder, a query selection module and a detection decoder; The step of inputting the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain the i-th detection query vector specifically includes: Inputting the text feature vector and the image feature vector into the detection encoder for feature fusion processing to obtain a processed text feature vector and a processed image feature vector; Inputting the processed text feature vector and the processed image feature vector into the query selection module for processing to obtain a first query vector; The first query vector is input into the detection decoder for modality alignment processing to obtain an i-th detection query vector.

[0010] Optionally, the detection encoder comprises: a first deformable self-attention module, a first self-attention module, an image-text cross-attention module, a text-image cross-attention module and an encoding feed-forward network layer; The step of inputting the text feature vector and the image feature vector into the detection encoder for feature fusion processing to obtain the processed text feature vector and the processed image feature vector specifically includes: Inputting the text feature vector into the first self-attention module for processing to obtain a first encoding processing result; Inputting the image feature vector into the first deformable self-attention module for processing to obtain a second encoding processing result; The first encoding processing result is used as K and V, and the second encoding processing result is used as Q, and inputted into the image-text cross attention module for processing to obtain processed K, V and Q; The processed K, V and Q are input into the text-image cross attention module for processing to obtain the third encoding processing result; The third encoding processing result is input into the encoding feedforward network layer for processing to obtain a processed text feature vector and a processed image feature vector.

[0011] Optionally, the detection decoder comprises: a self-attention layer, an image cross-attention layer, a text cross-attention layer, and a decoding feed-forward network layer; The step of inputting the first query vector into the detection decoder for modality alignment processing to obtain an i-th detection query vector specifically includes: Inputting the first query vector into the self-attention layer for processing to obtain a first decoding processing result; The processed image feature vector is used as K and V, and the first decoding processing result is used as Q, and inputted into the image cross attention layer for processing to obtain a second decoding processing result; The processed text feature vector is used as K and V, and the second decoding processing result is used as Q, and inputted into the text cross attention layer for processing to obtain a third decoding processing result; The third decoding processing result is input into the decoding feedforward network layer for processing to obtain an i-th detection query vector.

[0012] Optionally, the tracking module includes: a second self-attention module, a second deformable self-attention module and a regional multi-layer perceptron; Processing the i-1th tracking query vector and the i-th detection query vector of the i-1th frame image through the tracking module of the trained target tracking model to obtain the detection result of the i-th frame image specifically includes: Inputting the i-1th tracking query vector of the i-1th frame image and the i-th detection query vector into the tracking encoder for splicing processing to obtain a splicing result; Inputting the splicing result into the second self-attention module to obtain a first tracking result; Inputting the first tracking result and the i-th detection query vector into the second deformable self-attention module for processing to obtain a second tracking result; The second tracking result is input into the regional multi-layer perceptron for processing to obtain the detection result of the i-th frame image.

[0013] Optionally, it also includes: Use the trained target tracking model to annotate the multi-target unmanned driving videos and obtain a video sample dataset; Adding the video sample dataset to a driving video dataset used for training an unmanned driving model to obtain an updated driving video dataset; The unmanned driving model is trained according to the updated driving video data set to obtain a trained unmanned driving model.

[0014] The present invention also provides a target tracking model training device, including: a model building module, a training sampling module, a training splicing module, a visual feature extraction module, a language feature extraction module, a training detection module, a training tracking module, a cyclic training module, a training calculation module and a training optimization module; The model building module is used to build a target tracking model; The training sampling module is used to input the video sample data set of the multi-target tracking task into the feature extraction module of the target tracking model, perform frame-by-frame sampling, and obtain a preset number of original sample image frames; The training splicing module is used to serially splice multiple target category labels in the i-th frame image of the original sample image frame into the i-th prompt text description; The visual feature extraction module is used to encode the i-th frame image through the visual feature extractor to obtain the i-th frame image feature vector; The language feature extraction module is used to encode the i-th prompt text description through a language feature extractor to obtain the i-th frame text feature vector; The training detection module is used to process the i-th frame image feature vector and the i-th frame text feature vector through the target detection module of the target tracking model to obtain the i-th frame detection query vector; The training tracking module is used to input the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; then the tracking module of the target tracking model processes the i-1th tracking query vector and the i-th frame detection query vector to obtain the detection result of the i-th frame image; The loop training module is used to repeat the above steps from the first frame of the video sample data set to the last frame of the video sample data set to obtain the detection result of each frame image of the video sample data set; The training calculation module is used to calculate the loss parameter of the target tracking model according to the prediction box category feature vector and the corresponding category feature vector label in the detection result of each frame image of the video sample data set, and the prediction box coordinates and the corresponding coordinate labels in the detection result of each frame image of the video sample data set; The training optimization module is used to train the target tracking model to be trained according to the loss parameter to obtain the trained target tracking model.

[0015] By applying the technical solution provided in the embodiment of the present application, multimodal fusion and learning are performed using contrast training between language and vision. This synergy enables the model to learn visual representations of language perception, better guide target detection and related tasks, and better capture and express complex target features through the target detection module and tracking module of the model, making it easier for the model network to learn effective feature representations, thereby improving convergence speed and overall performance. Rapid tuning is achieved and model iteration efficiency is improved.

[0016] The solution of the present invention combines prompt text with the target to obtain a target tracking solution with strong scalability and high accuracy. When using the model to annotate multiple targets in a video, the detection results of possible targets in each frame of the image can be output according to the prompt phrases and videos that need to be annotated, thereby reducing the increase in manual correction workload caused by missed targets or inaccurate entity category recognition, and obtaining more high-quality labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of a target tracking model training method provided in an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of a multi-target tracking model based on visual positioning provided in an embodiment of the present invention; Figure 3 is a flowchart of the steps of a multi-target video labeling method provided in an embodiment of the present invention; Figure 4 is a flowchart of the steps of the visual feature extraction and language feature extraction method provided in an embodiment of the present invention; Figure 5 is a flowchart of the processing steps of the target detection module provided in an embodiment of the present invention; Figure 6 is a schematic diagram of the structure of a target detection module provided in an embodiment of the present invention; Figure 7 is a flowchart of the steps of the processing process of detecting the encoder provided in an embodiment of the present invention; Figure 8 is a flowchart of the steps of the processing process of detecting a decoder provided in an embodiment of the present invention; Fig. 9 is a flowchart of the steps of the processing process of the tracking module provided in an embodiment of the present invention; Fig.10 is a schematic diagram of the structure of a tracking module provided in an embodiment of the present invention; Fig.11 It is a structural block diagram of a target tracking model training device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Reference Figure 1 As shown, a flowchart of a method for training a target tracking model is shown, which can be applied to computer vision, such as Figure 1 As shown, the following steps may be specifically included: Step S101, building a target tracking model; In this embodiment, a target tracking model is built through a feature extraction model, a target detection model and a tracking module; Specifically, the feature extraction model can be composed of ViT-L and BERT dual encoders, and the model weights are frozen; the target detection model can be Grounding Dino 1.5 Pro, specifically including: an encoder, a query selection module and a decoder, and the weights are initialized through pre-training; the tracking module can be: the MOTR tracker architecture in MOTRv2, specifically including: a decoder, which updates the loss parameters according to the real-time training process.

[0020] Step S102: inputting the video sample data set of the multi-target tracking task into the feature extraction module of the target tracking model, performing frame-by-frame sampling, and obtaining a preset number of original sample image frames; In this embodiment, the feature extraction module includes: a visual feature extractor and language feature extractor ; Step S103, concatenating multiple target category labels in the i-th frame image of the original sample image frame into an i-th prompt text description; In this embodiment, during the training process, each image contains a corresponding category label of the target, and the category labels in the image are concatenated to obtain a prompt text description. The category label in the image can include at least one.

[0021] Step S104, encoding the i-th frame image by a visual feature extractor to obtain a feature vector of the i-th frame image; In this embodiment, the i-th frame image in the original image frame is input into the visual feature extractor Extract image features and obtain feature vector image encoding ; Step S105, encoding the i-th prompt text description by a language feature extractor to obtain an i-th frame text feature vector; In this embodiment, the prompt text description is input into the language feature extractor to extract language features and obtain the feature vector text encoding ; Among them, the visual feature extractor and language feature extractor The weights are fixed values.

[0022] Step S106, processing the image feature vector of the i-th frame and the text feature vector of the i-th frame by the target detection module of the target tracking model to obtain a detection query vector of the i-th frame; In this embodiment, the text feature vector and the image feature vector The feature fusion process is performed in the input detection encoder to obtain the processed text feature vector And the processed image feature vector . The processed text feature vector And the processed image feature vector Input the first query vector to the detection decoder for modality alignment processing to obtain the i-th detection query vector .

[0023] Step S107, inputting the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; In this embodiment, the i-1th detection query vector Input into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector .

[0024] Step S108, processing the i-1th tracking query vector and the i-th frame detection query vector through the tracking module of the target tracking model to obtain the detection result of the i-th frame image; the tracking module includes: a tracking encoder; In this embodiment, the i-1th tracking query vector and the i-th detection query vector Input into the tracking encoder for splicing processing to obtain the splicing result ; The tracking encoder is used to Perform tracking processing to obtain the detection result of the i-th frame image.

[0025] Step S109, repeating the above steps from the first frame of the video sample data set to the last frame of the video sample data set, to obtain the detection result of each frame image of the video sample data set; Step S110, calculating the loss parameter of the target tracking model according to the prediction box category feature vector and the corresponding category feature vector label in the detection result of each frame image of the video sample data set, and the prediction box coordinates and the corresponding coordinate labels in the detection result of each frame image of the video sample data set; Step S111: training the target tracking model to be trained according to the loss parameter to obtain a trained target tracking model.

[0026] For example, the detection result of the previous frame is used as the tracking query vector , input to the current frame for prediction of the trajectory of the detected target. The concatenation is input into the decoder as the initial query vector , as shown in Formula 1.

[0027] (Formula 1), The aggregated vector After the self-attention module (Self-Attention) and the deformable self-attention module (Deformable-Attention). The self-attention module further integrates the historical tracking target and the current detection target, and the deformable self-attention module recalibrates the target coordinate information to achieve modeling of the query vector target features and trajectories, while completing the relocation of the old target and determining the changes in the appearance and disappearance of the target. Finally, the detection results of all current frames are output and used as the tracking vector of the next frame. The detection results are passed through the regional multi-layer perceptron (Region head) to output the visual features of the prediction box With coordinate information , as shown in Formula 2. The predicted box is matched with the real box using the Hungarian matching algorithm, and the classification loss and coordinate regression loss are calculated for each predicted box, as shown in Formula 3, and the parameters are updated. The classification loss uses focal loss to optimize the result of cross-modal contrastive learning fused by dot product, where The visual and language feature vectors are fused after the dot product. is the similarity of the annotated visual and language features, as shown in Formula 4; the target box uses L1 loss and GIoU loss as losses, To mark the coordinate information, Contains the same elements , where x and y are the coordinates of the center point of the rectangular box, and w and h are the width and length of the rectangular box. L1 loss is used to calculate the absolute error of the coordinates, and GIoU loss is used to calculate the error of the overlapping area of ​​the target box.

[0028] (Formula 2), (Formula 3), (Formula 4), in, Represents the visual features of the predicted box of the detection result, The coordinate information of the detection results, express The loss term weight, GIoU loss term weight, The dot product is a fusion of the visual and language similarity vectors. is the similarity vector of the annotated visual and language features, is the text feature vector.

[0029] For example, refer to Figure 2 As shown in FIG. 1 , a schematic diagram of the structure of a multi-target tracking model based on visual positioning is shown. Figure 2 As shown, the target tracking model includes a dual encoder module, a target detection module and a tracking module; the dual encoder module includes: a visual feature extractor and language feature extractor The dual encoder module uses pre-trained ViT-L and BERT as feature extraction models for images and texts respectively. The object detection module includes encoder and decoder. The object detection module uses the open set object detection model Grounding Dino 1.5 Pro, which is initialized with pre-trained weights, which are updated during the training process. The tracking module uses the MOTR tracker architecture in MOTRv2.

[0030] In one example, a classification loss and a coordinate regression loss are calculated based on the matching result of the detection result of each frame of image and the true result of each frame of image; The loss parameters of the target tracking model are calculated based on the classification loss, the coordinate regression loss, the predicted box category feature vector and the corresponding category feature vector label in the detection results of each frame image, and the predicted box coordinates and the corresponding coordinate labels in the detection results of each frame image.

[0031] In one example, referring to Figure 3 As shown, a flowchart of a multi-target video annotation method is shown, such as Figure 3 As shown, the following steps may be specifically included: Step S301, input the video to be annotated and the category to be annotated into the trained target tracking model; wherein the trained target tracking model is trained according to a target tracking model training method in any of the above schemes; in this embodiment, the target tracking model is built according to the feature extraction model, the target detection model and the tracking module. The target tracking model is built by the feature extraction model, the target detection model and the tracking module; Specifically, the feature extraction model can be composed of ViT-L and BERT dual encoders, and the model weights are frozen; the target detection model can be Grounding Dino 1.5 Pro, specifically including: an encoder, a query selection module and a decoder, and the weights are initialized through pre-training; the tracking module can be: the MOTR tracker architecture in MOTRv2, specifically including: a decoder, which updates the loss parameters according to the real-time training process.

[0032] For example, for all image frames in a video stream, the corresponding query text is the same.

[0033] Step S302, sampling the video frame by frame through the feature extraction module of the trained target tracking model to obtain the original image frame; for example, sampling the video frame by frame, is the number of image frames in the video, get the original image frame .

[0034] For example, this solution uses pre-trained ViT-L and BERT as feature extraction models for images and texts, respectively. With language feature extractor The weights in are always frozen and are not updated during training. The weights are trained using a public pre-training method.

[0035] Step S303, generating a prompt text description according to the category to be labeled; In this embodiment, the query category labels are concatenated into a prompt text description , the entire video corresponds to the same query text.

[0036] Step S304, inputting the prompt text description and the i-th frame image in the original image frame into the feature extraction module for feature extraction processing to obtain a text feature vector and an image feature vector; wherein i≥1; In this embodiment, the i-th frame image in the original image frame is input into the visual feature extractor Extract image features and obtain feature vector image encoding ; Input the prompt text description into the language feature extractor to extract language features and obtain feature vector text encoding ; Among them, the visual feature extractor and language feature extractor The weights are fixed values.

[0037] For example, for Frame, image-text pair Input the dual encoder to generate corresponding modal features.

[0038] Step S305, inputting the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain an i-th detection query vector; In this embodiment, the text feature vector and the image feature vector The feature fusion process is performed in the input detection encoder to obtain the processed text feature vector And the processed image feature vector . The processed text feature vector And the processed image feature vector Input the first query vector to the detection decoder for modality alignment processing to obtain the i-th detection query vector .

[0039] For example, the image features and text feature encoding , input to the encoder used to fuse image and text features. The encoder converts the image vector , text feature vector As input, the 6-layer deformable self-attention module (Deformable-Attention) and self-attention module (Self-Attention) in the encoder, as well as the two-layer cross-attention module (Image-to-text Cross-Attention, Text-to-image Cross-Attention) are used for feature fusion, and then input into the feed-forward network (FFN) respectively. Finally, the encoder outputs the enhanced image vector Vector with text .

[0040] The feature vector is input to the language-guided query selection model. The query selection module uses the image vector Vector with text As input, the output is a query vector that is more relevant to the text Query vector Through the self-attention mechanism, the weight of the query image content is dynamically adjusted according to the hint text, so as to locate the object of interest more accurately.

[0041] The feature vector is fed into the cross-modal decoder. The query vector Initialize the decoder input, pass through the self-attention layer, image cross-attention layer, and text cross-attention layer for modality alignment, and finally pass through the feed-forward network (FFN) to output the detection query vector . Detection query vector , the query vector of S222 is globally interacted with the frame through the cross-modal decoder using the cross-modal attention mechanism, and the selected visual and textual features are fused using the cross-modal attention mechanism to ensure the accuracy and consistency of the prediction.

[0042] Step S306, inputting the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; In this embodiment, the i-1th detection query vector Input into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector .

[0043] Step S307, processing the (i-1)th tracking query vector and the (i)th detection query vector by the tracking module of the trained target tracking model to obtain a detection result of the (i)th frame image; In this embodiment, the detection result of the previous frame is used as the tracking query vector , input to the current frame for prediction of the trajectory of the detected target. The concatenation is input into the decoder as the initial query vector , as shown in Formula 1.

[0044] (Formula 1), The aggregated vector After the self-attention module (Self-Attention) and the deformable self-attention module (Deformable-Attention). The self-attention module further integrates the historical tracking target and the current detection target, and the deformable self-attention module recalibrates the target coordinate information to achieve modeling of the query vector target features and trajectories, while completing the relocation of the old target and determining the changes in the appearance and disappearance of the target. Finally, the detection results of all current frames are output and used as the tracking vector of the next frame. .

[0045] The detection results are output through the regional multi-layer perceptron (Region head) to predict the visual features of the box With coordinate information , as shown in Formula 2.

[0046] (Formula 2), In this embodiment, the tracking module includes: a tracking encoder; and the i-th detection query vector Input into the tracking encoder for splicing processing to obtain the splicing result ; The splicing result is obtained by the tracking encoder Perform tracking processing to obtain the detection result of the i-th frame image.

[0047] Step S308, starting from the first frame of the video to be labeled, repeat the above steps until the last frame of the video to be labeled, so as to complete the pre-labeling of the video to be labeled.

[0048] In one example, referring to Figure 4 As shown, a flowchart of the steps of the visual feature extraction and language feature extraction method is shown, such as Figure 4 As shown, the following steps may be specifically included: The target tracking model includes: the feature extraction module, the target detection module and the tracking module; The feature extraction module includes: a visual feature extractor and a language feature extractor; Step S401, inputting the prompt text description into the language feature extractor to extract language features and obtain a text feature vector; Step S402, inputting the i-th frame image in the original image frame into the visual feature extractor to extract image features and obtain an image feature vector; wherein the weight of the visual feature extractor and the weight of the language feature extractor are fixed values ​​respectively.

[0049] For example, data is input to the dual encoder module. The video is sampled frame by frame. is the number of image frames in the video, get the original image frame .

[0050] Concatenate the corresponding category labels of the target in each image into a prompt text description For video Frame, single image Instead of its prompt phrase Through visual feature extractor , Language Feature Extractor , get their respective feature vector image encoding With text encoding .

[0051] In one example, referring to Figure 5 As shown, a flowchart of the steps of the processing process of the target detection module is shown, such as Figure 5 As shown, the following steps may be specifically included: The target detection module includes: a detection encoder, a query selection module and a detection decoder; Step S501, inputting the text feature vector and the image feature vector into the detection encoder for feature fusion processing to obtain a processed text feature vector and a processed image feature vector; In one example, referring to Figure 6 As shown, a schematic diagram of the target detection module structure is shown. Figure 6 As shown, it specifically includes: the target detection module structure is as follows Figure 6 As shown in the figure, the Self-Attention part is the processing module of the text dimension, the Deformable-Attention part is the processing module of the visual dimension, and the white two-layer cross-attention module part is the multimodal processing module.

[0052] Eigenvector Image Coding With text encoding Input is used to fuse the image and text features into an encoder. The encoder transforms the image vector , text feature vector As input, the 6-layer deformable self-attention module (Deformable-Attention) and self-attention module (Self-Attention) in the encoder, as well as the two-layer cross-attention module (Image-to-text Cross-Attention, Text-to-image Cross-Attention) are respectively input into the feed-forward network (FFN), and the encoder finally outputs the enhanced image vector Vector with text .

[0053] Step S502, inputting the processed text feature vector and the processed image feature vector into the query selection module for processing to obtain a first query vector; For example, the feature vector is input to a language-guided query selection model. The query selection module uses the image vector Vector with text As input, the output is a query vector that is more relevant to the text .

[0054] Step S503: input the first query vector into the detection decoder for modality alignment processing to obtain an i-th detection query vector.

[0055] In one example, referring to Figure 6 As shown, a schematic diagram of the target detection module structure is shown. Figure 6 As shown in FIG, the query selection module uses a language-guided query selection model as the feature vector input. The query selection module uses an image vector Vector with text As input, the output is a query vector that is more relevant to the text

[0056] Query Vector Input to the cross-modal decoder. Query vector Initialize the decoder input, pass through the self-attention layer, image cross-attention layer, and text cross-attention layer for modality alignment, and finally pass through the feed-forward network (FFN) to output the detection query vector .

[0057] This method retains the existence of the detector, increases the interaction with the historical frames, and transmits the trajectory information frame by frame. The iteration across time is more conducive to the feature alignment between frames. While retaining the detection performance, the ability of target matching and association is improved. In particular, it is more adaptable to scenes with more complex motion trajectories, such as changes in the speed and direction of a single target, or overlap and occlusion between targets.

[0058] In one example, referring to Figure 7 As shown, a flow chart of the steps of detecting the encoder is shown, such as Figure 7 As shown, the following steps may be specifically included: The detection encoder comprises: a first deformable self-attention module, a first self-attention module, an image-text cross-attention module, a text-image cross-attention module and an encoding feed-forward network layer; Step S701, inputting the text feature vector into the first self-attention module for processing to obtain a first encoding processing result; Step S702, inputting the image feature vector into the first deformable self-attention module for processing to obtain a second encoding processing result; Step S703, taking the first encoding processing result as K and V, and taking the second encoding processing result as Q, and inputting them into the image-text cross attention module for processing to obtain processed K, V and Q; Step S704, inputting the processed K, V and Q into the text-image cross attention module for processing to obtain a third encoding processing result; Step S705: input the third encoding processing result into the encoding feedforward network layer for processing to obtain a processed text feature vector and a processed image feature vector.

[0059] In one example, referring to Figure 8 As shown, a flowchart of the steps of the processing process of the detection decoder is shown, such as Figure 8 As shown, the following steps may be specifically included: The detection decoder comprises: a self-attention layer, an image cross-attention layer, a text cross-attention layer and a decoding feed-forward network layer; Step S801, inputting the first query vector into the self-attention layer for processing to obtain a first decoding processing result; Step S802, taking the processed image feature vector as K and V, taking the first decoding processing result as Q, inputting them into the image cross attention layer for processing, and obtaining a second decoding processing result; Step S803, taking the processed text feature vector as K and V, taking the second decoding processing result as Q, and inputting them into the text cross attention layer for processing to obtain a third decoding processing result; Step S804: input the third decoding processing result into the decoding feed-forward network layer for processing to obtain the i-th detection query vector.

[0060] In one example, referring to Fig. 9 As shown, a flowchart of the steps of the processing process of the tracking module is shown, such as Fig. 9 As shown, the following steps may be specifically included: The tracking module includes: a second self-attention module, a second deformable self-attention module and a regional multi-layer perceptron; Step S901, inputting the i-1th tracking query vector of the i-1th frame image and the i-th detection query vector into the tracking encoder for splicing processing to obtain a splicing result; Step S902, inputting the splicing result into the second self-attention module to obtain a first tracking result; Step S903, inputting the first tracking result and the i-th detection query vector into the second deformable self-attention module for processing to obtain a second tracking result; Step S904: input the second tracking result into the regional multi-layer perceptron for processing to obtain the detection result of the i-th frame image.

[0061] For example, refer to Fig.10 As shown, a schematic diagram of the tracking module structure is shown. Fig.10 As shown, the following steps may be specifically included: The detection result of the previous frame is used as the tracking query vector , input to the current frame for prediction of the trajectory of the detected target. The concatenation is input into the decoder as the initial query vector ; The aggregated vector After the self-attention module (Self-Attention) and the deformable self-attention module (Deformable-Attention). The self-attention module further integrates the historical tracking target and the current detection target, and the deformable self-attention module recalibrates the target coordinate information to achieve modeling of the query vector target features and trajectories, while completing the relocation of the old target and determining the changes in the appearance and disappearance of the target. Finally, the detection results of all current frames are output and used as the tracking vector of the next frame. .

[0062] The detection results are passed through the regional multi-layer perceptron (Region head) to output the visual features of the prediction box With coordinate information .

[0063] This method decomposes target tracking into two subtasks: target detection and tracking. The tracking module uses the spatiotemporal positioning and feature encoding provided by the detection module for further optimization and training. The exchange of information enables the tracking module to focus more on target re-identification, thereby alleviating the conflict between the two related tasks. In addition, the decoupled subtasks can be pre-trained on their respective datasets, or the modules in them can be replaced and applied to other scenarios and tasks.

[0064] Multimodal fusion and learning are performed using contrastive training between language and vision. Visual localization aligns the data-rich semantic information from text with image information. This synergy enables the model to learn language-aware visual representations and better guide object detection and association tasks.

[0065] In one example, a trained target tracking model is used to annotate a multi-target unmanned driving video to obtain a video sample dataset; Adding the video sample dataset to a driving video dataset used for training an unmanned driving model to obtain an updated driving video dataset; The unmanned driving model is trained according to the updated driving video data set to obtain a trained unmanned driving model.

[0066] For example, a video pre-labeling method based on multi-target tracking may include: Build a target tracking model based on visual positioning, including a dual encoder module, a target detection module and a tracking module.

[0067] Model training includes: using a dataset for multi-target tracking tasks for training, where each frame in the video contains the coordinates and category of the target. Starting from the first frame, the image-category text description forms a text pair, which is input into the target tracking network in sequence, and finally the bounding box and category of each frame target are obtained.

[0068] Data is input to the dual encoder module. Frame image-text pairs Input to the visual feature extractor , Language Feature Extractor , get the image encoding and text encoding ; The feature vector is input to the target detection module. The target detection module encodes the image and text encoding As input, the multimodal fusion encoder obtains cross-modal image and text features; this feature is then input into the decoder to output the detection vector ; The feature vector is input to the tracking module. The tracking module detects the vector Tracking vector with the previous frame As input, get the prediction result and tracking vector of the current frame ; Construct a loss function. Use the binary Hungarian matching algorithm to associate annotations and prediction boxes, and use contrast loss to measure the localization and classification performance of each object. Repeat the above steps until the last frame of the video.

[0069] Model inference includes: The data is input to the dual encoder module. Frame image With query text , forming image-text pairs Input to dual encoder module; The feature vector is input to the target detection module. The output image is encoded and text encoding , the detection query vector of each frame is obtained through the target detection model ; The feature vector is input to the tracking module. The detection query vector of the current frame The tracking query vector of the previous frame Input to the tracking model to obtain the tracking query vector for this frame And detect target boxes and categories; Repeat the above steps until the last frame of the video.

[0070] Specifically, for the video stream to be processed, the image sampling interval can be set during model training and inference. This description is only for the scenario where frame skipping is not set.

[0071] For example, the model inference process includes: The video to be annotated and the target category to be annotated are input into the trained visual positioning-based target tracking model. For all image frames in the video stream, the corresponding query text is the same.

[0072] Data input to the dual encoder module.

[0073] The video is sampled frame by frame. is the number of image frames in the video, get the original image frame .

[0074] Concatenate the query category labels into a prompt text description , the entire video corresponds to the same query text.

[0075] For Frame, image-text pair Input the dual encoder to generate corresponding modal features.

[0076] The feature vector is input to the target detection module. Image features and text feature encoding , the detection query vector of the frame object is obtained through the target detection model .

[0077] The feature vector is input to the tracking module. The detection query vector of the current frame object The tracking query vector of the object in the previous frame Input to the tracking model to obtain the target box and category detected in this frame, as well as the tracking query vector for relocalization in the next frame .

[0078] Starting from the first frame of the video, repeat the above steps until the last frame of the video.

[0079] Compared with the detection-based tracking method, this method retains the existence of the detector, increases the interaction with the historical frames, and the trajectory information is transmitted frame by frame. The iteration across time is more conducive to the alignment of features between frames. While retaining the detection performance, the ability of target matching and association is improved. In particular, it is more adaptable to scenes with more complex motion trajectories, such as changes in the speed and direction of a single target, or overlap and occlusion between targets. Thus, it has stable multi-target tracking performance.

[0080] This method decomposes target tracking into two subtasks: target detection and tracking. The tracking module uses the spatiotemporal positioning and feature encoding provided by the detection module for further optimization and training. The exchange of information enables the tracking module to focus more on target re-identification, thereby alleviating the conflict between the two related tasks. In addition, the decoupled subtasks can be pre-trained on their respective datasets, or the modules in them can be replaced and applied to other scenarios and tasks.

[0081] This method uses the model weights obtained by training on tens of millions of data for initialization. While accelerating the training convergence speed, its excellent generalization ability enables it to be transferred to various types and numbers of targets, as well as to different scenarios and environments. In addition, when there is little or no training data, it can also be quickly applied to new tasks and fields, and effectively expanded to new target types.

[0082] This solution uses language to guide the target tracking task and uses contrast training between language and vision for multimodal fusion and learning. Visual localization aligns the rich semantic information from text data with image information. This synergy encourages the model to learn language-aware visual representations to better guide target detection and association tasks. This allows the model to maintain strong performance in the current data representation space while obtaining better zero-shot and few-shot transfer learning capabilities.

[0083] This solution processes visual and language data based on the Transformer architecture. This allows the entire model to inherit the Transformer's strong scalability and better capture and express complex features. The consistent structure can flexibly aggregate the two modal features at each stage to obtain a similar and compact semantic space, making it easier for the network to learn effective feature representations, thereby improving convergence speed and overall performance. The Transformer architecture represents the attention architecture of the target detection module and the tracking module.

[0084] This solution fine-tunes the open-set target detection model to make the model have strong data migration capabilities; and introduces text modality through visual positioning for comparative learning to improve the understanding of language perception and image features. The transfer learning performance it possesses makes it convenient for it to be generalized to more types of targets by only fine-tuning on a small amount of data in practical applications. And through text prompts, zero-sample detection and tracking can be achieved to a certain extent, providing solutions for pre-labeling in different fields.

[0085] In actual use, this solution obtains the category and target box of the relevant objects in each frame by inputting the query text. It adopts a multi-target tracking method based on query tracking, which enables it to handle more complex motion trajectories; and the tracking model optimizes learning based on the prior information specially provided by the target detection model, which collaboratively improves the accuracy of target detection and tracking. It reduces the increase in manual correction workload caused by missed targets or inaccurate entity category recognition, and obtains more high-quality labeled data. It reduces the transfer cost and training cost of the model in different scenarios, realizes rapid tuning, and improves the efficiency of model iteration.

[0086] In one example, referring to Fig.11 As shown, a structural block diagram of a target tracking model training device according to an embodiment of the present invention is shown. Fig.11 As shown, the device may specifically include the following modules: Model building module 1001, used to build a target tracking model; The training sampling module 1002 is used to input the video sample data set of the multi-target tracking task into the feature extraction module of the target tracking model, perform frame-by-frame sampling, and obtain a preset number of original sample image frames; A training splicing module 1003 is used to serially splice multiple target category labels in the i-th frame image of the original sample image frame into an i-th prompt text description; A visual feature extraction module 1004 is used to encode the i-th frame image through a visual feature extractor to obtain a feature vector of the i-th frame image; A language feature extraction module 1005 is used to encode the i-th prompt text description through a language feature extractor to obtain an i-th frame text feature vector; A training detection module 1006 is used to process the image feature vector of the i-th frame and the text feature vector of the i-th frame through the target detection module of the target tracking model to obtain a detection query vector of the i-th frame; The training tracking module 1007 is used to input the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; and then process the i-1th tracking query vector and the i-th frame detection query vector through the tracking module of the target tracking model to obtain the detection result of the i-th frame image; A loop training module 1008 is used to repeat the above steps from the first frame of the video sample data set to the last frame of the video sample data set to obtain the detection result of each frame image of the video sample data set; A training calculation module 1009 is used to calculate the loss parameter of the target tracking model according to the prediction box category feature vector and the corresponding category feature vector label in the detection result of each frame image of the video sample data set, and the prediction box coordinates and the corresponding coordinate labels in the detection result of each frame image of the video sample data set; The training optimization module 1010 is used to train the target tracking model to be trained according to the loss parameter to obtain a trained target tracking model.

[0087] The target tracking model training device provided in the above embodiment can implement the technical solution described in the above target tracking model training method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above target tracking model training method embodiment, which will not be repeated here.

[0088] In an example, a multi-target video annotation device according to an embodiment of the present invention may specifically include the following modules: The input module is used to input the video to be labeled and the category to be labeled into the trained target tracking model; wherein the trained target tracking model is trained according to any of the above-mentioned target tracking model training methods; The sampling module is used to sample the video frame by frame through the feature extraction module of the trained target tracking model to obtain the original image frame; The text description generation module is used to generate a prompt text description according to the category to be labeled; The feature extraction module is used to input the prompt text description and the i-th frame image in the original image frame into the feature extraction module for feature extraction processing to obtain a text feature vector and an image feature vector; wherein i≥1; The detection module is used to input the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain an i-th detection query vector; The tracking module is used to input the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; The tracking module is used to process the i-1th tracking query vector and the i-th detection query vector through the tracking module of the trained target tracking model to obtain the detection result of the i-th frame image; The labeling module is used to repeat the above steps starting from the first frame of the video to be labeled until the last frame of the video to be labeled, so as to complete the pre-labeling of the video to be labeled.

[0089] The multi-target video annotation device provided in the above embodiment can implement the technical solution described in the above embodiment of a multi-target video annotation method. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of a multi-target video annotation method, which will not be repeated here.

[0090] The target tracking model training method, multi-target video labeling method and device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A target tracking model training method, characterized in that: include: Build a target tracking model; Inputting the video sample data set of the multi-target tracking task into the feature extraction module of the target tracking model, sampling frame by frame, and obtaining a preset number of original sample image frames; Concatenate multiple target category labels in the i-th frame image of the original sample image frame into an i-th prompt text description; Encode the i-th frame image through the visual feature extractor to obtain the i-th frame image feature vector; Encode the i-th prompt text description by a language feature extractor to obtain an i-th frame text feature vector; Processing the image feature vector of the i-th frame and the text feature vector of the i-th frame through the target detection module of the target tracking model to obtain a detection query vector of the i-th frame; Inputting the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; Processing the i-1th tracking query vector and the i-th frame detection query vector through the tracking module of the target tracking model to obtain a detection result of the i-th frame image; Repeat the above steps from the first frame of the video sample data set to the last frame of the video sample data set to obtain the detection result of each frame image of the video sample data set; Calculate the loss parameter of the target tracking model according to the predicted box category feature vector and the corresponding category feature vector label in the detection result of each frame image of the video sample data set, and the predicted box coordinates and the corresponding coordinate labels in the detection result of each frame image of the video sample data set; The target tracking model to be trained is trained according to the loss parameter to obtain a trained target tracking model.

2. A target tracking model training method according to claim 1, characterized in that: The step of calculating the loss parameter of the target tracking model according to the predicted box category feature vector and the corresponding category feature vector label in the detection result of each frame image of the video sample data set, and the predicted box coordinates and the corresponding coordinate labels in the detection result of each frame image of the video sample data set, specifically includes: According to the detection result of each frame of image and the matching result of the real result of each frame of image, a classification loss and a coordinate regression loss are calculated; The loss parameters of the target tracking model are calculated based on the classification loss, the coordinate regression loss, the predicted box category feature vector and the corresponding category feature vector label in the detection results of each frame image, and the predicted box coordinates and the corresponding coordinate labels in the detection results of each frame image.

3. A multi-target video labeling method, characterized in that: include: Inputting the video to be labeled and the category to be labeled into a trained target tracking model; wherein the trained target tracking model is trained according to a target tracking model training method according to claim 1 or 2; The video is sampled frame by frame by using the feature extraction module of the trained target tracking model to obtain original image frames; Generate a prompt text description according to the category to be labeled; Input the prompt text description and the i-th frame image in the original image frame into the feature extraction module for feature extraction processing to obtain a text feature vector and an image feature vector; wherein i≥1; Inputting the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain an i-th detection query vector; Inputting the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; Processing the (i-1)th tracking query vector and the (i)th detection query vector by the tracking module of the trained target tracking model to obtain a detection result of the (i)th frame image; Starting from the first frame of the video to be labeled, the above steps are repeated until the last frame of the video to be labeled, so as to complete the pre-labeling of the video to be labeled.

4. A multi-target video labeling method according to claim 3, characterized in that: The target tracking model includes: the feature extraction module, the target detection module and the tracking module; The feature extraction module includes: a visual feature extractor and a language feature extractor; The prompt text description and the i-th frame image in the original image frame are respectively input into the feature extraction module for feature extraction processing to obtain a text feature vector and an image feature vector, specifically including: Inputting the prompt text description into the language feature extractor to extract language features and obtain a text feature vector; The i-th frame image in the original image frame is input into the visual feature extractor to extract image features, so as to obtain an image feature vector; wherein the weight of the visual feature extractor and the weight of the language feature extractor are respectively fixed values.

5. A multi-target video labeling method according to claim 3, characterized in that: The target detection module includes: a detection encoder, a query selection module and a detection decoder; The step of inputting the text feature vector and the image feature vector into the target detection module of the trained target tracking model for processing to obtain the i-th detection query vector specifically includes: Inputting the text feature vector and the image feature vector into the detection encoder for feature fusion processing to obtain a processed text feature vector and a processed image feature vector; Inputting the processed text feature vector and the processed image feature vector into the query selection module for processing to obtain a first query vector; The first query vector is input into the detection decoder for modality alignment processing to obtain an i-th detection query vector.

6. A multi-target video labeling method according to claim 5, characterized in that: The detection encoder comprises: a first deformable self-attention module, a first self-attention module, an image-text cross-attention module, a text-image cross-attention module and an encoding feed-forward network layer; The step of inputting the text feature vector and the image feature vector into the detection encoder for feature fusion processing to obtain the processed text feature vector and the processed image feature vector specifically includes: Inputting the text feature vector into the first self-attention module for processing to obtain a first encoding processing result; Inputting the image feature vector into the first deformable self-attention module for processing to obtain a second encoding processing result; The first encoding processing result is used as K and V, and the second encoding processing result is used as Q, and inputted into the image-text cross attention module for processing to obtain processed K, V and Q; The processed K, V and Q are input into the text-image cross attention module for processing to obtain the third encoding processing result; The third encoding processing result is input into the encoding feedforward network layer for processing to obtain a processed text feature vector and a processed image feature vector.

7. A multi-target video labeling method according to claim 5, characterized in that: The detection decoder comprises: a self-attention layer, an image cross-attention layer, a text cross-attention layer and a decoding feed-forward network layer; The step of inputting the first query vector into the detection decoder for modality alignment processing to obtain an i-th detection query vector specifically includes: Inputting the first query vector into the self-attention layer for processing to obtain a first decoding processing result; The processed image feature vector is used as K and V, and the first decoding processing result is used as Q, and inputted into the image cross attention layer for processing to obtain a second decoding processing result; The processed text feature vector is used as K and V, and the second decoding processing result is used as Q, and inputted into the text cross attention layer for processing to obtain a third decoding processing result; The third decoding processing result is input into the decoding feedforward network layer for processing to obtain an i-th detection query vector.

8. A multi-target video labeling method according to claim 3, characterized in that: The tracking module includes: a second self-attention module, a second deformable self-attention module and a regional multi-layer perceptron; Processing the i-1th tracking query vector and the i-th detection query vector of the i-1th frame image through the tracking module of the trained target tracking model to obtain the detection result of the i-th frame image specifically includes: Inputting the i-1th tracking query vector of the i-1th frame image and the i-th detection query vector into the tracking encoder for splicing processing to obtain a splicing result; Inputting the splicing result into the second self-attention module to obtain a first tracking result; Inputting the first tracking result and the i-th detection query vector into the second deformable self-attention module for processing to obtain a second tracking result; The second tracking result is input into the regional multi-layer perceptron for processing to obtain the detection result of the i-th frame image.

9. A multi-target video labeling method according to claim 3, characterized in that: Also includes: Use the trained target tracking model to annotate the multi-target unmanned driving videos and obtain a video sample dataset; Adding the video sample dataset to a driving video dataset used for training an unmanned driving model to obtain an updated driving video dataset; The unmanned driving model is trained according to the updated driving video data set to obtain a trained unmanned driving model.

10. A target tracking model training device, characterized in that: include: Model building module, training sampling module, training splicing module, visual feature extraction module, language feature extraction module, training detection module, training tracking module, loop training module, training calculation module and training optimization module; The model building module is used to build a target tracking model; The training sampling module is used to input the video sample data set of the multi-target tracking task into the feature extraction module of the target tracking model, perform frame-by-frame sampling, and obtain a preset number of original sample image frames; The training splicing module is used to serially splice multiple target category labels in the i-th frame image of the original sample image frame into the i-th prompt text description; The visual feature extraction module is used to encode the i-th frame image through the visual feature extractor to obtain the i-th frame image feature vector; The language feature extraction module is used to encode the i-th prompt text description through a language feature extractor to obtain the i-th frame text feature vector; The training detection module is used to process the i-th frame image feature vector and the i-th frame text feature vector through the target detection module of the target tracking model to obtain the i-th frame detection query vector; The training tracking module is used to input the i-1th frame detection query vector of the i-1th frame image into the tracking module of the target tracking model for processing to obtain the i-1th tracking query vector; Then, the tracking module of the target tracking model processes the i-1th tracking query vector and the i-th frame detection query vector to obtain a detection result of the i-th frame image; The loop training module is used to repeat the above steps from the first frame of the video sample data set to the last frame of the video sample data set to obtain the detection result of each frame image of the video sample data set; The training calculation module is used to calculate the loss parameter of the target tracking model according to the prediction box category feature vector and the corresponding category feature vector label in the detection result of each frame image of the video sample data set, and the prediction box coordinates and the corresponding coordinate labels in the detection result of each frame image of the video sample data set; The training optimization module is used to train the target tracking model to be trained according to the loss parameter to obtain the trained target tracking model.

Images