A video object tracking system and method based on contrastive learning

Abstract

A video target tracking system and method based on contrast learning, the system comprising: a feature extraction module, a contrast learning head, a tracking head, a dynamic template updating module and a loss function calculation module; the method comprising: inputting a search frame, a template and a dynamic template image into the feature extraction module to obtain an output search frame contrast feature, a template contrast feature and an output search frame feature; inputting the search frame contrast feature and the template contrast feature into the contrast learning head for feature mapping, inputting the search frame feature into the tracking head to output a target bounding box, calculating a contrast learning loss function for the feature mapping result, calculating a classification loss and a regression loss function for the target bounding box, obtaining a contrast learning, classification loss and regression loss function loss value, and then summing to obtain a total loss value, and using the total loss value for back propagation; using the current search frame confidence for updating the dynamic template updating module; the present application has the advantages of high tracking precision and wide adaptability.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and specifically relates to a video target tracking system and method based on contrastive learning. Background Technology

[0002] Semantic information is crucial for target tracking tasks. In complex scenarios, the semantic information of a target may involve different object categories, various changes in the target, and different scene contexts. This makes it difficult for existing methods to effectively extract and utilize the semantic information of the target in complex environments, and makes it difficult to track the target continuously, stably, and accurately.

[0003] With the continuous development of computer technologies such as software engineering and machine learning, people are drawing upon their own visual perception of the world to process and understand videos and images through computer vision, and then using the results to inform a series of subsequent decisions. Object tracking, as one of the most popular tasks in computer vision, is not only extensively studied in theory, but also widely applied in fields such as security monitoring, autonomous driving, and virtual reality. Object tracking is divided into single-object tracking and multi-object tracking.

[0004] Zeng et al. proposed DRCI (Zeng, Dan, et al. "Towards Discriminative Representations with Contrastive Instances for Real-Time UAV Tracking." 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023.), applying contrastive learning to the field of UAV target tracking. Its positive samples are obtained by randomly sampling two frames from the video sequence containing the template image, while negative samples are samples from different video sequences. The input includes the template image, positive sample image, and search frame image. A Siamese network architecture based on a convolutional neural network is used to extract features from the input, resulting in template features, positive sample features, and search frame features. The template features and search frame features are then cross-correlated to obtain cross-correlation features, which are then fed into the tracking task head to obtain the target bounding box. The template features and positive sample features are then fed into the contrastive learning head for feature mapping, and the model parameters are optimized through calculation and backpropagation using the contrastive learning loss function. However, its positive samples consist only of the template image and two frames from the same video, resulting in a limited number of positive samples. This means the model's ability to extract features from similar targets is not sufficiently enhanced, failing to adequately capture target-related semantic information. Secondly, its use of a convolutional neural network-based backbone for feature extraction, compared to a Transformer-based backbone, leads to the loss of target-related details during feature extraction, resulting in poor tracking performance in complex environments.

[0005] In summary, the existing technology has the following drawbacks:

[0006] First, existing Transformer-based target tracking methods lack the extraction and utilization of semantic information. In complex scenes, the semantic information of a target may involve different object categories, various changes in the target, and different scene contexts. This makes it difficult to effectively distinguish the target from the background or other similar objects, thus hindering the achievement of continuous, stable, and accurate target tracking.

[0007] Second, existing methods only associate the current layer template features with the current layer search frame features, or only use deep template features to associate with search frame features. However, the shape of the target in the search frame may differ significantly from that in the template, and the diversity of target information in a single-level template feature is insufficient to meet the model's requirement for diverse target information.

[0008] Third, existing tracking methods do not consider the different amounts of discriminative information required for target localization in images with different degrees of background complexity when performing spatial pruning on the features of the search frame. They prune according to a fixed ratio, which leads to the loss of effective discriminative information in images with complex backgrounds and the excessive retention of background redundancy information in images with simple backgrounds. This makes the model prone to confusion with the target and its surrounding environment. Summary of the Invention

[0009] To overcome the shortcomings of the prior art, the present invention aims to provide a video target tracking system and method based on contrastive learning. By introducing a contrastive learning strategy into a Transformer-based target tracking model, and designing a contrastive learning head and loss function, the semantic information of positive and negative sample labels is fully utilized. This enhances the feature similarity between similar targets while reducing the feature similarity between different targets, thereby improving the model's ability to extract category semantic information. Furthermore, a cross-layer semantic association module associates multi-level semantic features, resulting in richer and more diverse target-related semantic information. Additionally, redundant information is pruned from search frame features, adaptively and effectively reducing the impact of complex backgrounds on the target based on image complexity, thus effectively improving target saliency. The present invention can provide high tracking accuracy while ensuring inference speed, and improves the model's ability to extract and utilize semantic information, thereby ensuring the model's generalization ability in various complex scenarios.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A video target tracking system based on contrastive learning includes a feature extraction module, a contrastive learning head, a tracking head, a dynamic template update module, and a loss function calculation module; wherein:

[0012] The feature extraction module includes a block embedding layer, a semantic self-association module, a cross-layer semantic association module, and a redundancy pruning module, used to extract features from the input image, wherein:

[0013] The block embedding layer is used to transform the input template image, dynamic template image, and search frame image into template features, dynamic template features, and search frame features, respectively.

[0014] The semantic self-association module is used to capture the global contextual relationships between template splicing features and search frame features themselves, and extract features with stronger representation capabilities.

[0015] The cross-layer semantic association module is used to associate search frame features with multi-layer template concatenation features to extract richer template target-related semantic information;

[0016] The redundancy information pruning module is used to prune redundant backgrounds in the search frame features, reduce the number of background features, and reduce the impact of complex backgrounds on the target.

[0017] The contrast learning head is used to perform feature mapping between the contrast features of the search frame output by the feature extraction module and the contrast features of the template. The contrast learning loss function is calculated using the feature mapping results of the contrast learning head to enhance the feature extraction module's ability to extract category semantic information and its feature representation capabilities.

[0018] The tracking head includes a scoring head, a scale head, and a bias head. The output of the scoring head is used to predict the center position of the target, and the maximum value of the center position is used to predict the confidence of the current search frame. The output of the bias head is used to compensate for the discretization error caused by the resolution. The output of the scale head is used to predict the width and height of the target. The center target position output by the scoring head, the discretization error predicted by the bias head, and the target width and height output by the scale head are used together to predict the target bounding box.

[0019] The dynamic template update module controls the update of the dynamic template based on the confidence level of the current search frame output by the score head, so as to adapt to changes in the appearance of the target during the tracking process.

[0020] The loss function calculation module includes classification loss function, regression loss function, and contrastive learning loss function. The classification loss function and regression loss function are calculated based on the predicted target bounding boxes output by the score head, scale head, and bias head. The contrastive learning loss function is calculated based on the feature mapping results of the contrastive learning head. During training, the sum of the classification loss function, regression loss function, and contrastive learning loss function is used for backpropagation to optimize the network parameters.

[0021] A video target tracking method based on contrastive learning includes the following steps:

[0022] Step 1: Input the search frame image, template image, and dynamic template image into the feature extraction module for processing. The feature extraction module includes a block embedding layer, a semantic self-association module, a cross-layer semantic association module, and a redundancy pruning module to obtain the output search frame comparison features s. c Template comparison features t c Output search frame features s o ;

[0023] Step 2, compare the search frame features s obtained in Step 1. c Feature t compared with template c All inputs are fed into the contrast learning head for feature mapping, and the feature mapping results of the contrast learning head are output.

[0024] Step 3, use the search frame features s obtained in Step 1. oThe input tracking head includes a scoring head, a scale head, and a bias head. The output of the scoring head is used to predict the target center position, and the maximum value of the target center position is used to predict the confidence of the current search frame. The output of the bias head is used to compensate for the discretization error caused by the resolution. The output of the scale head is used to predict the width and height of the target. The center target position output by the scoring head, the discretization error predicted by the bias head, and the target width and height output by the scale head are used together to predict the target bounding box.

[0025] Step 4: During training, the contrastive learning head feature mapping result output in Step 2 is used to calculate the loss function through the contrastive learning loss function in the loss function calculation module. The target bounding box output in Step 3 is used to calculate the classification loss function and the regression loss function to obtain the loss values ​​of the contrastive learning loss function, the classification loss function and the regression loss function. These values ​​are then summed to obtain the total loss value. The total loss value is used for backpropagation to optimize the network parameters.

[0026] Step 5: Use the current search frame confidence score output from the score head in Step 3 to control whether to perform dynamic template update; every M frames, select the search frame with the highest confidence score among the M frames and crop it as a new dynamic template.

[0027] The specific method for step 1 is as follows:

[0028] Step 1.1: The input consists of a search frame image, a template image, and a dynamic template image. The template image and the dynamic template image are both 3×H in size. t ×W t The size of the search frame image is 3×H s ×W s H t and W t H represents the height and width of the template, respectively. s and W s These represent the height and width of the search frame, respectively.

[0029] Step 1.2: The search frame image, template image, and dynamic template image input in Step 1.1 are fed into the block embedding layer to obtain search frame features s, template features t, and dynamic template features d, with sizes N respectively. s ×C、N t ×C、N d ×C, where C represents the number of channels for the feature, and N s =N d N s N t N d These represent the number of features in the search frame features, template features, and dynamic template features, respectively.

[0030] Step 1.3: Add the search frame features, template features, and dynamic template features obtained in Step 1.2 to the positional encoding respectively to obtain s1 and t. t t d s1, t t t d These represent search frame features, template features, and dynamic template features, respectively, with dimensions N. s ×C、N t ×C、N t ×C;

[0031] Step 1.4, for the template feature t obtained in step 1.3 t Dynamic template features t d The templates are spliced ​​together to obtain the splicing feature t1, with a size of 2N. t ×C;

[0032] Step 1.5: Input the template concatenation feature t1 obtained in Step 1.4 into the n-layer semantic self-association module to obtain the template concatenation feature t2, which is the template comparison feature t. c The size is 2N t ×C;

[0033] Step 1.6: Input the template concatenation feature t2 obtained in Step 1.5 into the semantic self-association module to obtain the template concatenation feature t3 with a size of 2N. t ×C;

[0034] Step 1.7: Input the template concatenation feature t3 obtained in Step 1.6 into the semantic self-association module to obtain the template concatenation feature t4 with a size of 2N. t ×C;

[0035] Step 1.8: Input the template concatenation feature t4 obtained in Step 1.7 into the semantic self-association module to obtain the template concatenation feature t5 with a size of 2N. t ×C;

[0036] Step 1.9: Input the search frame feature s1 obtained in Step 1.3 into the n-layer semantic self-association module to obtain the search frame feature s2, which is the search frame comparison feature s. c Size N s ×C;

[0037] Step 1.10: The search frame feature s2 obtained in Step 1.9, the template concatenation feature t2 obtained in Step 1.5, and the template concatenation feature t5 obtained in Step 1.8 are fed into the cross-layer semantic association module to obtain the search frame feature s3, with a size of N. s ×C;

[0038] Step 1.11: The search frame feature s3 obtained in Step 1.10, the template concatenation feature t3 obtained in Step 1.6, and the template concatenation feature t5 obtained in Step 1.8 are fed into the cross-layer semantic association module to obtain the search frame feature s4, with a size of N. s ×C;

[0039] Step 1.12: The search frame feature s4 obtained in Step 1.11, the template concatenation feature t4 obtained in Step 1.7, and the template concatenation feature t5 obtained in Step 1.8 are fed into the cross-layer semantic association module to obtain the search frame feature s5, with a size of N. s ×C;

[0040] Step 1.13: Input the search frame feature s5 obtained in step 1.12 into the redundancy information pruning module to obtain the output search frame feature s o Size N s1 ×C.

[0041] The block embedding layer includes a 3×3 convolution operation, a layer normalization operation, and a dimension transformation operation. The dimension transformation operation transforms the feature size from H×W×C to N×C, where N=H×W.

[0042] The semantic self-association module is computed by a multi-head attention and a feedforward neural network (FFN), as shown in the following formula:

[0043] O = X in +MultiHead(X in ,X in ,X in )

[0044] Y out =O + FFN(O)

[0045] Among them, X in Y represents the input feature. out Indicates output features;

[0046] The calculation process for multi-head attention is shown in the following formula:

[0047]

[0048] H i =Attention(QW i Q ,KW i K VW i V )

[0049] Attention(Q,K,V)=softmax(A(Q,K))V

[0050]

[0051] Where, n h W represents the number of heads. i Q W i K W i V W i O Let Q, K, V, and O represent the parameters of the linear mappings, respectively, and let A(Q,K) represent the attention weight matrix. k This represents the dimension of the KEY vector obtained from the linear mapping K;

[0052] The feedforward neural network consists of two linear layers with a ReLU function inserted in the middle. The calculation formula is as follows:

[0053] FFN(O)=ReLU(OW1+b1)W2+b2

[0054] Where W1 and W2 are the weight matrices of the linear layer, and b1 and b2 are the basis vectors;

[0055] The formula for calculating the linear rectified function ReLU is as follows:

[0056]

[0057] Where 'a' represents an element in the input features;

[0058] The cross-layer semantic association module takes template feature t as its input. i The i-th layer is the current layer's search frame features s i The last layer template feature t5, 2<=i<=5; its calculation process is shown in the formula:

[0059] O = s i +MultiHead(s i ,[s i ;t i ;t5],[s i ;t i ;t5])

[0060] Y out =O + FFN(O)

[0061] Among them, [s i ;t i ;t5] indicates that t i s i , t5 spliced ​​together;

[0062] The redundant information pruning module takes the attention weight matrix A(Q,K) corresponding to the search frame feature S5, performs average pooling and maximum-minimum normalization operations, and uses it as the confidence score, i.e., energy, of each feature in the search frame feature S5, with a value in the range [0-1]. The feature confidence scores are sorted from largest to smallest to obtain the confidence score. Then, the confidence scores are summed to obtain the total energy E of the candidate region, and ρ%×E is taken as the energy threshold, where ρ is the proportion of energy retained. Finally, the top K features with the highest energy in the candidate region are retained to satisfy... And discard the remaining features.

[0063] In step 2, the contrastive learning head consists of region-of-interest pooling, 3×3 convolution, ReLU, 1×1 convolution, and batch normalization; the search frame contrastive features s c Feature t compared with template c The data are processed separately using region-of-interest pooling, 3×3 convolution, ReLU, 1×1 convolution, and batch normalization, and then concatenated together for comparison learning loss function calculation.

[0064] In step 3, the scoring head, scale head, and bias head are all implemented using 5 layers of 1×1 convolutional blocks; the first four layers consist of 1×1 convolutions, batch normalization, and ReLU, while the fifth layer consists of 1×1 convolutions; the process of the scoring head, scale head, and bias head predicting the target bounding box is as follows:

[0065] ① The inputs to the scoring head, scale head, and bias head are the output features s. o Each input is processed to obtain a probability score for its center position. Deviation prediction Scale prediction H s W s The height and width of the search frame are represented by d, and the network step size is represented by d.

[0066] ② In the central probability P, the position corresponding to the maximum value is the target center position, i.e. (x d ,y d ) = arg max (x,y) P xy ;x d y d The x and y coordinates represent the center position of the target.

[0067] ③ The target bounding box is calculated using the following formula:

[0068] (x,y,w,h)=(x d +O(0,x d ,yd ),y d +O(1,x d ,y d ),S(0,x d ,y d ),S(1,x d ,y d ))

[0069] Where x represents the x-coordinate of the center position of the target bounding box, y represents the y-coordinate of the center position of the target bounding box, w represents the width of the target bounding box, and h represents the length of the target bounding box; in addition, the maximum value of the center probability P output by the score head is used as the confidence of the current search frame.

[0070] In step 4, the loss function calculation module includes a classification loss function L. cls Regression loss function and contrastive learning loss function L cl During the training phase, the classification loss function L is used. cls Regression loss function and contrastive learning loss function L cl End-to-end training is performed; the classification loss function uses the Gaussian weighted focal loss function; the regression loss function consists of the L1 loss function and the GIOU loss function. giou Composition; the total loss function is L, and the classification loss function is L... cls Regression loss function and contrastive learning loss function L cl The weighted sum; the calculation formula is as follows:

[0071] L=λ cls L cls +λ1L1+λ giou L giou +λ cl L cl

[0072] Where, λ cls =1, λ1=2, λ giou =5,λ cl =0.1 are both regularization parameters;

[0073] The specific loss function is calculated as follows:

[0074] For the target center within each real target bounding box and its low-resolution variant Generate a Gaussian truth heatmap using a Gaussian kernel. σ is the standard deviation of the target scale adaptation, and the classification loss function formula is L. cls as follows:

[0075]

[0076] Where α and β are hyperparameters controlling the distribution at each point, α = 2, β = 4; p xy This is the predicted probability that the location is the center of the target.

[0077] The formula for the L1 loss function is as follows:

[0078]

[0079] Among them, y b x represents the coordinates of the true target bounding box. b The coordinates of the predicted target bounding box;

[0080] GIOU loss function L giou The formula is:

[0081]

[0082]

[0083] Where A is the area of ​​the true target bounding box, B is the area of ​​the predicted target bounding box, and C is the smallest convex closed box containing A and B.

[0084] The contrastive learning loss function uses the InfoNCE-V loss function, which is an extension of the InfoNCE (info noise contrastive estimation) loss function. The InfoNCE loss function treats samples transformed from the same image as positive sample pairs, and samples from other images as negative sample pairs. The InfoNCE loss function is calculated as follows:

[0085]

[0086] Where q is the query sample, k d For positive samples, k i For negative samples; τ is a hyperparameter, i.e., the temperature coefficient; this loss function aims to increase q and k. d The similarity between them reduces the difference between q and k. i Similarity between them;

[0087] The InfoNCE-V loss function treats all samples from the same video sequence within the same training batch as positive samples. That is, template images, dynamic template images, and search frame images from the same video sequence are considered positive samples, while template images, dynamic template images, and search frame images from other video sequences within the same batch are considered negative samples. The improved loss function calculation formula is as follows:

[0088]

[0089] Where, q i j and This represents the query sample and the positive and negative samples, where i represents the video sequence index and j represents the index of each sample in the same video sequence.

[0090] In step 5:

[0091] ① After completing the target bounding box prediction for the current search frame, obtain the maximum confidence score from the split outputs as the confidence score S for the current search frame. now If S now The confidence S is greater than the optimal historical search frame. before Then, the dynamic template candidate is set as the current search frame, and the confidence S of the best historical search frame is set. before The current search frame quality S now ;

[0092] ② Every M frames, the dynamic template candidates are pruned to obtain new dynamic templates. During pruning, the corresponding target bounding box is used as the center. Then, the dynamic template with the longest interval from the current frame in the template set is replaced with the new dynamic template, and the confidence S of the best historical search frame is set. before Set to 0.

[0093] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0094] 1. This invention improves the model's ability to extract category semantic information by employing a contrastive learning strategy, namely, designing a contrastive learning head and a contrastive learning loss function in the loss function calculation module. This gives the invention the advantages of wide applicability and high tracking accuracy.

[0095] 2. This invention uses a cross-layer semantic association module to associate multi-layer template features with search frame features, further supplementing target semantic information. This provides richer and more diverse target semantic information guidance for search frame features, improving the model's ability to perceive and locate targets, and giving this invention the advantage of high tracking accuracy.

[0096] 3. This invention utilizes a redundancy pruning module to adaptively remove redundant background regions based on the complexity of the image, reducing the impact of redundant background information on the target. This improves accuracy while reducing computational load, giving the invention the advantage of high tracking precision.

[0097] In summary, this invention can improve the model's ability to extract category semantic information by setting a contrastive learning strategy, provide richer and more diverse target semantic information guidance for search frame features by setting a cross-layer semantic association module, and adaptively delete redundant background regions by setting a redundant information pruning module. This makes the invention have the advantages of high tracking accuracy and wide applicability. Attached Figure Description

[0098] Figure 1 This is the overall structural diagram of the target tracking model of the present invention.

[0099] Figure 2 This is a structural diagram of the feature extraction module in the target tracking model of this invention.

[0100] Figure 3 This is a structural diagram of the semantic self-association module in the target tracking model of this invention.

[0101] Figure 4 This is a structural diagram of the cross-layer semantic association module in the target tracking model of this invention.

[0102] Figure 5 This is a structural diagram of the contrast learning head in this invention. Detailed Implementation

[0103] The invention will now be further described with reference to the accompanying drawings.

[0104] This invention enhances the model's extraction and utilization of semantic information through three key designs: a contrastive learning strategy, a cross-layer semantic association module, and a redundancy pruning module. First, it introduces a contrastive learning strategy into the Transformer-based target tracking method. By designing a contrastive learning head and a contrastive learning loss function, it utilizes the semantic information of positive and negative sample labels to increase the similarity of features between similar targets and reduce the similarity of features between different targets, thereby improving the ability to extract category-specific semantic information. Second, the cross-layer semantic association module associates multi-level semantic features, thereby extracting richer target-related semantic information. Finally, the redundancy pruning module adaptively and selectively removes redundant background information. Excessive redundant background information reduces the saliency of target features, causing the model to confuse the target and its surrounding environment. This invention aims to extract richer semantic information, improve the model's understanding and adaptability to various semantic scenarios, and thus improve model performance while maintaining inference speed.

[0105] This invention targets single-target tracking tasks, that is, for any target in a video sequence, given its initial frame position, the algorithm processes the data to give the position of the target in all subsequent frames.

[0106] like Figure 1 As shown, a video target tracking system based on contrastive learning includes a feature extraction module, a contrastive learning head, a tracking head, a dynamic template update module, and a loss function calculation module, wherein:

[0107] The feature extraction module, built upon the Vision Transformer backbone network, includes a block embedding layer, a semantic self-association module, a cross-layer semantic association module, and a redundancy pruning module. It is used to extract features from the input image, providing meaningful feature representations for subsequent modules.

[0108] The block embedding layer is used to transform the input template image, dynamic template image, and search frame image into template features, dynamic template features, and search frame features, respectively.

[0109] The semantic self-association module is used to capture the global contextual relationships between template splicing features and search frame features themselves, and extract features with stronger representation capabilities.

[0110] The cross-layer semantic association module is used to associate search frame features with multi-layer template concatenation features to extract richer template target-related semantic information;

[0111] The redundancy information pruning module is used to prune redundant backgrounds in the search frame features, thereby reducing the number of background features.

[0112] The contrast learning head is used to perform feature mapping between the contrast features of the search frame output by the feature extraction module and the contrast features of the template. The contrast learning loss function is calculated using the feature mapping results of the contrast learning head to enhance the feature extraction module's ability to extract category semantic information and its feature representation capabilities.

[0113] The tracking head includes a scoring head, a scale head, and a bias head. The output of the scoring head is used to predict the center position of the target, and the maximum value of the center position is used to predict the confidence of the current search frame. The output of the bias head is used to compensate for the discretization error caused by the resolution. The output of the scale head is used to predict the width and height of the target. The center target position output by the scoring head, the discretization error predicted by the bias head, and the target width and height output by the scale head are used together to predict the target bounding box.

[0114] The dynamic template update module controls the update of the dynamic template based on the search frame confidence output from the score head in the prediction module, which is used to adapt to changes in the target's appearance during tracking.

[0115] The loss function calculation module includes classification loss function, regression loss function, and contrastive learning loss function. The classification loss function and regression loss function are calculated based on the predicted target bounding boxes output by the score head, scale head, and bias head. The contrastive learning loss function is calculated based on the feature mapping results of the contrastive learning head. During training, the sum of the classification loss function, regression loss function, and contrastive learning loss function is used for backpropagation to optimize the network parameters.

[0116] like Figure 2As shown, a video target tracking method based on contrastive learning includes the following steps:

[0117] Step 1: Input the search frame image, template image, and dynamic template image into the feature extraction module for processing. The feature extraction module is built based on the Vision Transformer backbone network and includes a block embedding layer, a semantic self-association module, a cross-layer semantic association module, and a redundancy pruning module to obtain the output search frame comparison features s. c Template comparison features t c Output search frame features s o The output search frame comparison feature s c Template comparison features t c The input is fed into the contrastive learning head for feature mapping.

[0118] The specific method for step 1 is as follows:

[0119] Step 1.1: The input consists of a search frame image, a template image, and a dynamic template image. The template image and the dynamic template image are both 3×H in size. t ×W t The size of the search frame image is 3×H s ×W s H t and W t H represents the height and width of the template, respectively. s and W s These represent the height and width of the search frame, respectively.

[0120] Step 1.2: The search frame image, template image, and dynamic template image input in Step 1.1 are fed into the block embedding layer to obtain search frame features s, template features t, and dynamic template features d, with sizes N respectively. s ×C、N t ×C、N d ×C, where C represents the number of channels for the feature, and N s =N d N s N t N d These represent the number of features in the search frame features, template features, and dynamic template features, respectively.

[0121] Step 1.3: Add the search frame features, template features, and dynamic template features obtained in Step 1.2 to the positional encoding respectively to obtain s1 and t. t t d s1, t t t d These represent search frame features, template features, and dynamic template features, respectively, with dimensions N. s ×C、Nt ×C、N t ×C;

[0122] Step 1.4, for the template feature t obtained in step 1.3 t Dynamic template features t d The templates are spliced ​​together to obtain the splicing feature t1, with a size of 2N. t ×C;

[0123] Step 1.5: Input the template concatenation feature t1 obtained in Step 1.4 into the n-layer semantic self-association module to obtain the template concatenation feature t2, which is the template comparison feature t. c The size is 2N t ×C, Template contrast feature t c Used for comparison learning heads to perform feature mapping;

[0124] Step 1.6: Input the template concatenation feature t2 obtained in Step 1.5 into the semantic self-association module to obtain the template concatenation feature t3 with a size of 2N. t ×C;

[0125] Step 1.7: Input the template concatenation feature t3 obtained in Step 1.6 into the semantic self-association module to obtain the template concatenation feature t4 with a size of 2N. t ×C;

[0126] Step 1.8: Input the template concatenation feature t4 obtained in Step 1.7 into the semantic self-association module to obtain the template concatenation feature t5 with a size of 2N. t ×C;

[0127] Step 1.9: Input the search frame feature s1 obtained in Step 1.3 into the n-layer semantic self-association module to obtain the search frame feature s2, which is the search frame comparison feature s. c Size N s ×C, Search frame comparison features s c Used for comparison learning heads to perform feature mapping;

[0128] Step 1.10: The search frame feature s2 obtained in Step 1.9, the template concatenation feature t2 obtained in Step 1.5, and the template concatenation feature t5 obtained in Step 1.8 are fed into the cross-layer semantic association module to obtain the search frame feature s3, with a size of N. s ×C;

[0129] Step 1.11: The search frame feature s3 obtained in Step 1.10, the template concatenation feature t3 obtained in Step 1.6, and the template concatenation feature t5 obtained in Step 1.8 are fed into the cross-layer semantic association module to obtain the search frame feature s4, with a size of N. s ×C;

[0130] Step 1.12: The search frame feature s4 obtained in Step 1.11, the template concatenation feature t4 obtained in Step 1.7, and the template concatenation feature t5 obtained in Step 1.8 are fed into the cross-layer semantic association module to obtain the search frame feature s5, with a size of N. s ×C;

[0131] Step 1.13: Input the search frame feature s5 obtained in step 1.12 into the redundancy information pruning module to obtain the output search frame feature s o Size N s1 ×C.

[0132] Next, we will further explain the important modules involved in the feature extraction module.

[0133] The block embedding layer includes a 3×3 convolution operation, a layer normalization operation, and a dimension transformation operation. The dimension transformation operation transforms the feature size from H×W×C to N×C, where N=H×W.

[0134] like Figure 3 As shown, the computation of the semantic self-association module consists of multi-head attention and a feedforward neural network (FFN), as illustrated in the following formula:

[0135] O = X in +MultiHead(X in ,X in ,X in )

[0136] Y out =O + FFN(O)

[0137] Among them, X in Y represents the input feature. out Indicates output features;

[0138] The calculation process for multi-head attention is shown in the following formula:

[0139]

[0140] H i =Attention(QW i Q ,KW i K VW i V )

[0141] Attention(Q,K,V)=softmax(A(Q,K))V

[0142]

[0143] Where, n h W represents the number of heads. i Q W i K W i V W i O Let Q, K, V, and O represent the parameters of the linear mappings, respectively, and let A(Q,K) represent the attention weight matrix. k This represents the dimension of the KEY vector obtained from the linear mapping K;

[0144] The feedforward neural network consists of two linear layers with a ReLU function inserted in the middle. The calculation formula is as follows:

[0145] FFN(O)=ReLU(OW1+b1)W2+b2

[0146] Where W1 and W2 are the weight matrices of the linear layer, and b1 and b2 are the basis vectors;

[0147] The formula for calculating the linear rectified function ReLU is as follows:

[0148]

[0149] Where 'a' represents an element in the input features.

[0150] Existing methods only associate the current layer template features with the current layer search frame features, or only associate deep layer template features with the search frame features, to guide the extraction of target information from the search frame features. However, the shape of the target in the search frame may differ significantly from that in the template, and a single-level template feature cannot meet the diverse target information requirements of the search frame features. The cross-layer semantic association module of this invention, by associating multi-level template features with search frame features, further supplements the target semantic information, thereby providing richer and more diverse target semantic information guidance for the search frame features and improving the model's ability to perceive and locate targets.

[0151] like Figure 4 As shown, the cross-layer semantic association module has the same structure as the semantic self-association module, but the input is the template feature t. i The i-th layer is the current layer's search frame features s i The last layer template feature t5, 2<=i<=5; its calculation process is shown in the formula:

[0152] O = s i +MultiHead(s i ,[s i ;t i ;t5],[si ;t i ;t5])

[0153] Y out =O + FFN(O)

[0154] Among them, [s i ;t i ;t5] indicates that t i s i t5 is spliced ​​together.

[0155] Excessive redundant background information reduces the saliency of target features, causing the model to confuse the target and its surrounding environment. To address this, this invention proposes a redundancy pruning module that adaptively and effectively reduces the impact of complex backgrounds on the target based on the image's complexity, thereby significantly improving the target's saliency.

[0156] The redundant information pruning module takes the attention weight matrix A(Q,K) corresponding to the search frame feature S5, performs average pooling and maximum-minimum normalization operations, and uses it as the confidence score, i.e., energy, of each feature in the search frame feature S5, with a value in the range [0-1]. The feature confidence scores are sorted from largest to smallest to obtain the confidence score. Then, the confidence scores are summed to obtain the total energy E of the candidate region, and ρ%×E is taken as the energy threshold, where ρ is the proportion of energy retained. Finally, the top K features with the highest energy in the candidate region are retained to satisfy... And discard the remaining features;

[0157] If the image background is simple, the sum of the energies of a few candidate regions can reach the energy threshold. Therefore, more candidate regions are discarded, and less information is retained so that the model can focus on the foreground region. Conversely, for complex background images, more candidate regions are needed to reach the energy threshold. Therefore, fewer candidate regions are discarded, and more information is retained to ensure the model's high discriminative power.

[0158] like Figure 5 As shown, in step 2, the search frame obtained in step 1 is compared with the feature s. c Feature t compared with template c All inputs are fed into the contrast learning head for feature mapping, and the feature mapping results of the contrast learning head are output.

[0159] The specific method for step 2 is as follows:

[0160] The contrastive learning head is used for feature mapping and consists of region-of-interest pooling, 3×3 convolution, ReLU, 1×1 convolution, and batch normalization; it searches for contrastive features s in the frame. c Feature t compared with template cThe model is processed by region-of-interest pooling, 3×3 convolution, ReLU, 1×1 convolution, and batch normalization, and then concatenated together for the calculation of the loss function for contrastive learning. The contrastive learning head is only used during model training and is discarded during inference, so it has no impact on the inference speed of the model.

[0161] Step 3, use the search frame features s obtained in Step 1. o The input tracking head includes a scoring head, a scale head, and a bias head. The output of the scoring head is used to predict the target center position, and the maximum value of the target center position is used to predict the confidence of the current search frame. The output of the bias head is used to compensate for the discretization error caused by the resolution. The output of the scale head is used to predict the width and height of the target. The center target position output by the scoring head, the discretization error predicted by the bias head, and the target width and height output by the scale head are used together to predict the target bounding box.

[0162] In step 3, the scoring head, scale head, and bias head are all implemented by 5 layers of 1×1 convolutional blocks; the first four layers are composed of 1×1 convolution, batch normalization, and ReLU linear rectified function, and the fifth layer is composed of 1×1 convolution; the output of the scoring head is used to predict the target center position, the maximum value of the target center position is used to predict the confidence of the current search frame, the output of the bias head is used to compensate for the discretization error caused by the resolution, and the output of the scale head is used to predict the width and height of the target. The center target position output by the scoring head, the discretization error predicted by the bias head, and the target width and height output by the scale head are used together to predict the target bounding box;

[0163] The process of predicting the target bounding box using the score head, scale head, and bias head is as follows:

[0164] ① The input to the deviation head, scale head, and score head is the output feature s. o Each input is processed to obtain a probability score for its center position. Deviation prediction Scale prediction H s W s The height and width of the search frame are represented by d, and the network step size is represented by d.

[0165] ② In the central probability P, the position corresponding to the maximum value is the target center position, i.e. (x d ,y d ) = argmax (x,y) P xy ;x d y d The x and y coordinates represent the center position of the target.

[0166] ③ The target bounding box is calculated using the following formula:

[0167] (x,y,w,h)=(x d +O(0,x d ,y d ),y d +O(1,x d ,y d ),S(0,x d ,y d ),S(1,x d ,y d ))

[0168] Where x represents the horizontal coordinate of the center position of the target bounding box, y represents the vertical coordinate of the center position of the target bounding box, w represents the width of the target bounding box, and h represents the length of the target bounding box.

[0169] Furthermore, the maximum value of the center probability P output by the score head is used as the confidence level of the current search frame.

[0170] Step 4: During training, the contrastive learning head feature mapping result output in Step 2 is used to calculate the loss function through the contrastive learning loss function in the loss function calculation module. The target bounding box output in Step 3 is used to calculate the classification loss function and the regression loss function to obtain the loss values ​​of the contrastive learning loss function, the classification loss function and the regression loss function. These values ​​are then summed to obtain the total loss value. The total loss value is used for backpropagation to optimize the network parameters.

[0171] In step 4, the loss function calculation module includes a classification loss function L. cls Regression loss function and contrastive learning loss function L cl During the training phase, the classification loss function L is used. cls Regression loss function and contrastive learning loss function L cl End-to-end training is performed; the classification loss function uses the Gaussian weighted focal loss function; the regression loss function consists of the L1 loss function and the GIOU loss function. giou Composition; the total loss function is L, and the classification loss function is L... cls Regression loss function and contrastive learning loss function L cl The weighted sum; the calculation formula is as follows:

[0172] L=λ cls L cls +λ1L1+λ giou L giou +λ cl L cl

[0173] Where, λcls =1, λ1=2, λ giou =5,λ cl =0.1 are both regularization parameters;

[0174] The specific loss function is calculated as follows:

[0175] For the target center within each real target bounding box and its low-resolution variant Generate a Gaussian truth heatmap using a Gaussian kernel. σ is the standard deviation of the target scale adaptation, and the classification loss function formula is L. cls as follows:

[0176]

[0177] Where α and β are hyperparameters controlling the distribution at each point, α = 2, β = 4; p xy This represents the predicted probability that the location is the center of the target.

[0178] The formula for the L1 loss function is as follows:

[0179]

[0180] Among them, y b x represents the coordinates of the true target bounding box. b The coordinates of the predicted target bounding box;

[0181] GIOU loss function L giou The formula is:

[0182]

[0183]

[0184] Where A is the area of ​​the true target bounding box, B is the area of ​​the predicted target bounding box, and C is the smallest convex closed box containing A and B.

[0185] The contrastive learning loss function used in this invention is the InfoNCE-V loss function, which is an extension of the InfoNCE (info Noise Contrastive Estimation) loss function. The InfoNCE loss function treats samples transformed from the same image as positive sample pairs, and samples from other images as negative sample pairs. The calculation formula for the InfoNCE loss function is as follows:

[0186]

[0187] Where q is the query sample, k d For positive samples, k iFor negative samples; τ is a hyperparameter, i.e., the temperature coefficient; this loss function aims to increase q and k. d The similarity between them reduces the difference between q and k. i The similarity between them.

[0188] However, this doesn't consider the temporal consistency issue within video sequences, as samples from the same video sequence are similar. The InfoNCE-V loss function extends this by treating samples from the same video sequence within the same training batch as positive samples. Specifically, template images, dynamic template images, and search frame images from the same video sequence are considered positive samples, while template images, dynamic template images, and search frame images from other video sequences are considered negative samples. The improved loss function is calculated as follows:

[0189]

[0190] Where, q i j and This represents the query sample and the positive and negative samples, where i represents the video sequence index and j represents the index of each sample in the same video sequence.

[0191] Step 5: Use the current search frame confidence score output from the score head in Step 3 to control whether to perform dynamic template update; every M frames, select the search frame with the highest confidence score among the M frames and crop it as a new dynamic template.

[0192] The dynamic template update module has two key aspects: the timing of template updates and the selection of dynamic templates. If template updates are too frequent, the feature information contained in the current dynamic template will be essentially the same as the one to be updated. In this case, updating the dynamic template will not only offer little benefit but also introduce background noise into the model. Conversely, if the time interval between template updates is too long, the model may have drifted, leading to tracking failure, rendering the model update meaningless. Regarding the selection of dynamic templates, if the quality of the dynamic template to be updated is low (i.e., the predicted target position deviates significantly from the actual target position), the dynamic template will contain a large amount of interference information and background noise. After updating the dynamic template, not only will the ability to distinguish targets not improve, but the dynamic template will also become contaminated. If dynamic template updates are performed as soon as a high-quality template is found, the update interval will be too short or too long, leading to the introduction of additional background noise. Conversely, if dynamic template updates are performed at the right time, the quality of the template to be updated cannot be controlled, also introducing noise. Therefore, this invention chooses a mode combining delayed updates and search frame confidence prediction. Delayed updates refer to updating the template every fixed number of frames, selecting the optimal template within this time interval for updating. The search frame confidence is the maximum confidence output by the score head, and the search frame with the highest confidence is taken as the optimal template.

[0193] In step 5:

[0194] ① After completing the target bounding box prediction for the current search frame, obtain the maximum confidence score from the split outputs as the confidence score S for the current search frame. now If S now The confidence S is greater than the optimal historical search frame. before Then, the dynamic template candidate is set as the current search frame, and the confidence S of the best historical search frame is set. before The current search frame quality S now ;

[0195] ② Every M frames, the dynamic template candidates are pruned to obtain new dynamic templates. During pruning, the corresponding target bounding box is used as the center. Then, the dynamic template with the longest interval from the current frame in the template set is replaced with the new dynamic template, and the confidence S of the best historical search frame is set. before Set to 0.

[0196] To address the shortcomings of existing technologies, this invention proposes a contrastive learning strategy, namely a contrastive learning head and a contrastive learning loss function. This strategy extracts features by comparing the similarity of features between different samples. Its main idea relies on the assumption that different views of the same sample obtained through some data augmentation operation have high similarity, while different samples are dissimilar. The goal of contrastive learning is to find this similarity, maximizing the similarity between different views of the same sample, so that the feature representations of different views of the same sample are as close as possible in the embedding space, and different samples are as far apart as possible. These different views of the same sample are called positive sample pairs in contrastive learning, and vice versa. Target tracking requires locating the target position in the search frame based on the target in the template, and masking the influence of similar objects. That is, it needs to maximize the similarity between targets in the same video sequence and minimize the similarity between the target and other objects. Since the two have certain similarities in their tasks, contrastive learning can be used to assist in the training of target tracking tasks. However, contrastive learning ignores the semantic structural features of the data; if two samples come from different instances, they are considered negative samples, regardless of their semantic similarity. This amplifies the problem when a large number of negative samples are present, because this method causes samples that are originally semantically similar to be far apart in the mapping space.

[0197] Therefore, the implementation of the contrastive learning strategy in this invention is as follows: template images, dynamic template images, and search frame images from the same video are used as positive samples, and other images in the same batch, i.e., images not belonging to the same video, are used as negative samples; by recombining the inputs of the feature extraction module, i.e., template images, dynamic template images, and search frame images, training positive and negative samples are obtained. For the extraction of contrastive learning features, this invention reuses the intermediate features of the feature extraction module, i.e., the search frame contrastive features s. c Feature t compared with template c Then, the contrastive learning features are extracted, and the template contrastive features t are used. c Compare features s with search frames c As contrastive learning features, these are fed into the contrastive learning head for feature mapping. Finally, the loss is calculated using the contrastive learning loss function to measure the similarity between positive and negative samples, and backpropagation is used to optimize the network parameters. Finally, the model is trained by minimizing the contrastive loss, enabling it to learn effective feature representations.

[0198] The overall framework of the model of this invention includes a feature extraction module, a contrastive learning head, a tracking head, a dynamic template update module, and a loss function calculation module; the overall structure diagram of this invention is as follows. Figure 1As shown, the model's inputs are first the search frame image, the template image, and the dynamic template image, which are then processed by the feature extraction module. Next, its outputs are fed into the contrastive learning head for processing. The output of the contrastive learning head is used to calculate the contrastive learning loss function. The tracking head includes a scoring head, a scale head, and a bias head. The output of the tracking head is used to predict the target bounding box, which is the final output of the model. The output of the scoring head is used to predict the confidence level of the current search frame, controlling whether to perform dynamic template updates.

[0199] This invention first designs a contrastive learning strategy, implemented using a contrastive learning head and a contrastive learning loss function. It utilizes semantic labels to construct positive and negative sample pairs during training and supervises the model using the label information of these samples. This enables the model to effectively distinguish between positive and negative samples during training, increasing feature similarity between similar targets while reducing feature similarity between different targets. This enhances the model's extraction and utilization of category semantic information, improving its accuracy. Secondly, this invention designs a cross-layer semantic association module to associate multi-layer semantic features, thereby obtaining richer and more diverse target-related semantic information. This more accurately guides the extraction of target-related features, achieving more accurate target location determination and improving tracking accuracy. Furthermore, this invention proposes a redundancy pruning module, which adaptively prunes based on content complexity. It discards redundant background information to prevent interference with tracking while retaining sufficient discriminative information, further improving tracking accuracy.

[0200] Furthermore, the present invention also has the following features:

[0201] 1. The feature extraction network used in this invention is based on the existing Vision Transformer. Replacing this network with other network architectures can achieve the same purpose.

[0202] 2. For the contrast learning head part of this invention, its input can be replaced with the input of other layers of the feature extraction network, which can also achieve the purpose of this invention.

[0203] 3. For the cross-layer semantic association part of the present invention, replacing the input with the template or the output of other layers of the search frame can also achieve the purpose of the present invention.

[0204] 4. The attention calculation used in the cross-layer semantic association module of this invention is an existing self-attention mechanism, which can achieve global and effective modeling of template features and search frame features. If an improved attention mechanism is used, such as block attention, step attention, hash attention, and linear attention, the objectives of this invention can also be achieved.

Claims

1. A video target tracking system based on contrastive learning, characterized in that: It includes a feature extraction module, a contrastive learning head, a tracking head, a dynamic template update module, and a loss function calculation module; among which: The feature extraction module includes a block embedding layer, a semantic self-association module, a cross-layer semantic association module, and a redundancy pruning module, used to extract features from the input image, wherein: The block embedding layer is used to transform the input template image, dynamic template image, and search frame image into template features, dynamic template features, and search frame features, respectively. The semantic self-association module is used to capture the global contextual relationships between template splicing features and search frame features, and extract features with strong representational capabilities. The semantic self-association module is computed by a multi-head attention (MultiHead) and a feedforward neural network (FFN). The cross-layer semantic association module is used to associate search frame features with multi-layer template concatenation features to extract richer template target-related semantic information; The cross-layer semantic association module takes template features as its structural input. t i , No. i Current layer search frame features s i Last layer template features t 5, 2 <= i <=5; the calculation process is shown in the formula: in, Indicates will t i , s i , t 5 pieces together; The redundancy information pruning module is used to prune redundant backgrounds in the search frame features, reduce the number of background features, and reduce the impact of complex backgrounds on the target. The redundancy pruning module extracts and calculates the features of the search frame. S Attention weight matrix corresponding to time 5 Average pooling and maximum-minimum normalization operations are performed to obtain the search frame features. S The confidence level, or energy, of each feature in 5 is in the range [0-1]. The confidence scores are obtained by sorting the feature confidence scores from largest to smallest. Then, the confidence level Summing gives the total energy of the candidate region. ,Pick As an energy threshold To preserve the proportion of energy; finally, retain the candidate region with the highest energy. A feature that satisfies and discard the remaining features; The contrast learning head is used to perform feature mapping between the contrast features of the search frame output by the feature extraction module and the contrast features of the template. The contrast learning loss function is calculated using the feature mapping results of the contrast learning head to enhance the feature extraction module's ability to extract category semantic information and its feature representation capabilities. The tracking head includes a scoring head, a scale head, and a bias head. The output of the scoring head is used to predict the center position of the target, and the maximum value of the center position is used to predict the confidence of the current search frame. The output of the bias head is used to compensate for the discretization error caused by the resolution. The output of the scale head is used to predict the width and height of the target. The center target position output by the scoring head, the discretization error predicted by the bias head, and the target width and height output by the scale head are used together to predict the target bounding box. The dynamic template update module controls the update of the dynamic template based on the confidence level of the current search frame output by the score head, so as to adapt to changes in the appearance of the target during the tracking process. The loss function calculation module includes classification loss function, regression loss function, and contrastive learning loss function. The classification loss function and regression loss function are calculated based on the predicted target bounding boxes output by the score head, scale head, and bias head. The contrastive learning loss function is calculated based on the feature mapping results of the contrastive learning head. During training, the sum of the classification loss function, regression loss function, and contrastive learning loss function is used for backpropagation to optimize the network parameters.

2. A video target tracking method based on contrastive learning, characterized in that: This method, applied to a video target tracking system based on contrastive learning as described in claim 1, includes the following steps: Step 1: Input the search frame image, template image, and dynamic template image into the feature extraction module for processing. The feature extraction module includes a block embedding layer, a semantic self-association module, a cross-layer semantic association module, and a redundancy pruning module to obtain the output search frame comparison features. s c Template comparison features t c Output search frame features s o ; Step 2: Compare the features of the search frames obtained in Step 1. s c Features compared with template t c All inputs are fed into the contrast learning head for feature mapping, and the feature mapping results of the contrast learning head are output. Step 3, use the search frame features obtained in Step 1. s o The input tracking head includes a scoring head, a scale head, and a bias head. The output of the scoring head is used to predict the target center position, and the maximum value of the target center position is used to predict the confidence of the current search frame. The output of the bias head is used to compensate for the discretization error caused by the resolution. The output of the scale head is used to predict the width and height of the target. The center target position output by the scoring head, the discretization error predicted by the bias head, and the target width and height output by the scale head are used together to predict the target bounding box. Step 4: During training, the contrastive learning head feature mapping result output in Step 2 is used to calculate the loss function through the contrastive learning loss function in the loss function calculation module. The target bounding box output in Step 3 is used to calculate the classification loss function and the regression loss function to obtain the loss values ​​of the contrastive learning loss function, the classification loss function and the regression loss function. These values ​​are then summed to obtain the total loss value. The total loss value is used for backpropagation to optimize the network parameters. Step 5: Use the current search frame confidence score output from the score head in Step 3 to control whether to perform dynamic template update; every M frames, select the search frame with the highest confidence score among the M frames and crop it as a new dynamic template.

3. The video target tracking method based on contrastive learning according to claim 2, characterized in that: The specific method for step 1 is as follows: Step 1.1: The input consists of a search frame image, a template image, and a dynamic template image. The template image and the dynamic template image are both of the same size. The size of the search frame image is , H t and W t These represent the height and width of the template, respectively. H s and W s These represent the height and width of the search frame, respectively. Step 1.2: The search frame image, template image, and dynamic template image input in Step 1.1 are fed into the block embedding layer to obtain the search frame features. s Template features t Dynamic template features d The dimensions are respectively , , , C The number of channels representing the feature. , N s , N t、 N d These represent the number of features in the search frame features, template features, and dynamic template features, respectively. Step 1.3: Add the search frame features, template features, and dynamic template features obtained in Step 1.2 to the positional encoding respectively, to obtain... s 1. t t , t d , s 1. t t , t d These represent search frame features, template features, and dynamic template features, respectively, with dimensions of [sizes to be filled in]. , , ; Step 1.4, for the template features obtained in Step 1.3 t t Dynamic template features t d By splicing the components together, we obtain the template splicing features. t 1. Dimensions are ; Step 1.5: Combine the template splicing features obtained in Step 1.

4. t 1. The template is fed into an n-layer semantic self-association module to obtain template concatenation features. t 2, namely template comparison features t c The size is ; Step 1.6: Combine the template splicing features obtained in Step 1.

5. t 2. The data is fed into the semantic self-association module to obtain template concatenation features. t 3. Dimensions are ; Step 1.7: Combine the template splicing features obtained in Step 1.

6. t 3. The data is fed into the semantic self-association module to obtain template concatenation features. t 4. Dimensions are ; Step 1.8: Combine the template splicing features obtained in Step 1.

7. t 4. The data is fed into the semantic self-association module to obtain template concatenation features. t 5. Dimensions are ; Step 1.9, use the search frame features obtained in Step 1.

3. s 1 input n The layer semantic self-association module obtains the search frame features. s 2, namely, search frame comparison features s c The size is ; Step 1.10, use the search frame features obtained in Step 1.9 s 2. Template splicing features obtained in step 1.5 t 2 and the template splicing features obtained in step 1.8 t 5. Together, they are fed into the cross-layer semantic association module to obtain the search frame features. s 3. Dimensions are ; Step 1.11, use the search frame features obtained in Step 1.

10. s 3. Template splicing features obtained in step 1.6 t 3 and the template splicing features obtained in step 1.8 t 5. Together, they are fed into the cross-layer semantic association module to obtain the search frame features. s 4. Dimensions are ; Step 1.12, use the search frame features obtained in Step 1.

11. s 4. Template splicing features obtained in step 1.7 t 4 and the template splicing features obtained in step 1.8 t 5. Together, they are fed into the cross-layer semantic association module to obtain the search frame features. s 5. Dimensions are ; Step 1.13, use the search frame features obtained in Step 1.

12. s 5. The redundant information is fed into the redundancy pruning module to obtain the output search frame features. s o The size is .

4. A video target tracking method based on contrastive learning according to claim 2 or 3, characterized in that: The block embedding layer includes Convolution operations, layer normalization operations, and dimensionality transformation operations. Dimensionality transformation operations change the feature size from... for ,in ; The semantic self-association module is computed by a multi-head attention (MultiHead) and a feedforward neural network (FFN), as shown in the following formula: in, Indicates input features, Indicates output features; The calculation process for multi-head attention is shown in the following formula: in, For the number of heads, , , , They represent linear mappings respectively. , , , The parameters, Represents the attention weight matrix. Represents a linear mapping Received The dimension of a vector; The feedforward neural network is configured by inserting a linear rectified function in the middle. It consists of two linear layers, and the calculation formula is as follows: in, , It is the weight matrix of the linear layer. , They are basis vectors; The formula for calculating the linear rectified function ReLU is as follows: in, For elements in the input features.

5. A video target tracking method based on contrastive learning according to claim 2, characterized in that: In step 2, the contrastive learning head consists of region-of-interest pooling, 3×3 convolution, ReLU, 1×1 convolution, and batch normalization; it searches for frame contrastive features. s c Features compared with template t c The data are processed separately using region-of-interest pooling, 3×3 convolution, ReLU, 1×1 convolution, and batch normalization, and then concatenated together for comparison learning loss function calculation.

6. The video target tracking method based on contrastive learning according to claim 2, characterized in that: In step 3, the scoring head, scale head, and deviation head each consist of 5 layers. Convolutional blocks are implemented; the first four layers consist of 1×1 convolutions, batch normalization, and ReLU linear rectified functions, while the fifth layer consists of 1×1 convolutions; the process of predicting the target bounding box using the score head, scale head, and bias head is as follows: ① The inputs to the scoring head, scale head, and deviation head are the output features. s o Each input is processed to obtain a probability score for its center position. Deviation prediction Scale prediction , , Indicates the height and width of the search frame. Indicates the network step size; ② In the central probability In the diagram, the position corresponding to the maximum value is the center position of the target, that is... ; , The x and y coordinates represent the center position of the target. , ; ③ The target bounding box is calculated using the following formula: in, x The x-coordinate representing the center position of the target bounding box. y The ordinate represents the center position of the target bounding box. w This indicates the width of the target bounding box. h This represents the length of the target bounding box; additionally, the center probability output by the scoring head. In this context, the maximum value is used as the confidence level of the current search frame.

7. The video target tracking method based on contrastive learning according to claim 2, characterized in that: In step 4, the loss function calculation module includes a classification loss function. Regression loss function and contrastive learning loss function During the training phase, a classification loss function is used. Regression loss function and contrastive learning loss function End-to-end training is performed; the classification loss function uses the Gaussian weighted focal loss function; the regression loss function is... loss function and GIOU loss function Composition; the total loss function is For classification loss function Regression loss function and contrastive learning loss function The weighted sum; the calculation formula is as follows: in, , , , All are regularization parameters; The specific loss function is calculated as follows: For the target center within each real target bounding box and its low-resolution variant Generate Gaussian truth heatmap using Gaussian kernel , It is the standard deviation of target scale adaptation, and the formula for the classification loss function. as follows: in, and The hyperparameters that control the distribution at each point, , ; This is the predicted probability that the location is the center of the target. The loss function formula is as follows: in, The coordinates of the actual target bounding box. The coordinates of the predicted target bounding box; GIOU loss function The formula is: in, This represents the area of ​​the actual target bounding box. To predict the area of ​​the target bounding box, For inclusion and The smallest convex closed frame; The contrastive learning loss function uses the InfoNCE-V loss function, which is an extension of the InfoNCE (info Noise Contrastive Estimation) loss function. The InfoNCE loss function treats samples transformed from the same image as positive sample pairs and samples from other images as negative sample pairs. The formula for calculating the InfoNCE loss function is as follows: in, q To query samples, k d As a positive sample, k i Negative samples; For hyperparameters, i.e., the temperature coefficient; this loss function aims to increase q and k d The similarity between them is reduced q and k i Similarity between them; The InfoNCE-V loss function treats all samples from the same video sequence within the same training batch as positive samples. That is, template images, dynamic template images, and search frame images from the same video sequence are considered positive samples, while template images, dynamic template images, and search frame images from other video sequences within the same batch are considered negative samples. The improved loss function calculation formula is as follows: in, and This represents the query sample and the positive and negative samples, where i represents the video sequence index and j represents the index of each sample in the same video sequence.

8. A video target tracking method based on contrastive learning according to claim 2, characterized in that: In step 5: ① After completing the target bounding box prediction for the current search frame, obtain the maximum confidence score from the split outputs as the confidence score for the current search frame. ;like Greater than the confidence level of the best historical search frame Then, the dynamic template candidate is set as the current search frame, and the confidence level of the best historical search frame is set. Quality of the current search frame ; ② Every In each frame, dynamic template candidates are truncated to obtain new dynamic templates. During truncating, the corresponding target bounding box is used as the center. Then, the dynamic template with the longest interval from the current frame in the template set is replaced with the new dynamic template, and the confidence of the best historical search frame is used. Set to 0.

Images