Target tracking method and system based on Siamese network with recursive dilated attention

Abstract

The present invention proposes a target tracking method and system based on a twin network of recursive dilated attention, the method comprising: constructing a special target tracking model based on a Transformer network and a recursive dilated attention module; based on a template and a search image, using a dual-branch feature extraction network to obtain a template and a search feature; splicing the template and the search feature to obtain a spliced ​​feature; based on the spliced ​​feature, combining a prediction head to perform calculations, obtain classification regression results, and predict the target according to the classification regression results; combining a large-scale data set to train the target tracking model in an iterative manner to obtain a trained target tracking model, and using the trained target tracking model to accurately track the target. The present invention optimizes the feature extraction process and provides a more efficient and accurate solution for image recognition and search tasks.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing technology, and in particular to a target tracking method and system based on a twin network of recursive expanded attention. Background Art

[0002] In the field of computer vision and image processing, visual tracking has many basic applications in computer vision, such as autonomous driving, video surveillance, traffic vehicle monitoring, and human-computer interaction. Therefore, with the improvement of practicality and real-time performance of tracking, more and more tracking-related applications are becoming more and more common in real life, which makes the research of video tracking technology more and more valuable.

[0003] In general, visual tracking remains a challenging task due to many reasons such as appearance changes, deformation, fast motion, and occlusion, and designing a robust tracker remains a challenging problem. In recent years, twin network trackers based on convolutional neural networks (CNNs) have been widely used in the field of visual tracking. Specifically, feature information based on the CNN model is introduced into the template and search branches, and one or more response maps obtained through the correlation operation are used to decode the extracted feature information. Among them, typical CNN models include AlexNet, VGG, and ResNet. At present, most CNN models mainly use multiple convolution operators (such as 3×3 or 5×5), which can effectively capture spatial local information, thereby modeling the global relationship of image content. However, this method weakens the model's ability to obtain important global context information when dealing with complex scenes.

[0004] In addition, the use of Transformer has led to rapid development in the field of computer vision. For many computer vision tasks such as tracking tasks such as target tracking, how to improve the model's expressiveness and performance is still a very important issue, and Transformer itself comes at the cost of low efficiency, so it is not desirable to improve performance at the cost of lower efficiency. How to effectively explore the context between consecutive frames without losing local information while adding only a few or no parameters, and how to improve the performance of the tracker and design a highly robust tracker have become important issues in target tracking. Summary of the invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a target tracking method and system based on a twin network with recursive expanded attention to solve the above technical problems.

[0006] The present invention provides a target tracking method based on a twin network of recursive dilated attention, the method comprising the following steps:

[0007] Step 1: Based on the Transformer network and the recursive dilation attention module, a dual-branch feature extraction network is constructed. The two branches of the dual-branch feature extraction network share weights, and a feature fusion network is constructed based on the self-attention mechanism. The target tracking model consists of the dual-branch feature extraction network, the feature fusion network and the prediction head;

[0008] Step 2: Input the template image into the target tracking model, extract features from the template image based on a dual-branch feature extraction network, and obtain template features; input the search image into the target tracking model, extract features from the search image based on a dual-branch feature extraction network, and obtain search features; splice the template features with the search features to obtain spliced ​​features;

[0009] Step 3: Input the concatenated features into the classification regression prediction head and perform calculations to obtain the classification regression results, and predict the target based on the classification regression results;

[0010] Step 4: Repeat steps 1 to 3 in an iterative manner in combination with a large-scale data set to train the target tracking model to obtain a trained target tracking model, and use the trained target tracking model to accurately track the target.

[0011] The present invention also proposes a target tracking system based on a twin network of recursive expanded attention, the system comprising:

[0012] Model building module, a dual-branch feature extraction network is built based on the Transformer network and the recursive dilation attention module. The two branches of the dual-branch feature extraction network share weights, and a feature fusion network is built based on the self-attention mechanism. The target tracking model consists of the dual-branch feature extraction network, the feature fusion network and the prediction head;

[0013] The feature extraction and fusion module inputs the template image into the target tracking model, extracts features from the template image based on the dual-branch feature extraction network, and obtains the template features; inputs the search image into the target tracking model, extracts features from the search image based on the dual-branch feature extraction network, and obtains the search features; splices the template features with the search features to obtain the spliced ​​features;

[0014] The prediction module inputs the concatenated features into the classification and regression prediction head and performs calculations to obtain the classification and regression results, and predicts the target based on the classification and regression results;

[0015] The target tracking module uses a large-scale data set to iteratively repeat steps 1 to 3 to train the target tracking model to obtain a trained target tracking model, and uses the trained target tracking model to accurately track the target.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. The present invention adopts a recursive expanded attention method, which further enhances the model's ability to learn complex features by recursively increasing the depth of the network.

[0018] 2. The present invention integrates the recursive expanded attention technology into each encoder of the feature extraction module, so that the model can perform global matching more accurately when processing target images and search images, thereby improving the accuracy and reliability of matching.

[0019] 3. The present invention not only optimizes the feature extraction process, but also provides a more efficient and accurate solution for image recognition and search tasks.

[0020] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of the target tracking method based on the recursive dilated attention twin network proposed in the present invention;

[0022] Figure 2 This is a framework diagram of the target tracking method based on the recursive expanded attention twin network proposed in the present invention;

[0023] Figure 3 A schematic diagram of the recursive dilated attention of the target tracking method based on the recursive dilated attention twin network proposed in the present invention;

[0024] Figure 4 A schematic diagram of the expanded attention of the target tracking method based on the recursive expanded attention twin network proposed in the present invention;

[0025] Figure 5 This is a framework diagram of the target tracking system based on the recursive expanded attention twin network proposed in the present invention. DETAILED DESCRIPTION

[0026] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0027] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0028] See also Figure 1 , an embodiment of the present invention provides a target tracking method based on a twin network of recursive dilated attention, the method comprising the following steps:

[0029] Step 1. Construct a dual-branch feature extraction network based on the Transformer network and the recursive dilated attention module. The two branches of the dual-branch feature extraction network share weights. Construct a feature fusion network based on the self-attention mechanism. The target tracking model consists of the dual-branch feature extraction network, the feature fusion network and the prediction head.

[0030] Step 2: Input the template image into the target tracking model, extract features from the template image based on a dual-branch feature extraction network, and obtain template features; input the search image into the target tracking model, extract features from the search image based on a dual-branch feature extraction network, and obtain search features; splice the template features with the search features to obtain spliced ​​features;

[0031] Please refer to Figure 2 , Figure 3 and Figure 4 In step 2, the template image is input into the target tracking model, and the template image is extracted based on the dual-branch feature extraction network to obtain the template features. The specific steps are as follows:

[0032] Transform the input template image into a labeled sequence using patch embedding;

[0033] Recursively perform dilated attention calculation on the labeled sequence to obtain intermediate features;

[0034] Repeat the above steps recursively for a specified number of times to get the attention output;

[0035] Repeat the above steps for a specified number of times to obtain the template features. The relationship between the process is:

[0036] ;

[0037] in, Represents the template feature, It means that after the normalization function operation, The query vector representing the template image, represents the transpose of the key vector obtained by linear embedding using the template image, represents the value vector obtained by linear embedding using the template image, represents the dimension of the key vector;

[0038] In step 2, the search image is input into the target tracking model, and features of the search image are extracted based on the dual-branch feature extraction network to obtain search features. The specific steps are as follows:

[0039] Transform the input search image into a labeled sequence using patch embedding;

[0040] Recursively perform dilated attention calculation on the labeled sequence to obtain intermediate features;

[0041] Repeat the above steps recursively for a specified number of times to get the attention output;

[0042] Repeat the above steps for a specified number of times to obtain the search feature. The relationship between the process is:

[0043] ;

[0044] in, Represents the search feature, represents the value vector obtained by linear embedding using the search image, represents the transpose of the key vector obtained by linear embedding using the search image, A vector of values ​​representing the search image, A query vector representing the search image;

[0045] The recursive dilated attention calculation is to perform the dilated attention in a recursive manner, and the corresponding process has the following relationship:

[0046] ;

[0047] in, Indicates expanded attention. represents a linear function, represents the step length, Indicates that at time step The input state, Indicates that at time step The input state, Indicates that at time step The input state, Indicates that at time step The hidden state of Indicates that at time step The hidden state of

[0048] The calculation formula for expanded attention is:

[0049] ;

[0050] in, represents the input feature map of the convolutional neural network, represents the query feature map obtained by convolution operation, represents the kernel weight, Represents the feature map obtained through convolution operation, represents the query vector, represents the feature channel, represents the size of the convolution kernel, represents the dilation rate of the convolution kernel, represents the number of groups of convolution kernels, represents the convolution operation, represents the Sigmoid function, Represents the input value;

[0051] Input the template feature map as the input feature map of the convolutional neural network into the above formula to obtain the query vector of the template image ;

[0052] The search feature map is used as the input feature map of the convolutional neural network and input into the above formula to obtain the query vector of the search image. ;

[0053] In step 2, the template feature is concatenated with the search feature to obtain the concatenated feature. The relationship between the process is:

[0054] ;

[0055] in, represents the features after splicing, express The output vector of The concatenation vector of the output vector of express The output vector and The concatenation vector of the output vector of .

[0056] Step 3: Input the concatenated features into the classification regression prediction head and perform calculations to obtain the classification regression results, and predict the target based on the classification regression results.

[0057] Step 4: Repeat steps 1 to 3 in an iterative manner in combination with a large-scale data set to train the target tracking model, obtain a trained target tracking model, and use the trained target tracking model to accurately track the target;

[0058] In step 4, the target tracking model is trained by repeating steps 1 to 3 in an iterative manner in combination with a large-scale dataset, wherein the training datasets used include: GOT-10k, laSOT, COCO and TrackingNet, and the test datasets include: GOT-10k, TrackingNet, LaSOT, NFS, TNL2K and UAV123;

[0059] In the step of training the target tracking model by iteratively repeating steps 1 to 3, the parameters in the target tracking model are adjusted using the total loss function, where the total loss is composed of weighted focal loss, GIoU loss and regression loss. The weighted focal loss expression is:

[0060] ;

[0061] in, represents the probability that the model predicts the positive category, Represents the weight coefficient for balancing positive and negative samples, represents the parameter that adjusts the model's attention to difficult and easy samples, represents the weighted focal loss;

[0062] GIoU Loss:

[0063] ;

[0064] in, represents the intersection-over-union ratio between the predicted bounding box and the true bounding box, represents the area of ​​the bounding box, represents the true bounding box, represents the predicted bounding box, represents the minimum closed area of ​​the predicted bounding box and the true bounding box, represents the diagonal length of the minimum enclosed area that protects the two bounding boxes, represents a small constant, represents GIOU loss;

[0065] Absolute value loss:

[0066] ;

[0067] in, represents the absolute value loss, represents the true value, represents the predicted value, represents the number of samples;

[0068] The total loss expression is:

[0069] ;

[0070] in, represents the total loss, Represents the weight coefficient used to adjust the contribution of GIoU loss in the total loss, Represents the weight coefficient used to adjust the contribution of absolute value loss in the total loss.

[0071] See also Figure 5 , an embodiment of the present invention further provides a target tracking system based on a twin network of recursive dilated attention, the system comprising:

[0072] Model building module, a dual-branch feature extraction network is built based on the Transformer network and the recursive dilation attention module. The two branches of the dual-branch feature extraction network share weights, and a feature fusion network is built based on the self-attention mechanism. The target tracking model consists of the dual-branch feature extraction network, the feature fusion network and the prediction head;

[0073] The feature extraction and fusion module inputs the template image into the target tracking model, extracts features from the template image based on the dual-branch feature extraction network, and obtains the template features; inputs the search image into the target tracking model, extracts features from the search image based on the dual-branch feature extraction network, and obtains the search features; splices the template features with the search features to obtain the spliced ​​features;

[0074] The prediction module inputs the concatenated features into the classification and regression prediction head and performs calculations to obtain the classification and regression results, and predicts the target based on the classification and regression results;

[0075] The target tracking module uses a large-scale data set to iteratively repeat steps 1 to 3 to train the target tracking model to obtain a trained target tracking model, and uses the trained target tracking model to accurately track the target.

[0076] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0078] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A target tracking method based on a recursive dilated attention twin network, characterized in that: The method comprises the following steps: Step 1: Based on the Transformer network and the recursive dilation attention module, a dual-branch feature extraction network is constructed. The two branches of the dual-branch feature extraction network share weights, and a feature fusion network is constructed based on the self-attention mechanism. The target tracking model consists of the dual-branch feature extraction network, the feature fusion network and the prediction head; Step 2: Input the template image into the target tracking model, extract features from the template image based on a dual-branch feature extraction network, and obtain template features; The search image is input into the target tracking model, and features of the search image are extracted based on a dual-branch feature extraction network to obtain search features; the template features are spliced ​​with the search features to obtain spliced ​​features; Step 3: Input the concatenated features into the classification regression prediction head and perform calculations to obtain the classification regression results, and predict the target based on the classification regression results; Step 4: Repeat steps 1 to 3 in an iterative manner in combination with a large-scale data set to train the target tracking model, obtain a trained target tracking model, and use the trained target tracking model to accurately track the target; In step 2, the template image is input into the target tracking model, and features of the template image are extracted based on a dual-branch feature extraction network to obtain template features. The specific steps are as follows: Transform the input template image into a labeled sequence using patch embedding; Recursively perform dilated attention calculation on the labeled sequence to obtain intermediate features; Repeat the above steps recursively for a specified number of times to get the attention output; Repeat the above steps for a specified number of times to obtain the template features. The relationship between the process is: ； in, Represents the template feature, It means that after the normalization function operation, The query vector representing the template image, represents the transpose of the key vector obtained by linear embedding using the template image, represents the value vector obtained by linear embedding using the template image, represents the dimension of the key vector; The search image is input into the target tracking model, and the search image is extracted based on the dual-branch feature extraction network to obtain the search features. The specific steps are as follows: Transform the input search image into a labeled sequence using patch embedding; Recursively perform dilated attention calculation on the labeled sequence to obtain intermediate features; Repeat the above steps recursively for a specified number of times to get the attention output; Repeat the above steps for a specified number of times to obtain the search feature. The relationship between the process is: ； in, Represents the search feature, represents the value vector obtained by linear embedding using the search image, represents the transpose of the key vector obtained by linear embedding using the search image, A vector of values ​​representing the search image, A query vector representing the search image; The recursive dilated attention calculation is to perform the dilated attention in a recursive manner, and the corresponding process has the following relationship: ； in, Indicates expanded attention. represents a linear function, represents the step length, Indicates that at time step The input state, Indicates that at time step The input state, Indicates that at time step The input state, Indicates that at time step The hidden state of Indicates that at time step The hidden state of The calculation formula for expanded attention is: ； in, represents the input feature map of the convolutional neural network, represents the query feature map obtained by convolution operation, represents the kernel weight, Represents the feature map obtained through convolution operation, represents the query vector, represents the feature channel, represents the size of the convolution kernel, represents the dilation rate of the convolution kernel, represents the number of groups of convolution kernels, represents the convolution operation, represents the Sigmoid function, Represents the input value; Input the template feature map as the input feature map of the convolutional neural network into the above formula to obtain the query vector of the template image ; The search feature map is used as the input feature map of the convolutional neural network and input into the above formula to obtain the query vector of the search image. .

2. The target tracking method based on the recursive dilated attention twin network according to claim 1, characterized in that: In step 2, the template feature and the search feature are spliced ​​to obtain the spliced ​​feature. The relationship in the process is: ； in, represents the features after splicing, express The output vector of The concatenation vector of the output vector of express The output vector and The concatenation vector of the output vector of .

3. The target tracking method based on the recursive dilated attention twin network according to claim 2, characterized in that: In step 4, steps 1 to 3 are iteratively repeated in combination with a large-scale dataset to train the target tracking model, wherein the training datasets used include GOT-10k, laSOT, COCO and TrackingNet, and the test datasets include GOT-10k, TrackingNet, LaSOT, NFS, TNL2K and UAV123.

4. The target tracking method based on the recursive dilated attention twin network according to claim 3 is characterized in that: In the step of training the target tracking model by iteratively repeating steps 1 to 3, the parameters in the target tracking model are adjusted using the total loss function, where the total loss is composed of weighted focal loss, GIoU loss and regression loss. The weighted focal loss expression is: ； in, represents the probability that the model predicts the positive category, Represents the weight coefficient for balancing positive and negative samples, represents the parameter that adjusts the model's attention to difficult and easy samples, represents the weighted focal loss; GIoU Loss: ； in, represents the intersection-over-union ratio between the predicted bounding box and the true bounding box, represents the area of ​​the bounding box, represents the true bounding box, represents the predicted bounding box, represents the minimum closed area of ​​the predicted bounding box and the true bounding box, represents the diagonal length of the minimum enclosed area that protects the two bounding boxes, represents a small constant, represents GIOU loss; Absolute value loss: ； in, represents the absolute value loss, represents the true value, represents the predicted value, represents the number of samples; The total loss expression is: ； in, represents the total loss, Represents the weight coefficient used to adjust the contribution of GIoU loss in the total loss, Represents the weight coefficient used to adjust the contribution of absolute value loss in the total loss.

5. A target tracking system based on a recursive dilated attention twin network, characterized in that: The system applies a target tracking method based on a twin network of recursive dilated attention according to any of claims 1 to 4 above, and the system comprises: Model building module, a dual-branch feature extraction network is built based on the Transformer network and the recursive dilation attention module. The two branches of the dual-branch feature extraction network share weights, and a feature fusion network is built based on the self-attention mechanism. The target tracking model consists of the dual-branch feature extraction network, the feature fusion network and the prediction head; The feature extraction and fusion module inputs the template image into the target tracking model, extracts features from the template image based on the dual-branch feature extraction network, and obtains the template features; inputs the search image into the target tracking model, extracts features from the search image based on the dual-branch feature extraction network, and obtains the search features; splices the template features with the search features to obtain the spliced ​​features; The prediction module inputs the concatenated features into the classification and regression prediction head and performs calculations to obtain the classification and regression results, and predicts the target based on the classification and regression results; The target tracking module uses a large-scale data set to iteratively repeat steps 1 to 3 to train the target tracking model to obtain a trained target tracking model, and uses the trained target tracking model to accurately track the target.

Images