Target tracking method and system based on multi-scale features and collinear attention module

Abstract

The invention provides a target tracking method and system based on multi-scale features and a collinear attention module. The method comprises the following steps: constructing a feature fusion network model; taking the template image and the search area image as input features of a single-residual multi-scale block to obtain output features of a single-residual multi-scale feature block; in the template branch, processing the output features of the single-residual multi-scale feature block of the template branch to obtain final self-attention; pre-training the feature fusion network model in combination with the final self-attention; taking the template image and the search area image as input features of a pre-trained feature fusion network model to obtain new output features of a single-residual multi-scale feature block, and further obtaining a maximum similarity score; and obtaining a prediction result according to the maximum similarity score. According to the method, the advantages of single-residual multi-scale features and collinear constraint attention are fully combined, so that better feature fusion is achieved, redundant calculation is reduced, and the balance between performance and speed is optimized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing technology, and in particular to a target tracking method and system based on multi-scale features and a collinear attention module. Background Art

[0002] Visual object tracking is a fundamental task in computer vision, which aims to track an arbitrary object given its initial state in a video sequence. In recent years, tracking technology has made remarkable progress with the development of deep neural networks. In particular, the use of Transformers has played a key role in the development of several high-performance trackers. However, most of the recent research works have focused only on achieving high performance without considering tracking speed.

[0003] While these state-of-the-art trackers can provide real-time performance on powerful GPUs, their efficiency decreases on devices with limited computational resources. For example, ARTrack, considered a top tracker, achieves a tracking speed of 37 frames per second (fps) on an NVIDIA RTX 2080Ti GPU, but drops to 5 fps on a common edge device, the Nvidia Jetson Orin NX. Tracking efficiency is critical for practical robotics applications, especially on edge devices. Earlier approaches such as ECO and ATOM focused on real-time operation but did not achieve the level of accuracy of the new trackers. Recent advances have adopted a backbone of lightweight designs to achieve efficient real-time tracking. However, these solutions still show a performance gap compared to SOTA heavyweight trackers.

[0004] Traditional efficient trackers mainly seek to achieve faster runtimes by directly using networks that were originally designed to be lightweight as their backbones. However, these lightweight networks are designed for efficiency, which leads to relatively mediocre performance on their upstream tasks such as image classification. Summary of the invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a target tracking method and system based on multi-scale features and co-linear attention modules to solve the above technical problems.

[0006] The present invention proposes a target tracking method based on multi-scale features and a collinear attention module, the method comprising the following steps: Step 1: Construct a feature fusion network model including a template branch and a search branch, wherein the template branch consists of a six-layer single residual multi-scale feature block and a collinear constrained self-attention block, and the search branch consists of a three-layer single residual multi-scale feature block; Step 2: Using the template image and the search area image as input features of the single residual multi-scale block, the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block are obtained; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output; Step 3: Use a large-scale data set to pre-train the feature fusion network model to obtain a pre-trained feature fusion network model; Step 4: Use the template image and the search area image as input features of the pre-trained feature fusion network model to obtain the output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block; The output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted as a series sequence to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score; Step 5: According to the maximum similarity score, the part belonging to the final sequence search is flattened into a 2D feature map through the decoder; Step 6: Input the 2D feature map into the head network, select the position with the highest confidence in the center score map as the target position according to the maximum similarity score, and use the corresponding regression coordinates to calculate a bounding box as the final prediction result.

[0007] The present invention also proposes a target tracking system based on multi-scale features and a collinear attention module, the system comprising: Building blocks for: Construct a feature fusion network model including a template branch and a search branch, where the template branch consists of a six-layer single residual multi-scale feature block and a collinear constrained self-attention block, and the search branch consists of a three-layer single residual multi-scale feature block; Feature extraction module for: The template image and the search area image are used as input features of the single residual multi-scale block, and the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block are obtained; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output; Pre-trained modules for: Using a large-scale data set, the feature fusion network model is pre-trained to obtain a pre-trained feature fusion network model; Encoding module for: The template image and the search area image are used as input features of the pre-trained feature fusion network model to obtain output features of a new template branch single residual multi-scale feature block and output features of a new search branch single residual multi-scale feature block; The output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted as a series sequence to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score; Decoding module for: According to the maximum similarity score, the part belonging to the final sequence search is flattened into a 2D feature map through the decoder; Prediction module for: The 2D feature map is input into the head network, and according to the maximum similarity score, the position with the highest confidence in the center score map is selected as the target position, and a bounding box is calculated using the corresponding regression coordinates as the final prediction result.

[0008] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention adopts a single residual multi-scale feature module in the template branch and the search branch. Compared with the traditional double residual structure, the single residual structure requires less memory access and saves costs. At the same time, the multi-scale features are used to enhance the network's long-distance learning ability; 2. A collinear constrained self-attention module is used in the template branch, which enhances the feature aggregation ability of the template branch compared with traditional self-attention; 3. The present invention connects the image features extracted by the template branch and the image features extracted by the search branch in series and sends them to the self-attention layer, balancing the accuracy and speed of the network.

[0009] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of the target tracking method based on multi-scale features and collinear attention module proposed in the present invention; Figure 2 This is a structural diagram of a feature fusion network and a target tracking framework of a target tracking method based on multi-scale features and collinear attention modules proposed in the present invention; Figure 3 A structural diagram of a single residual multi-scale feature module of a target tracking method based on multi-scale features and a collinear attention module proposed in the present invention; Figure 4 Schematic diagram of the overall framework of the target tracking system based on multi-scale features and co-linear attention modules proposed in the present invention. DETAILED DESCRIPTION

[0011] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0012] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0013] See also Figure 1 The embodiment of the present invention proposes a target tracking method based on multi-scale features and a collinear attention module, the method comprising the following steps: Step 1: Construct a feature fusion network model including a template branch and a search branch, wherein the template branch consists of a six-layer single residual multi-scale feature block and a collinear constrained self-attention block, and the search branch consists of a three-layer single residual multi-scale feature block.

[0014] Step 2: Using the template image and the search area image as input features of the single residual multi-scale block, the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block are obtained; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output; See also Figure 2 and Figure 3 In step 2, the template image and the search area image are used as the input features of the single residual multi-scale block, and the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block are obtained. The specific steps of the working principle of the single residual multi-scale block are as follows: The input features are dynamically segmented to generate four components. The corresponding relationship is: ; in, Indicates that after the channel splitting operation, represents the input features, , , and Both represent the components of the input features; According to the four components, a refined representation matrix is ​​obtained, and the relationship between the corresponding process is: ; in, Indicates that it has been processed by a deep convolution of size 3×3 pixels; Indicates that at a specific level, the features are upsampled to the original resolution by the closest interpolation; Indicates the original resolution, It means that after downsampling to a resolution of processing, , , and Both represent multi-scale features. Representation matrix representing refinement; The input features are expanded and the channel dimension is calculated to obtain the expansion matrix. The corresponding process relationship is: ; in, represents the expansion matrix, Represents the calculation after expanding the channel dimension; Based on the expansion matrix and the refined representation matrix, the output features of the single residual multi-scale feature block are obtained. The relationship between the corresponding process is: ; in, It means that after the depth convolution operation with a convolution kernel size of 7×7, represents the skip connection operation, It represents the result obtained after deep convolution with a convolution kernel size of 7×7. It represents the result of expansion after deep convolution with a convolution kernel size of 7×7. Represents the output features of the single residual multi-scale feature block; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output. The working principle of the collinear constrained self-attention block is as follows: The query vector is obtained by word embedding vector, and the relationship between the corresponding process is: ; in, represents the query vector, represents the weight matrix of the query, represents the word embedding vector, represents the index of the query vector, Represents the index of the word embedding vector; A constraint coefficient is introduced for each marking position, and the relationship between the corresponding process is: ; in, represents the constraint coefficient of the word embedding vector, The weight matrix representing the keys without constraint coefficients added; Applying collinearity conditions to the constraint coefficients, the corresponding process relationship is: ; in, and are constraint coefficients, represents a positive integer; The key vector is obtained based on the query vector and the constraint coefficient. The corresponding relationship in the process is: ; in, represents the key vector, Represents an array consisting of elements, represents element-wise multiplication, All represent elements, represents the total number of elements, Represents the index of the key vector; Apply rotation position encoding to the query vector and the key vector to obtain the encoded query vector and the encoded key vector. The relationship between the corresponding processes is: ; in, represents the operation after rotational position encoding, represents the encoded query vector, represents the encoded key vector; Based on the encoded query vector and the encoded key vector, self-attention is obtained. The relationship between the corresponding process is: ; in, represents the attention score, represents the final self-attention, Represents a vector of values.

[0015] Step 3: Use a large-scale data set to pre-train the feature fusion network model to obtain a pre-trained feature fusion network model.

[0016] Step 4: Use the template image and the search area image as input features of the pre-trained feature fusion network model to obtain the output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block; The output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted as a series sequence to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score; In step 4, the output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted to the multi-head self-attention encoding layer as a series sequence for calculation to obtain the maximum similarity score. The relationship between the corresponding process is: ; in, represents the maximum similarity score, The query vector representing the search area image, The transpose of the key vector representing the search area image, represents the transpose of the key vector of the template image, represents the dimension of the key vector, A vector of values ​​representing the image of the search area, A vector of values ​​representing the template image.

[0017] Step 5: According to the maximum similarity score, the part belonging to the final sequence search is flattened into a 2D feature map through the decoder.

[0018] Step 6: Input the 2D feature map into the head network, select the position with the highest confidence in the center score map as the target position according to the maximum similarity score, and use the corresponding regression coordinates to calculate a bounding box as the final prediction result.

[0019] See also Figure 4 The embodiment of the present invention further provides a target tracking system based on multi-scale features and a collinear attention module, the system comprising: Building blocks for: Construct a feature fusion network model including a template branch and a search branch, where the template branch consists of a six-layer single residual multi-scale feature block and a collinear constrained self-attention block, and the search branch consists of a three-layer single residual multi-scale feature block; Feature extraction module for: The template image and the search area image are used as input features of the single residual multi-scale block, and the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block are obtained; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output; Pre-trained modules for: Using a large-scale data set, the feature fusion network model is pre-trained to obtain a pre-trained feature fusion network model; Encoding module for: The template image and the search area image are used as input features of the pre-trained feature fusion network model to obtain output features of a new template branch single residual multi-scale feature block and output features of a new search branch single residual multi-scale feature block; The output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted as a series sequence to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score; Decoding module for: According to the maximum similarity score, the part belonging to the final sequence search is flattened into a 2D feature map through the decoder; Prediction module for: The 2D feature map is input into the head network, and according to the maximum similarity score, the position with the highest confidence in the center score map is selected as the target position, and a bounding box is calculated using the corresponding regression coordinates as the final prediction result.

[0020] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0021] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0022] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A target tracking method based on multi-scale features and collinear attention module, characterized in that: The method comprises the following steps: Step 1: Construct a feature fusion network model including a template branch and a search branch, wherein the template branch consists of a six-layer single residual multi-scale feature block and a collinear constrained self-attention block, and the search branch consists of a three-layer single residual multi-scale feature block; Step 2: Using the template image and the search area image as input features of the single residual multi-scale block, the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block are obtained; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output; Step 3: Use a large-scale data set to pre-train the feature fusion network model to obtain a pre-trained feature fusion network model; Step 4: Use the template image and the search area image as input features of the pre-trained feature fusion network model to obtain the output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block; The output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted as a series sequence to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score; Step 5: According to the maximum similarity score, the part belonging to the final sequence search is flattened into a 2D feature map through the decoder; Step 6: Input the 2D feature map into the head network, select the position with the highest confidence in the center score map as the target position according to the maximum similarity score, and use the corresponding regression coordinates to calculate a bounding box as the final prediction result.

2. The target tracking method based on multi-scale features and collinear attention module according to claim 1, characterized in that: In step 2, the template image and the search area image are used as input features of the single residual multi-scale block to obtain output features of the template branch single residual multi-scale feature block and output features of the search branch single residual multi-scale feature block, wherein the working principle of the single residual multi-scale block is as follows: The input features are dynamically segmented to generate four components. The corresponding relationship is: ； in, Indicates that after the channel splitting operation, represents the input features, , , and Both represent the components of the input features; According to the four components, a refined representation matrix is ​​obtained, and the relationship between the corresponding process is: ； in, Indicates that it has been processed by a deep convolution of size 3×3 pixels; Indicates that at a specific level, the features are upsampled to the original resolution by the closest interpolation; Indicates the original resolution, It means that after downsampling to a resolution of processing, , , and Both represent multi-scale features. Represents the refined representation matrix.

3. The target tracking method based on multi-scale features and collinear attention module according to claim 2, characterized in that: The working principle of the single residual multi-scale block also includes the following steps: The input features are expanded and the channel dimension is calculated to obtain the expansion matrix. The corresponding process relationship is: ； in, represents the expansion matrix, Represents the calculation after expanding the channel dimension; Based on the expansion matrix and the refined representation matrix, the output features of the single residual multi-scale feature block are obtained. The relationship between the corresponding process is: ； in, It means that after the depth convolution operation with a convolution kernel size of 7×7, represents the skip connection operation, It represents the result obtained after deep convolution with a convolution kernel size of 7×7. It represents the result of expansion after deep convolution with a convolution kernel size of 7×7. Represents the output features of the single residual multi-scale feature block.

4. The target tracking method based on multi-scale features and collinear attention module according to claim 3 is characterized in that: In step 2, in the template branch, the output features of the single residual multi-scale feature block of the template branch are processed by the collinear constraint self-attention block to obtain the final self-attention output, wherein the working principle of the collinear constraint self-attention block is specifically as follows: The query vector is obtained by word embedding vector, and the relationship between the corresponding process is: ； in, represents the query vector, represents the weight matrix of the query, represents the word embedding vector, represents the index of the query vector, Represents the index of the word embedding vector; A constraint coefficient is introduced for each marking position, and the relationship between the corresponding process is: ； in, represents the constraint coefficient of the word embedding vector, The weight matrix representing the keys without constraint coefficients added; Applying collinearity conditions to the constraint coefficients, the corresponding process relationship is: ； in, and are constraint coefficients, represents a positive integer; The key vector is obtained based on the query vector and the constraint coefficient. The corresponding relationship in the process is: ； in, represents the key vector, Represents an array consisting of elements, represents element-wise multiplication, All represent elements, Represents the total number of elements, Represents an index into the key vector.

5. The target tracking method based on multi-scale features and collinear attention module according to claim 4, characterized in that: The working principle of the collinear constraint self-attention block also includes the following steps: Apply rotation position encoding to the query vector and the key vector to obtain the encoded query vector and the encoded key vector. The relationship between the corresponding processes is: ； in, represents the operation after rotational position encoding, represents the encoded query vector, Represents the encoded key vector.

6. The target tracking method based on multi-scale features and collinear attention module according to claim 5, characterized in that: The working principle of the collinear constraint self-attention block also includes the following steps: Based on the encoded query vector and the encoded key vector, self-attention is obtained. The relationship between the corresponding process is: ； in, represents the attention score, represents the final self-attention, Represents a vector of values.

7. The target tracking method based on multi-scale features and collinear attention module according to claim 6, characterized in that: In step 4, the output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted to the multi-head self-attention encoding layer as a series sequence for calculation to obtain the maximum similarity score. The relationship between the corresponding process is: ； in, represents the maximum similarity score, The query vector representing the search area image, The transpose of the key vector representing the search area image, represents the transpose of the key vector of the template image, represents the dimension of the key vector, A vector of values ​​representing the image of the search area, A vector of values ​​representing the template image.

8. A target tracking system based on multi-scale features and collinear attention module, characterized in that: The system applies the target tracking method based on multi-scale features and co-linear attention module according to any one of claims 1 to 7, and the system comprises: Building blocks for: Construct a feature fusion network model including a template branch and a search branch, where the template branch consists of a six-layer single residual multi-scale feature block and a collinear constrained self-attention block, and the search branch consists of a three-layer single residual multi-scale feature block; Feature extraction module for: The template image and the search area image are used as input features of the single residual multi-scale block, and the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block are obtained; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output; Pre-trained modules for: Using a large-scale data set, the feature fusion network model is pre-trained to obtain a pre-trained feature fusion network model; Encoding module for: The template image and the search area image are used as input features of the pre-trained feature fusion network model to obtain output features of a new template branch single residual multi-scale feature block and output features of a new search branch single residual multi-scale feature block; The output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted as a series sequence to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score; Decoding module for: According to the maximum similarity score, the part belonging to the final sequence search is flattened into a 2D feature map through the decoder; Prediction module for: The 2D feature map is input into the head network, and according to the maximum similarity score, the position with the highest confidence in the center score map is selected as the target position, and a bounding box is calculated using the corresponding regression coordinates as the final prediction result.

Images