Object Tracking Method and System Based on Multi-Scale Features and Collinear Attention Module

Abstract

The present invention proposes an object tracking method and system based on multi-scale features and collinear attention modules. The method includes: constructing a feature fusion network model; using the template image and the search area image as the input features of the single-residual multi-scale block to obtain the output features of the single-residual multi-scale feature block; in the template branch, processing the output features of the single-residual multi-scale feature block in the template branch to obtain the final self-attention; combining the final self-attention to pre-train the feature fusion network model; using the template image and the search area image as the input features of the pre-trained feature fusion network model to obtain the output features of the new single-residual multi-scale feature block, and further obtaining the maximum similarity score; obtaining the prediction result according to the maximum similarity score. The present invention fully combines the advantages of single-residual multi-scale features and collinear constraint attention to obtain better feature fusion and reduce redundant calculations, optimizing the balance between performance and speed.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and particularly to an object tracking method and system based on multi-scale features and a collinear attention module. Background Art

[0002] Visual object tracking is a fundamental task in computer vision, which aims to track an arbitrary object in a given video sequence from its initial state. In recent years, with the development of deep neural networks, significant progress has been made in tracking technology. In particular, the use of Transformer has played a key role in the development of several high-performance trackers. However, most of the recent research work has focused only on achieving high performance without considering the tracking speed.

[0003] Although these state-of-the-art trackers can provide real-time performance on powerful GPUs, their efficiency decreases on devices with limited computing resources. For example, ARTrack, considered a top tracker, reaches a tracking speed of 37 frames per second (fps) on an NVIDIA RTX 2080Ti GPU, but drops to 5 frames per second on a common edge device, the Nvidia Jetson Orin NX. Tracking efficiency is crucial for practical robotic applications, especially on edge devices. Early methods such as ECO and ATOM focused on real-time operation but did not reach the accuracy level of the new trackers. Recent progress has adopted lightweight-designed backbones to achieve efficient real-time tracking. However, compared with SOTA heavyweight trackers, these solutions still show a performance gap.

[0004] Traditional efficient trackers mainly seek to achieve faster running times by directly using networks originally designed as lightweight as their backbones. However, these lightweight networks are designed for efficiency, which results in relatively mediocre performance in their upstream tasks such as image classification. Summary of the Invention

[0005] In view of the above situation, the main object of the present invention is to propose an object tracking method and system based on multi-scale features and a collinear attention module to solve the above technical problems.

[0006] The present invention proposes an object tracking method based on multi-scale features and a collinear attention module, and the method includes the following steps:

[0007] Step 1, construct a feature fusion network model including a template branch and a search branch, wherein the template branch is composed of six layers of single-residual multi-scale feature blocks and a collinear constraint self-attention block, and the search branch is composed of three layers of single-residual multi-scale feature blocks;

[0008] Step 2: Using the template image and the search region image as the input features of the single-residual multi-scale feature block, obtain the output features of the template branch single-residual multi-scale feature block and the output features of the search branch single-residual multi-scale feature block;

[0009] In the template branch, use the collinear constraint self-attention block to process the output features of the template branch single-residual multi-scale feature block to obtain the final self-attention output;

[0010] Step 3: Use a large-scale dataset to pre-train the feature fusion network model to obtain the pre-trained feature fusion network model;

[0011] Step 4: Use the template image and the search region image as the input features of the pre-trained feature fusion network model to obtain the new output features of the template branch single-residual multi-scale feature block and the new output features of the search branch single-residual multi-scale feature block;

[0012] Feed the output features of the new template branch single-residual multi-scale feature block and the new output features of the search branch single-residual multi-scale feature block as a concatenated sequence into the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score;

[0013] Step 5: According to the maximum similarity score, use the decoder to flatten the part belonging to the final sequence search into a 2D feature map;

[0014] Step 6: Input the 2D feature map into the head network. According to the maximum similarity score, select the position with the highest confidence in the center score map as the target position, and use the corresponding regression coordinates to calculate a bounding box as the final prediction result.

[0015] The present invention also proposes an object tracking system based on multi-scale features and a collinear attention module, and the system includes:

[0016] A construction module, used for:

[0017] Construct a feature fusion network model including a template branch and a search branch, where the template branch is composed of six layers of single-residual multi-scale feature blocks and a collinear constraint self-attention block, and the search branch is composed of three layers of single-residual multi-scale feature blocks;

[0018] A feature extraction module, used for:

[0019] Using the template image and the search region image as the input features of the single-residual multi-scale feature block to obtain the output features of the template branch single-residual multi-scale feature block and the output features of the search branch single-residual multi-scale feature block;

[0020] In the template branch, the output features of the template branch single-residual multi-scale feature block are processed using a collinear constraint self-attention block to obtain the final self-attention output;

[0021] A pre-training module for:

[0022] Using a large-scale dataset to pre-train the feature fusion network model to obtain a pre-trained feature fusion network model;

[0023] An encoding module for:

[0024] Taking the template image and the search region image as the input features of the pre-trained feature fusion network model to obtain the output features of the new template branch single-residual multi-scale feature block and the output features of the new search branch single-residual multi-scale feature block;

[0025] Taking the output features of the new template branch single-residual multi-scale feature block and the output features of the new search branch single-residual multi-scale feature block as a concatenated sequence and feeding them into the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score;

[0026] A decoding module for:

[0027] According to the maximum similarity score, through the decoder, the part belonging to the final sequence search is flattened into a 2D feature map;

[0028] A prediction module for:

[0029] Inputting the 2D feature map into the head network, according to the maximum similarity score, selecting the position with the highest confidence in the center score map as the target position, and using the corresponding regression coordinates to calculate a bounding box as the final prediction result.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] 1. The present invention adopts a single-residual multi-scale feature module in the template branch and the search branch. Compared with the traditional double-residual structure, the single-residual structure requires less memory access and saves costs. At the same time, multi-scale features are used to enhance the long-distance learning ability of the network;

[0032] 2. A collinear constraint self-attention module is used in the template branch, which enhances the feature aggregation ability of the template branch compared with traditional self-attention;

[0033] 3. The present invention concatenates the image features extracted from the template branch and the image features extracted from the search branch and then feeds them into the self-attention layer, balancing the accuracy and speed of the network.

[0034] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the embodiments of the present invention. Description of the Drawings

[0035] Figure 1 It is a flowchart of the object tracking method based on multi-scale features and collinear attention module proposed by the present invention;

[0036] Figure 2 It is a structural diagram of the feature fusion network and the object tracking framework of the object tracking method based on multi-scale features and collinear attention module proposed by the present invention;

[0037] Figure 3 It is a structural diagram of the single-residual multi-scale feature module of the object tracking method based on multi-scale features and collinear attention module proposed by the present invention;

[0038] Figure 4 It is a schematic diagram of the overall framework of the object tracking system based on multi-scale features and collinear attention module proposed by the present invention. Detailed Embodiments

[0039] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0040] Referring to the following description and drawings, these and other aspects of the embodiments of the present invention will become clear. In these descriptions and drawings, some specific embodiments of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention. However, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0041] Please refer to Figure 1 , the embodiments of the present invention propose an object tracking method based on multi-scale features and collinear attention module. The method includes the following steps:

[0042] Step 1: Construct a feature fusion network model including a template branch and a search branch. Among them, the template branch is composed of six layers of single-residual multi-scale feature blocks and a collinear constraint self-attention block, and the search branch is composed of three layers of single-residual multi-scale feature blocks.

[0043] Step 2: Use the template image and the search area image as the input features of the single-residual multi-scale feature blocks to obtain the output features of the single-residual multi-scale feature blocks of the template branch and the output features of the single-residual multi-scale feature blocks of the search branch;

[0044] In the template branch, the output features of the single-residual multi-scale feature block in the template branch are processed using the collinear constraint self-attention block to obtain the final self-attention output;

[0045] Please refer to Figure 2 and Figure 3 , in step 2, the template image and the search area image are used as the input features of the single-residual multi-scale feature block to obtain the output features of the single-residual multi-scale feature block in the template branch and the output features of the single-residual multi-scale feature block in the search branch. The specific steps of the working principle of the single-residual multi-scale feature block are as follows:

[0046] The input features are subjected to dynamic segmentation operations to generate four component parts. The relational expressions existing in the corresponding process are:

[0047] ;

[0048] Among them, represents the result after channel segmentation operation, represents the input features, , , and all represent the component parts of the input features;

[0049] Based on the four component parts, a refined representation matrix is obtained. The relational expressions existing in the corresponding process are:

[0050] ;

[0051] Among them, represents the result after depth convolution processing with a size of 3×3 pixels; represents that at a specific level, the features are upsampled to the original resolution through the closest interpolation; represents the original resolution, represents the result after downsampling to a resolution size of , , , and all represent multi-scale features, represents the refined representation matrix;

[0052] The input features are subjected to extended channel dimension calculation to obtain an extended matrix. The relational expressions existing in the corresponding process are:

[0053] ;

[0054] Among them, represents the extended matrix, represents the result of extended channel dimension calculation; ​​

[0055] The output features of the single-residual multi-scale feature block are obtained based on the extended matrix and the refined representation matrix. The relational expressions in the corresponding process are as follows:

[0056] ;

[0057] Among them, represents the depth convolution operation with a convolution kernel size of 7×7, represents the skip connection operation, represents the result obtained after the depth convolution processing with a convolution kernel size of 7×7, represents the result after extension following the depth convolution processing with a convolution kernel size of 7×7, represents the output features of the single-residual multi-scale feature block;

[0058] In the template branch, the output features of the single-residual multi-scale feature block in the template branch are processed using the collinear constraint self-attention block to obtain the final self-attention output. Among them, the specific steps of the working principle of the collinear constraint self-attention block are as follows:

[0059] The query vector is obtained through the word embedding vector. The relational expressions in the corresponding process are as follows:

[0060] ;

[0061] Among them, represents the query vector, represents the weight matrix of the query, represents the word embedding vector, represents the index of the query vector, represents the index of the word embedding vector;

[0062] Constraint coefficients are introduced for each token position. The relational expressions in the corresponding process are as follows:

[0063] ;

[0064] Among them, represents the constraint coefficient of the word embedding vector, represents the weight matrix of the key without the added constraint coefficient;

[0065] The collinearity condition is imposed on the constraint coefficients. The relational expressions in the corresponding process are as follows:

[0066] ;

[0067] Among them, and both represent constraint coefficients, represents a positive integer;

[0068] The key vector is obtained based on the query vector and the constraint coefficient, and the relational expression in the corresponding process is:

[0069] ;

[0070] Among them, represents the key vector, represents an array composed of elements, represents element-wise multiplication, both represent elements, represents the total number of elements, represents the index of the key vector;

[0071] The rotation position encoding is applied to the query vector and the key vector to obtain the encoded query vector and the encoded key vector. The relational expression in the corresponding process is:

[0072] ;

[0073] Among them, represents the operation of rotation position encoding, represents the encoded query vector, represents the encoded key vector;

[0074] The self-attention is obtained based on the encoded query vector and the encoded key vector. The relational expression in the corresponding process is:

[0075] ;

[0076] Among them, represents the attention score, represents the final self-attention, represents the value vector.

[0077] Step 3: Use a large-scale dataset to pre-train the feature fusion network model to obtain the pre-trained feature fusion network model.

[0078] Step 4: Use the template image and the search area image as the input features of the pre-trained feature fusion network model to obtain the output features of the new template branch single-residual multi-scale feature block and the output features of the new search branch single-residual multi-scale feature block;

[0079] The output features of the new template branch single-residual multi-scale feature block and the output features of the new search branch single-residual multi-scale feature block are used as a concatenated sequence and fed into the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score;

[0080] In step 4, the output features of the new template branch single-residual multi-scale feature block and the output features of the new search branch single-residual multi-scale feature block are fed as a concatenated sequence into the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score. The relational expression existing in the corresponding process is:

[0081] ;

[0082] Among them, represents the maximum similarity score, represents the query vector of the search region image, represents the transpose of the key vector of the search region image, represents the transpose of the key vector of the template image, represents the dimension of the key vector, represents the value vector of the search region image, represents the value vector of the template image.

[0083] Step 5: According to the maximum similarity score, through the decoder, the part belonging to the final sequence search is flattened into a 2D feature map.

[0084] Step 6: Input the 2D feature map into the head network. According to the maximum similarity score, select the position with the highest confidence in the center score map as the target position, and calculate a bounding box using the corresponding regression coordinates as the final prediction result.

[0085] Please refer to Figure 4 , the embodiment of the present invention also provides an object tracking system based on multi-scale features and a collinear attention module. The system includes:

[0086] A construction module, used for:

[0087] Construct a feature fusion network model including a template branch and a search branch, where the template branch is composed of six layers of single-residual multi-scale feature blocks and a collinear constraint self-attention block, and the search branch is composed of three layers of single-residual multi-scale feature blocks;

[0088] A feature extraction module, used for:

[0089] Using the template image and the search region image as the input features of the single-residual multi-scale feature block to obtain the output features of the template branch single-residual multi-scale feature block and the output features of the search branch single-residual multi-scale feature block;

[0090] In the template branch, use the collinear constraint self-attention block to process the output features of the template branch single-residual multi-scale feature block to obtain the final self-attention output;

[0091] A pre-training module, used for:

[0092] Pre-train the feature fusion network model using a large-scale dataset to obtain a pre-trained feature fusion network model;

[0093] An encoding module for:

[0094] Take the template image and the search area image as the input features of the pre-trained feature fusion network model to obtain the output features of the new template branch single-residual multi-scale feature block and the output features of the new search branch single-residual multi-scale feature block;

[0095] Take the output features of the new template branch single-residual multi-scale feature block and the output features of the new search branch single-residual multi-scale feature block as a concatenated sequence and send it to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score;

[0096] A decoding module for:

[0097] According to the maximum similarity score, use the decoder to flatten the part belonging to the final sequence search into a 2D feature map;

[0098] A prediction module for:

[0099] Input the 2D feature map into the head network. According to the maximum similarity score, select the position with the highest confidence in the center score map as the target position, and use the corresponding regression coordinates to calculate a bounding box as the final prediction result.

[0100] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following well-known technologies in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0101] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0102] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several variations and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

1. A target tracking method based on multi-scale features and collinear attention module, characterized in that: The method comprises the following steps: Step 1: Construct a feature fusion network model including a template branch and a search branch, wherein the template branch consists of a six-layer single residual multi-scale feature block and a collinear constrained self-attention block, and the search branch consists of a three-layer single residual multi-scale feature block; Step 2: Using the template image and the search area image as input features of the single residual multi-scale feature block, obtaining output features of the template branch single residual multi-scale feature block and output features of the search branch single residual multi-scale feature block; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output; Step 3: Use a large-scale data set to pre-train the feature fusion network model to obtain a pre-trained feature fusion network model; Step 4: Use the template image and the search area image as input features of the pre-trained feature fusion network model to obtain the output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block; The output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted as a series sequence to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score; Step 5: According to the maximum similarity score, the part belonging to the final sequence search is flattened into a 2D feature map through the decoder; Step 6: Input the 2D feature map into the head network, select the position with the highest confidence in the center score map as the target position according to the maximum similarity score, and use the corresponding regression coordinates to calculate a bounding box as the final prediction result.

2. The target tracking method based on multi-scale features and collinear attention module according to claim 1, characterized in that: In step 2, the template image and the search area image are used as input features of the single residual multi-scale feature block to obtain the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block. The specific steps of the working principle of the single residual multi-scale feature block are as follows: The input features are dynamically segmented to generate four components. The corresponding relationship is: ； in, Indicates that after channel splitting operation, represents the input features, , , and Both represent the components of the input features; According to the four components, a refined representation matrix is ​​obtained, and the relationship between the corresponding process is: ； in, Indicates that it has been processed by a deep convolution of size 3×3 pixels; Indicates that at a specific level, the features are upsampled to the original resolution by the closest interpolation; Indicates the original resolution, It means that after downsampling to a resolution of processing, , , and Both represent multi-scale features. Represents the refined representation matrix.

3. The target tracking method based on multi-scale features and collinear attention module according to claim 2, characterized in that: The working principle of the single residual multi-scale feature block also includes the following steps: The input features are expanded and the channel dimension is calculated to obtain the expansion matrix. The corresponding process relationship is: ； in, represents the expansion matrix, Represents the calculation after expanding the channel dimension; Based on the expansion matrix and the refined representation matrix, the output features of the single residual multi-scale feature block are obtained. The relationship between the corresponding process is: ； in, It means that after the depth convolution operation with a convolution kernel size of 7×7, represents the skip connection operation, It represents the result obtained after deep convolution with a convolution kernel size of 7×7. It represents the result of expansion after deep convolution with a convolution kernel size of 7×7. Represents the output features of the single residual multi-scale feature block.

4. The target tracking method based on multi-scale features and collinear attention module according to claim 3 is characterized in that: In step 2, in the template branch, the output features of the single residual multi-scale feature block of the template branch are processed by the collinear constraint self-attention block to obtain the final self-attention output, wherein the working principle of the collinear constraint self-attention block is specifically as follows: The query vector is obtained by word embedding vector, and the relationship between the corresponding process is: ； in, represents the query vector, represents the weight matrix of the query, represents the word embedding vector, represents the index of the query vector, Represents the index of the word embedding vector; A constraint coefficient is introduced for each marking position, and the relationship between the corresponding process is: ； in, represents the constraint coefficient of the word embedding vector, The weight matrix representing the keys without constraint coefficients added; Applying collinearity conditions to the constraint coefficients, the corresponding process relationship is: ； in, and are constraint coefficients, represents a positive integer; The key vector is obtained based on the query vector and the constraint coefficient. The corresponding relationship in the process is: ； in, represents the key vector, Represents an array consisting of elements, represents element-wise multiplication, All represent elements, Represents the total number of elements, Represents an index into the key vector.

5. The target tracking method based on multi-scale features and collinear attention module according to claim 4, characterized in that: The working principle of the collinear constraint self-attention block also includes the following steps: Apply rotation position encoding to the query vector and the key vector to obtain the encoded query vector and the encoded key vector. The relationship between the corresponding processes is: ； in, represents the operation after rotational position encoding, represents the encoded query vector, Represents the encoded key vector.

6. The target tracking method based on multi-scale features and collinear attention module according to claim 5, characterized in that: The working principle of the collinear constraint self-attention block also includes the following steps: Based on the encoded query vector and the encoded key vector, self-attention is obtained. The relationship between the corresponding process is: ； in, represents the attention score, represents the final self-attention, Represents a vector of values.

7. The target tracking method based on multi-scale features and collinear attention module according to claim 6, characterized in that: In step 4, the output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted to the multi-head self-attention encoding layer as a series sequence for calculation to obtain the maximum similarity score. The relationship between the corresponding process is: ； in, represents the maximum similarity score, The query vector representing the search area image, The transpose of the key vector representing the search area image, represents the transpose of the key vector of the template image, represents the dimension of the key vector, A vector of values ​​representing the image of the search area, A vector of values ​​representing the template image.

8. A target tracking system based on multi-scale features and collinear attention module, characterized in that: The system applies the target tracking method based on multi-scale features and co-linear attention module according to any one of claims 1 to 7, and the system comprises: Building blocks for: Construct a feature fusion network model including a template branch and a search branch, where the template branch consists of a six-layer single residual multi-scale feature block and a collinear constrained self-attention block, and the search branch consists of a three-layer single residual multi-scale feature block; Feature extraction module for: The template image and the search area image are used as input features of the single residual multi-scale feature block, and the output features of the template branch single residual multi-scale feature block and the output features of the search branch single residual multi-scale feature block are obtained; In the template branch, the collinear constrained self-attention block is used to process the output features of the single residual multi-scale feature block of the template branch to obtain the final self-attention output; Pre-trained modules for: Using a large-scale data set, the feature fusion network model is pre-trained to obtain a pre-trained feature fusion network model; Encoding module for: The template image and the search area image are used as input features of the pre-trained feature fusion network model to obtain output features of a new template branch single residual multi-scale feature block and output features of a new search branch single residual multi-scale feature block; The output features of the new template branch single residual multi-scale feature block and the output features of the new search branch single residual multi-scale feature block are transmitted as a series sequence to the multi-head self-attention encoding layer for calculation to obtain the maximum similarity score; Decoding module for: According to the maximum similarity score, the part belonging to the final sequence search is flattened into a 2D feature map through the decoder; Prediction module for: The 2D feature map is input into the head network, and according to the maximum similarity score, the position with the highest confidence in the center score map is selected as the target position, and a bounding box is calculated using the corresponding regression coordinates as the final prediction result.

Images