Target tracking method and system based on hybrid features and multi-scale fusion attention

Abstract

The present invention proposes a target tracking method and system based on hybrid features and multi-scale fusion attention. The method includes: inputting a template image and a search image into a main-stage hybrid feature module to selectively extract the global features of the template image and the search image, and using convolution operations to extract local features of the template image and the search image; inputting the output of the main-stage hybrid feature module into a multi-scale fusion module to extract features of different scales; inputting the output of the main-stage hybrid feature module and the output of the multi-scale fusion module into a secondary-stage hybrid feature module for interactive fusion; and inputting the template image and the search image into a tracking model after parameter adjustment to obtain tracking results. The present invention extracts and fuses the global features and local features of the input image through the hybrid feature module, thereby enhancing the network's ability to capture image details, while retaining local details and enriching contextual information.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing technology, and in particular to a target tracking method and system based on hybrid features and multi-scale fusion attention. Background Art

[0002] Video tracking is an important research area in computer vision, playing a key role in numerous practical applications, including autonomous driving, video surveillance, traffic management, and high-speed photography. The core task of video tracking is to identify the target object in the first frame of a video sequence and accurately predict and locate it in subsequent frames, thereby enabling continuous tracking. As the practicality and real-time performance of video tracking technology continue to improve, its application in daily life is becoming increasingly widespread, and its research value is becoming increasingly prominent. However, video tracking remains a challenging task due to complex factors such as possible object deformation, rapid motion, and occlusion.

[0003] In recent years, the introduction of the Transformer architecture has brought significant progress to the field of computer vision. Through its self-attention mechanism, the Transformer effectively explores the connections between consecutive frames, thereby acquiring rich contextual information and achieving excellent tracking performance. However, a characteristic of the self-attention mechanism is that it requires comprehensive processing of all input features. As video and image resolution continues to increase, this not only imposes a greater computational burden but also may introduce additional noise interference. Furthermore, the self-attention mechanism primarily focuses on global information, ignoring the importance of local information. Summary of the Invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a target tracking method and system based on hybrid features and multi-scale fusion attention to solve the above technical problems.

[0005] The present invention proposes a target tracking method based on hybrid features and multi-scale fusion attention, the method comprising the following steps:

[0006] Step 1: Build a hybrid feature module based on the Transformer network framework, and build a multi-scale fusion module based on the depthwise separable convolution and dynamic attention weight fusion mechanism. The hybrid feature module and the multi-scale fusion module constitute the tracking model.

[0007] Among them, the hybrid feature module includes a main stage hybrid feature module and a secondary stage hybrid feature module;

[0008] Step 2: Input the template image and the search image into the main stage mixed feature module, use the selective attention mechanism to extract the global features of the template image and the search image, use the convolution operation to extract the local features of the template image and the search image, and fuse the global features of the template image and the search image with the local features of the template image and the search image to obtain the output of the main stage mixed feature module;

[0009] Step 3: Input the output of the main stage hybrid feature module into the multi-scale fusion module, and combine it with the dynamic weights generated by the channel feature selection branch and the spatial feature selection branch to obtain the output of the multi-scale fusion module;

[0010] Step 4: Input the output of the main stage hybrid feature module and the output of the multi-scale fusion module into the secondary stage hybrid feature module for interactive fusion processing to obtain the output of the secondary stage hybrid feature module;

[0011] Calculate the classification loss based on the output of the hybrid feature module in the second stage, and use the classification loss to optimize the tracking model to obtain the optimized tracking model;

[0012] Step 5: Pre-train the optimized tracking model using a large-scale data set and adjust the model parameters to obtain a tracking model with adjusted parameters.

[0013] Step 6: Input the template image and the search image into the tracking model after parameter adjustment, and iteratively repeat steps 2 to 4 to obtain the output of the final secondary hybrid feature module.

[0014] The output of the final sub-stage mixed feature module is input to the prediction head to obtain the tracking result.

[0015] The present invention also proposes a target tracking system based on hybrid features and multi-scale fusion attention, the system comprising:

[0016] Building blocks for:

[0017] A hybrid feature module is built based on the Transformer network framework, and a multi-scale fusion module is built based on the depthwise separable convolution and dynamic attention weight fusion mechanism. The hybrid feature module and the multi-scale fusion module constitute the tracking model.

[0018] Among them, the hybrid feature module includes a main stage hybrid feature module and a secondary stage hybrid feature module;

[0019] Extraction module for:

[0020] Input the template image and the search image into the main stage mixed feature module, use the selective attention mechanism to extract the global features of the template image and the search image, use the convolution operation to extract the local features of the template image and the search image, and fuse the global features of the template image and the search image with the local features of the template image and the search image to obtain the output of the main stage mixed feature module;

[0021] Compute module for:

[0022] The output of the main stage hybrid feature module is input into the multi-scale fusion module, and the dynamic weights generated by the channel feature selection branch and the spatial feature selection branch are combined to obtain the output of the multi-scale fusion module;

[0023] Learning modules for:

[0024] The output of the main stage hybrid feature module and the output of the multi-scale fusion module are input into the secondary stage hybrid feature module for interactive fusion processing to obtain the output of the secondary stage hybrid feature module;

[0025] Calculate the classification loss based on the output of the hybrid feature module in the second stage, and use the classification loss to optimize the tracking model to obtain the optimized tracking model;

[0026] Pre-training modules for:

[0027] Pre-train the optimized tracking model using a large-scale dataset and adjust the model parameters to obtain a tracking model with adjusted parameters.

[0028] Tracking module for:

[0029] The template image and the search image are input into the tracking model after parameter adjustment. The extraction module, calculation module and learning module are repeated in an iterative manner as input. After reaching the preset number of iterations, the output of the final secondary stage hybrid feature module is obtained.

[0030] The output of the final sub-stage mixed feature module is input to the prediction head to obtain the tracking result.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention extracts and fuses the global features and local features of the input image through the hybrid feature module, thereby enhancing the network's ability to capture image details, while retaining local details and enriching contextual information;

[0033] 2. The present invention uses the selective attention in the hybrid feature module to set a threshold to filter out elements with low similarity scores and retain elements with high scores, thereby reducing the interference of irrelevant information and computational redundancy, and the performance is more obvious on large-scale data sets;

[0034] 3. The present invention provides different scale features for the input image through the multi-scale fusion module, making the generated feature representation richer and more comprehensive, optimizing the receptive field, and enhancing the feature representation capability and adapting to complex scenes.

[0035] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of the steps of the target tracking method based on hybrid features and multi-scale fusion attention proposed in the present invention.

[0037] Figure 2 This is a structural diagram of the target tracking framework of the target tracking method based on hybrid features and multi-scale fusion attention proposed in this invention.

[0038] Figure 3 This is a schematic diagram of the hybrid feature module of the target tracking method based on hybrid features and multi-scale fusion attention proposed in the present invention.

[0039] Figure 4 This is a schematic diagram of the multi-scale fusion module of the target tracking method based on hybrid features and multi-scale fusion attention proposed in the present invention.

[0040] Figure 5 This is a structural diagram of the target tracking system based on hybrid features and multi-scale fusion attention proposed in this invention. DETAILED DESCRIPTION

[0041] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0042] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to provide some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0043] See also Figure 1, an embodiment of the present invention proposes a target tracking method based on hybrid features and multi-scale fusion attention, the method comprising the following steps:

[0044] Step 1: Build a hybrid feature module based on the Transformer network framework, and build a multi-scale fusion module based on the depthwise separable convolution and dynamic attention weight fusion mechanism. The hybrid feature module and the multi-scale fusion module constitute the tracking model.

[0045] Among them, the hybrid feature module includes a main-stage hybrid feature module and a secondary-stage hybrid feature module.

[0046] Step 2: Input the template image and the search image into the main stage mixed feature module, use the selective attention mechanism to extract the global features of the template image and the search image, use the convolution operation to extract the local features of the template image and the search image, and fuse the global features of the template image and the search image with the local features of the template image and the search image to obtain the output of the main stage mixed feature module.

[0047] See also Figure 2 and Figure 3 In step 2, the template image and the search image are input into the main stage mixed feature module, the global features of the template image and the search image are extracted using the selective attention mechanism, the local features of the template image and the search image are extracted using the convolution operation, and the global features of the template image and the search image are fused with the local features of the template image and the search image to obtain the output of the main stage mixed feature module, which specifically includes the following steps:

[0048] S101: Input the image into the main stage mixed feature module, perform linear projection processing and dot product calculation in sequence to obtain the attention score matrix;

[0049] S102, presetting an attention score matrix threshold, screening the attention score matrix using the attention score matrix threshold, and obtaining a selected attention score matrix;

[0050] S103, performing weighted calculation on the selected attention score matrix using an activation function to obtain the weighted global features in the main stage mixed feature module;

[0051] S104, performing local feature extraction on the image through convolution operation and activation function to obtain local features extracted in the main stage mixed feature module;

[0052] S105, sequentially performing concatenation and linear layer processing on the selected weighted global features in the main stage mixed feature module and the local features extracted in the main stage mixed feature module to obtain the output of the main stage mixed feature module;

[0053] S106 , repeating S101 to S105 with the template image and the search image as input, to obtain the output of the main stage mixed feature module of the template image and the search image.

[0054] The template image and search image are input into the main stage mixed feature module respectively, and linear projection processing and dot product calculation are performed in sequence to obtain the attention score matrix. The relationship between the corresponding process is as follows:

[0055] ;

[0056] in, represents the attention score matrix, Indicates the similarity between the query and the corresponding key, represents the query matrix, represents the bond matrix, represents transpose, Represents dimension;

[0057] It should be noted that The current input feature vector, is with The feature vector to be matched.

[0058] In the step of screening the attention score matrix using the attention score matrix threshold to obtain the selected attention score matrix, the corresponding process has the following relationship:

[0059] ;

[0060] in, Represents the attention score matrix Middle Rank The result after column selection, Indicates the set threshold;

[0061] In the step of performing weighted calculation on the selected attention score matrix using the activation function to obtain the weighted global features in the main stage mixed feature module, the corresponding process has the following relationship:

[0062] ;

[0063] in, Represents the global features after selection and weighting in the main stage mixed feature module, represents the first activation function, Represents the attention score matrix after setting the threshold selection, represents the value matrix;

[0064] In the step of extracting local features from the template image and the search image through convolution operations and activation functions to obtain the local features extracted in the main stage mixed feature module, the corresponding process has the following relationship:

[0065] ;

[0066] in, represents the first local feature extracted in the main stage hybrid feature module, represents the second local feature extracted in the main stage mixed feature module, Represents a convolution operation with a convolution kernel size of 3x3. represents the second activation function, Represents input features;

[0067] In the step of obtaining the output of the main-stage mixed feature module after sequentially concatenating and linearizing the selected weighted global features and the local features extracted from the main-stage mixed feature module, the corresponding process has the following relationship:

[0068] ;

[0069] in, represents the output of the main stage mixed feature module, represents a linear layer, Represents a splicing operation.

[0070] Furthermore, the main stage hybrid feature module extracts global features by selective attention, highlights important features and suppresses unimportant features, improves the model's perception of key information, reduces computational redundancy, and the fusion of global features and local features gives the model a richer feature representation. This type of fusion method retains local details while enhancing global contextual information.

[0071] Step 3: Input the output of the main stage hybrid feature module into the multi-scale fusion module, and combine the dynamic weights generated by the channel feature selection branch and the spatial feature selection branch to obtain the output of the multi-scale fusion module.

[0072] See also Figure 4 In step 3, the output of the main stage hybrid feature module is input into the multi-scale fusion module, and the dynamic weights generated by the channel feature selection branch and the spatial feature selection branch are combined to obtain the output of the multi-scale fusion module. The specific steps include the following:

[0073] The output of the main stage hybrid feature module is input into the multi-scale fusion module to extract features using depthwise separable convolution to obtain two extracted features of different scales.

[0074] Splicing the two extracted features of different scales to obtain the spliced ​​features;

[0075] The spliced ​​features are sent to the channel feature selection branch, and channel average pooling and channel maximum pooling are performed respectively to obtain channel average pooling features and channel maximum pooling features. The channel average pooling features and channel maximum pooling features are sequentially spliced ​​and convolved to obtain two channel attention features;

[0076] The two channel attention features are activated using the activation function to obtain two channel masks;

[0077] The concatenated features are sent to the spatial feature selection branch, and spatial global average pooling, fully connected layer processing and activation processing are performed in sequence to obtain two spatial masks;

[0078] Multiply the two channel masks and the two spatial masks to obtain two dynamically adjustable weights;

[0079] The two dynamically adjustable weights are respectively subjected to weighted fusion processing with the two extracted features of different scales to obtain a weighted feature map, which is then convolved to obtain the output of the multi-scale fusion module.

[0080] The output of the main stage mixed feature module is input into the multi-scale fusion module to extract features using depthwise separable convolution, and two extracted features of different scales are obtained. The relationship between the corresponding processes is as follows:

[0081] ;

[0082] in, Represents the first extracted features of different scales, Represents the second extracted features of different scales, Represents a convolution operation with a convolution kernel size of 1x1. represents a depth-wise separable convolution operation with a convolution kernel size of 5x5. represents a depth-wise separable convolution operation with a convolution kernel size of 3x3. Represents the features input from the main stage mixed feature module to the multi-scale fusion module;

[0083] In the step of splicing two extracted features of different scales to obtain the spliced ​​features, the corresponding process has the following relationship:

[0084] ;

[0085] in, Represents the features after splicing;

[0086] After the spliced ​​features are sent to the channel feature selection branch, channel average pooling processing and channel maximum pooling processing are performed respectively to obtain channel average pooling features and channel maximum pooling features. The channel average pooling features and channel maximum pooling features are sequentially spliced ​​and convolved to obtain two channel attention features. The relationship between the corresponding processes is as follows:

[0087] ;

[0088] in, Represents the channel attention features obtained after the convolution operation of the concatenated features, represents the channel average pooling operation, Represents the channel maximum pooling operation;

[0089] In the step of activating the two channel attention features using the activation function to obtain the two channel masks, the corresponding process has the following relationship:

[0090] ;

[0091] in, Indicates the channel masks, represents the channel mask index, represents the second activation function, Indicates the channel attention features, Represents the channel attention feature index.

[0092] The spliced ​​features are sent to the spatial feature selection branch, and spatial global average pooling processing, fully connected layer processing and activation processing are performed in sequence to obtain two spatial masks. The relationship between the corresponding processes is as follows:

[0093] ;

[0094] in, represents the spatial attention features generated after processing by the fully connected layer, represents the fully connected layer, Represents spatial global average pooling processing, Indicates the A spatial mask, represents the spatial mask index, Indicates the spatial attention features, represents the spatial attention feature index;

[0095] In the step of multiplying two channel masks and two spatial masks to obtain two dynamically adjustable weights, the corresponding process has the following relationship:

[0096] ;

[0097] in, Indicates the first dynamically adjustable weight, Indicates the second dynamically adjustable weight, Indicates the first channel mask, Indicates the second channel mask, Indicates the first spatial mask, Indicates the second spatial mask;

[0098] In the steps of performing weighted fusion processing on the two dynamically adjustable weights and the two extracted features of different scales to obtain a weighted feature map, performing convolution processing on the weighted feature map, and obtaining the output of the multi-scale fusion module, the corresponding process has the following relationship:

[0099] ;

[0100] in, Represents the output of the multi-scale fusion module.

[0101] Furthermore, the multi-scale fusion module uses depthwise separable convolution with convolution kernels of different sizes to obtain features of different scales and contextual features of different receptive fields. These features of different scales are then passed through a convolutional layer to integrate channel information and unify dimensions. The features of these different receptive fields are then concatenated and fed into a channel feature selection branch and a spatial feature selection branch to generate two dynamically adjustable weights. The channel feature selection branch processes the concatenated features through maximum pooling and average pooling, concatenating the concatenated features. These concatenated features are then passed through a convolutional layer to obtain two attention features. These two attention features are then processed through a sigmoid activation function to obtain two channel masks. The spatial feature selection branch processes the concatenated features of different receptive fields through global average pooling, a fully connected layer, and a softmax activation function to generate two spatial masks. The generated channel masks and spatial masks are multiplied to obtain two dynamically adjustable weights. The features processed by the different convolution kernels are then fused with the two weights to obtain a weighted feature map. The weighted feature map is then convolved to achieve feature fusion of different scales.

[0102] Step 4: Input the output of the main stage hybrid feature module and the output of the multi-scale fusion module into the secondary stage hybrid feature module for interactive fusion processing to obtain the output of the secondary stage hybrid feature module;

[0103] The classification loss is calculated based on the output of the hybrid feature module in the next stage, and the tracking model is optimized using the classification loss to obtain the optimized tracking model.

[0104] In step 4, the output of the main stage hybrid feature module and the output of the multi-scale fusion module are input into the secondary stage hybrid feature module for interactive fusion processing to obtain the output of the secondary stage hybrid feature module, which specifically includes the following steps:

[0105] The output of the main stage hybrid feature module and the output of the multi-scale fusion module are input into the secondary stage hybrid feature module to obtain image features, and the template image sequence features and the search image sequence features are obtained respectively;

[0106] The template image sequence features and the search image sequence features are spliced ​​together to obtain the features of the connected image sequence. The features of the connected image sequence are used as input to repeat steps S101 to S105 to perform selective attention calculations, thereby obtaining the weighted global features and the spliced ​​value matrix in the next-stage hybrid feature module.

[0107] The concatenated features of the image sequence and the spliced ​​value matrix are subjected to convolution, activation, and convolution respectively to obtain two local features extracted by the hybrid feature module in the next stage;

[0108] The global features selected and weighted in the second-stage mixed feature module and the two local features extracted by the second-stage mixed feature module are sequentially spliced ​​and processed by linear layers to obtain the output of the second-stage mixed feature module.

[0109] The template image sequence features and the search image sequence features are spliced ​​to obtain the features of the connected image sequence. The features of the connected image sequence are used as input to repeat steps S101 to S105 to perform selective attention calculation, and the weighted global features and the spliced ​​value matrix in the next stage hybrid feature module are obtained respectively. The relationship between the corresponding processes is as follows:

[0110] ;

[0111] in, represents the features of the image sequence after connection, represents the template image sequence features, Represents the search image sequence features, represents the concatenated query matrix, The query matrix of the template image, represents the query matrix for the search image, represents the concatenated bond matrix, represents the key matrix of the template image, represents the key matrix of the search image, represents the concatenated value matrix, A matrix of values ​​representing the template image, A matrix of values ​​representing the search image, represents the attention score matrix in the next stage mixed feature module, represents the similarity between the concatenated query matrix and the corresponding concatenated key matrix, Represents the attention score matrix in the next stage mixed feature module No. Rank The result after column selection, Represents the global features after selection and weighting in the next stage hybrid feature module, Represents the attention score matrix after threshold selection in the next stage hybrid feature module;

[0112] In the steps of performing convolution operations, activation processing, and convolution processing on the features after the connection of the image sequence and the spliced ​​value matrix, and obtaining two local features extracted by the mixed feature module in the next stage, the relationship between the corresponding processes is as follows:

[0113] ;

[0114] in, Represents the first local feature extracted by the hybrid feature module in the next stage, Represents the second local feature extracted by the hybrid feature module in the next stage;

[0115] In the step of sequentially performing splicing processing and linear layer processing on the global features selected and weighted in the secondary mixed feature module and the two local features extracted by the secondary mixed feature module to obtain the output of the secondary mixed feature module, the corresponding process has the following relationship:

[0116] ;

[0117] in, Represents the output of the next stage mixed feature module.

[0118] Furthermore, a secondary hybrid feature module is used to interactively fuse the template image features and search image features processed by the primary hybrid feature module and the multi-scale fusion module. In the secondary hybrid feature module, global and local features are extracted simultaneously from the input template image and search image. The extracted global and local features are then concatenated, enhancing global contextual information while preserving local details. The hybrid feature module extracts global features, highlighting important features and suppressing unimportant ones, improving the tracking model's ability to perceive key information and reducing computational redundancy. The fusion of global and local features provides the model with a richer feature representation.

[0119] Step 5: Use a large-scale data set to pre-train the optimized tracking model and adjust the model parameters to obtain a tracking model with adjusted parameters.

[0120] Furthermore, 60,000 images were randomly selected from four public datasets, TrackingNet, GOT-10k, LaSOT, and COCO2017, for training. The number of training rounds was 300. After 240 rounds of training, the learning rate was reduced to 10% of the initial value to enable the model to converge quickly.

[0121] Step 6: Input the template image and the search image into the tracking model after parameter adjustment, and iteratively repeat steps 2 to 4 until the preset number of iterations is reached to obtain the output of the final secondary hybrid feature module.

[0122] The output of the final sub-stage mixed feature module is input to the prediction head to obtain the tracking result.

[0123] Please refer to Figure 5 The present invention also proposes a target tracking system based on hybrid features and multi-scale fusion attention, the system comprising:

[0124] Building blocks for:

[0125] A hybrid feature module is built based on the Transformer network framework, and a multi-scale fusion module is built based on the depthwise separable convolution and dynamic attention weight fusion mechanism. The hybrid feature module and the multi-scale fusion module constitute the tracking model.

[0126] Among them, the hybrid feature module includes a main stage hybrid feature module and a secondary stage hybrid feature module;

[0127] Extraction module for:

[0128] Input the template image and the search image into the main stage mixed feature module, use the selective attention mechanism to extract the global features of the template image and the search image, use the convolution operation to extract the local features of the template image and the search image, and fuse the global features of the template image and the search image with the local features of the template image and the search image to obtain the output of the main stage mixed feature module;

[0129] Compute module for:

[0130] The output of the main stage hybrid feature module is input into the multi-scale fusion module, and the dynamic weights generated by the channel feature selection branch and the spatial feature selection branch are combined to obtain the output of the multi-scale fusion module;

[0131] Learning modules for:

[0132] The output of the main stage hybrid feature module and the output of the multi-scale fusion module are input into the secondary stage hybrid feature module for interactive fusion processing to obtain the output of the secondary stage hybrid feature module;

[0133] Calculate the classification loss based on the output of the hybrid feature module in the second stage, and use the classification loss to optimize the tracking model to obtain the optimized tracking model;

[0134] Pre-training modules for:

[0135] Pre-train the optimized tracking model using a large-scale dataset and adjust the model parameters to obtain a tracking model with adjusted parameters.

[0136] Tracking module for:

[0137] The template image and the search image are input into the tracking model after parameter adjustment. The extraction module, calculation module and learning module are repeated in an iterative manner as input. After reaching the preset number of iterations, the output of the final secondary stage hybrid feature module is obtained.

[0138] The output of the final sub-stage mixed feature module is input to the prediction head to obtain the tracking result.

[0139] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0140] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0141] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A target tracking method based on hybrid features and multi-scale fusion attention, characterized in that: The method comprises the following steps: Step 1: Build a hybrid feature module based on the Transformer network framework, and build a multi-scale fusion module based on the depthwise separable convolution and dynamic attention weight fusion mechanism. The hybrid feature module and the multi-scale fusion module constitute the tracking model. Among them, the hybrid feature module includes a main stage hybrid feature module and a secondary stage hybrid feature module; Step 2: Input the template image and the search image into the main stage mixed feature module, use the selective attention mechanism to extract the global features of the template image and the search image, use the convolution operation to extract the local features of the template image and the search image, and fuse the global features of the template image and the search image with the local features of the template image and the search image to obtain the output of the main stage mixed feature module; Step 3: Input the output of the main stage hybrid feature module into the multi-scale fusion module, and combine it with the dynamic weights generated by the channel feature selection branch and the spatial feature selection branch to obtain the output of the multi-scale fusion module; Step 4: Input the output of the main stage hybrid feature module and the output of the multi-scale fusion module into the secondary stage hybrid feature module for interactive fusion processing to obtain the output of the secondary stage hybrid feature module; Calculate the classification loss based on the output of the hybrid feature module in the second stage, and use the classification loss to optimize the tracking model to obtain the optimized tracking model; Step 5: Pre-train the optimized tracking model using a large-scale data set and adjust the model parameters to obtain a tracking model with adjusted parameters. Step 6: Input the template image and the search image into the tracking model after parameter adjustment, and iteratively repeat steps 2 to 4 until the preset number of iterations is reached to obtain the output of the final secondary hybrid feature module. The output of the final sub-stage mixed feature module is input to the prediction head to obtain the tracking result.

2. The target tracking method based on hybrid features and multi-scale fusion attention according to claim 1 is characterized in that: In step 2, the template image and the search image are input into the main stage mixed feature module, the global features of the template image and the search image are extracted using the selective attention mechanism, the local features of the template image and the search image are extracted using the convolution operation, and the global features of the template image and the search image are fused with the local features of the template image and the search image to obtain the output of the main stage mixed feature module, which specifically includes the following steps: S101: Input the image into the main stage mixed feature module, perform linear projection processing and dot product calculation in sequence to obtain the attention score matrix; S102, presetting an attention score matrix threshold, screening the attention score matrix using the attention score matrix threshold, and obtaining a selected attention score matrix; S103, performing weighted calculation on the selected attention score matrix using an activation function to obtain the weighted global features in the main stage mixed feature module; S104, performing local feature extraction on the image through convolution operation and activation function to obtain local features extracted in the main stage mixed feature module; S105, sequentially performing concatenation and linear layer processing on the selected weighted global features in the main stage mixed feature module and the local features extracted in the main stage mixed feature module to obtain the output of the main stage mixed feature module; S106 , repeating S101 to S105 with the template image and the search image as input, to obtain the output of the main stage mixed feature module of the template image and the search image.

3. The target tracking method based on hybrid features and multi-scale fusion attention according to claim 2 is characterized in that: The template image and search image are input into the main stage mixed feature module respectively, and linear projection processing and dot product calculation are performed in sequence to obtain the attention score matrix. The relationship between the corresponding process is as follows: ； in, represents the attention score matrix, Indicates the similarity between the query and the corresponding key, represents the query matrix, represents the bond matrix, represents transpose, Represents dimension; In the step of screening the attention score matrix using the attention score matrix threshold to obtain the selected attention score matrix, the corresponding process has the following relationship: ； in, Represents the attention score matrix Middle Rank The result after column selection, Indicates the set threshold; In the step of performing weighted calculation on the selected attention score matrix using the activation function to obtain the weighted global features in the main stage mixed feature module, the corresponding process has the following relationship: ； in, Represents the global features after selection and weighting in the main stage mixed feature module, represents the first activation function, Represents the attention score matrix after setting the threshold selection, represents the value matrix; In the step of extracting local features from the template image and the search image through convolution operations and activation functions to obtain the local features extracted in the main stage mixed feature module, the corresponding process has the following relationship: ； in, represents the first local feature extracted in the main stage hybrid feature module, represents the second local feature extracted in the main stage mixed feature module, Represents a convolution operation with a convolution kernel size of 3x3. represents the second activation function, Represents input features; In the step of obtaining the output of the main-stage mixed feature module after sequentially concatenating and linearizing the selected weighted global features and the local features extracted from the main-stage mixed feature module, the corresponding process has the following relationship: ； in, represents the output of the main stage mixed feature module, represents a linear layer, Represents a splicing operation.

4. The target tracking method based on hybrid features and multi-scale fusion attention according to claim 3 is characterized in that: In step 3, the output of the main stage hybrid feature module is input into the multi-scale fusion module, and the dynamic weights generated by the channel feature selection branch and the spatial feature selection branch are combined to obtain the output of the multi-scale fusion module, which specifically includes the following steps: The output of the main stage hybrid feature module is input into the multi-scale fusion module to extract features using depthwise separable convolution to obtain two extracted features of different scales. Splicing the two extracted features of different scales to obtain the spliced ​​features; The spliced ​​features are sent to the channel feature selection branch, and channel average pooling and channel maximum pooling are performed respectively to obtain channel average pooling features and channel maximum pooling features. The channel average pooling features and channel maximum pooling features are sequentially spliced ​​and convolved to obtain two channel attention features; The two channel attention features are activated using the activation function to obtain two channel masks; The concatenated features are sent to the spatial feature selection branch, and spatial global average pooling, fully connected layer processing and activation processing are performed in sequence to obtain two spatial masks; Multiply the two channel masks and the two spatial masks to obtain two dynamically adjustable weights; The two dynamically adjustable weights are respectively subjected to weighted fusion processing with the two extracted features of different scales to obtain a weighted feature map, which is then convolved to obtain the output of the multi-scale fusion module.

5. The target tracking method based on hybrid features and multi-scale fusion attention according to claim 4 is characterized in that: The output of the main stage mixed feature module is input into the multi-scale fusion module to extract features using depthwise separable convolution, and two extracted features of different scales are obtained. The relationship between the corresponding processes is as follows: ； in, Represents the first extracted features of different scales, Represents the second extracted features of different scales, Represents a convolution operation with a convolution kernel size of 1x1. represents a depth-wise separable convolution operation with a convolution kernel size of 5x5. represents a depth-wise separable convolution operation with a convolution kernel size of 3x3. Represents the features input from the main stage mixed feature module to the multi-scale fusion module; In the step of splicing two extracted features of different scales to obtain the spliced ​​features, the corresponding process has the following relationship: ； in, Represents the features after splicing; After the spliced ​​features are sent to the channel feature selection branch, channel average pooling processing and channel maximum pooling processing are performed respectively to obtain channel average pooling features and channel maximum pooling features. The channel average pooling features and channel maximum pooling features are sequentially spliced ​​and convolved to obtain two channel attention features. The relationship between the corresponding processes is as follows: ； in, Represents the channel attention features obtained after the convolution operation of the concatenated features, represents the channel average pooling operation, Represents the channel maximum pooling operation; In the step of activating the two channel attention features using the activation function to obtain the two channel masks, the corresponding process has the following relationship: ； in, Indicates the channel masks, represents the channel mask index, represents the second activation function, Indicates the channel attention features, Represents the channel attention feature index.

6. The target tracking method based on hybrid features and multi-scale fusion attention according to claim 5, characterized in that: The spliced ​​features are sent to the spatial feature selection branch, and spatial global average pooling processing, fully connected layer processing and activation processing are performed in sequence to obtain two spatial masks. The relationship between the corresponding processes is as follows: ； in, represents the spatial attention features generated after processing by the fully connected layer, represents the fully connected layer, Represents spatial global average pooling processing, Indicates the A spatial mask, represents the spatial mask index, Indicates the spatial attention features, represents the spatial attention feature index; In the step of multiplying two channel masks and two spatial masks to obtain two dynamically adjustable weights, the corresponding process has the following relationship: ； in, Indicates the first dynamically adjustable weight, Indicates the second dynamically adjustable weight, Indicates the first channel mask, Indicates the second channel mask, Indicates the first spatial mask, Indicates the second spatial mask; In the steps of performing weighted fusion processing on the two dynamically adjustable weights and the two extracted features of different scales to obtain a weighted feature map, performing convolution processing on the weighted feature map, and obtaining the output of the multi-scale fusion module, the corresponding process has the following relationship: ； in, Represents the output of the multi-scale fusion module.

7. The target tracking method based on hybrid features and multi-scale fusion attention according to claim 6, characterized in that: In step 4, the output of the main stage hybrid feature module and the output of the multi-scale fusion module are input into the secondary stage hybrid feature module for interactive fusion processing to obtain the output of the secondary stage hybrid feature module, which specifically includes the following steps: The output of the main stage hybrid feature module and the output of the multi-scale fusion module are input into the secondary stage hybrid feature module to obtain image features, and the template image sequence features and the search image sequence features are obtained respectively; The template image sequence features and the search image sequence features are spliced ​​together to obtain the features of the connected image sequence. The features of the connected image sequence are used as input to repeat steps S101 to S105 to perform selective attention calculations, thereby obtaining the weighted global features and the spliced ​​value matrix in the next-stage hybrid feature module. The concatenated features of the image sequence and the spliced ​​value matrix are subjected to convolution, activation, and convolution respectively to obtain two local features extracted by the hybrid feature module in the next stage; The global features selected and weighted in the second-stage mixed feature module and the two local features extracted by the second-stage mixed feature module are sequentially spliced ​​and processed by linear layers to obtain the output of the second-stage mixed feature module.

8. The target tracking method based on hybrid features and multi-scale fusion attention according to claim 7, characterized in that: The template image sequence features and the search image sequence features are spliced ​​to obtain the features of the connected image sequence. The features of the connected image sequence are used as input to repeat steps S101 to S105 to perform selective attention calculation, and the weighted global features and the spliced ​​value matrix in the next stage hybrid feature module are obtained respectively. The relationship between the corresponding processes is as follows: ； in, represents the features of the image sequence after connection, represents the template image sequence features, Represents the search image sequence features, represents the concatenated query matrix, The query matrix of the template image, represents the query matrix for the search image, represents the concatenated bond matrix, represents the key matrix of the template image, represents the key matrix of the search image, represents the concatenated value matrix, A matrix of values ​​representing the template image, A matrix of values ​​representing the search image, represents the attention score matrix in the next stage mixed feature module, represents the similarity between the concatenated query matrix and the corresponding concatenated key matrix, Represents the attention score matrix in the next stage mixed feature module No. Rank The result after column selection, Represents the global features after selection and weighting in the next stage hybrid feature module, Represents the attention score matrix after threshold selection in the next stage hybrid feature module; In the steps of performing convolution operations, activation processing, and convolution processing on the features after the connection of the image sequence and the spliced ​​value matrix, and obtaining two local features extracted by the mixed feature module in the next stage, the relationship between the corresponding processes is as follows: ； in, Represents the first local feature extracted by the hybrid feature module in the next stage, Represents the second local feature extracted by the hybrid feature module in the next stage; In the step of sequentially performing splicing processing and linear layer processing on the global features selected and weighted in the secondary mixed feature module and the two local features extracted by the secondary mixed feature module to obtain the output of the secondary mixed feature module, the corresponding process has the following relationship: ； in, Represents the output of the next stage mixed feature module.

9. A target tracking system based on hybrid features and multi-scale fusion attention, characterized in that: The system applies the target tracking method based on hybrid features and multi-scale fusion attention as described in any one of claims 1 to 8 above, and the system includes: Building blocks for: A hybrid feature module is built based on the Transformer network framework, and a multi-scale fusion module is built based on the depthwise separable convolution and dynamic attention weight fusion mechanism. The hybrid feature module and the multi-scale fusion module constitute the tracking model. Among them, the hybrid feature module includes a main stage hybrid feature module and a secondary stage hybrid feature module; Extraction module for: Input the template image and the search image into the main stage mixed feature module, use the selective attention mechanism to extract the global features of the template image and the search image, use the convolution operation to extract the local features of the template image and the search image, and fuse the global features of the template image and the search image with the local features of the template image and the search image to obtain the output of the main stage mixed feature module; Compute module for: The output of the main stage hybrid feature module is input into the multi-scale fusion module, and the dynamic weights generated by the channel feature selection branch and the spatial feature selection branch are combined to obtain the output of the multi-scale fusion module; Learning modules for: The output of the main stage hybrid feature module and the output of the multi-scale fusion module are input into the secondary stage hybrid feature module for interactive fusion processing to obtain the output of the secondary stage hybrid feature module; Calculate the classification loss based on the output of the hybrid feature module in the second stage, and use the classification loss to optimize the tracking model to obtain the optimized tracking model; Pre-training modules for: Pre-train the optimized tracking model using a large-scale dataset and adjust the model parameters to obtain a tracking model with adjusted parameters. Tracking module for: The template image and the search image are input into the tracking model after parameter adjustment. The extraction module, calculation module and learning module are repeated in an iterative manner as input. After reaching the preset number of iterations, the output of the final secondary stage hybrid feature module is obtained. The output of the final sub-stage mixed feature module is input to the prediction head to obtain the tracking result.

Images