Target tracking method and system based on channel separation attention module

Abstract

The present invention proposes a target tracking method and system based on a channel separation attention module, the method comprising: inputting a template image and a search image into a model to obtain the first half input data and the second half input data; obtaining a template image of a result feature map and a search image of the result feature map through the first half input data and the second half input data, and obtaining the final result feature map by adopting an iterative method; inputting the final result feature map into an online evaluation module to obtain a confidence score; training the model and using the trained model to track the target online. The present invention uses a channel separation attention module to fuse dynamic convolution and attention, so that the tracker has dynamic local and global feature extraction capabilities, so as to enhance the feature extraction and fusion capabilities of the tracker, so that the tracker can better cope with changes in the appearance of objects in complex scenes, thereby improving the performance of target tracking.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing technology, and in particular to a target tracking method and system based on a channel separation attention module. Background Art

[0002] Visual tracking is an important research direction in the field of computer vision and image processing. Its main task is to use the initial appearance of the target to accurately locate the target in subsequent video frames. This technology has a wide range of applications in many fields, such as visual positioning, autonomous driving, and smart cities. However, due to many challenges in the real world, such as partial occlusion, target movement out of view, background clutter, viewpoint changes, and scale changes, it is still very difficult to develop a robust tracker.

[0003] Currently, tracking models usually adopt a two-stream two-stage model architecture. In this approach, features from the template and the search region are extracted separately. However, this approach has certain disadvantages, mainly attributed to the high computational complexity of the traditional attention mechanism. In addition, local feature information is often ignored when extracting features with global context. Recently, single-stream architectures have emerged as a viable alternative. These architectures bring faster processing and enhanced feature fusion capabilities, achieving significant success in tracking performance. The reason behind their effectiveness lies in the model architecture's ability to establish an unhindered information flow between the template and the search region at an early stage, especially from the original image pair. This helps in extracting target-specific features and prevents the loss of discriminative information. Summary of the invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a target tracking method and system based on a channel separation attention module to solve the above technical problems.

[0005] The present invention proposes a target tracking method based on a channel separation attention module, the method comprising the following steps:

[0006] Step 1: Input the template image and the search image into the single-stage overall model, obtain the template token and the search token through the backbone feature extraction module and the feature enhancement module of the single-stage overall model, and then concatenate the template token and the search token to obtain the total input token;

[0007] Perform channel separation operation on the total input tokens to obtain the first half of the input data and the second half of the input data;

[0008] Step 2: Perform an adaptive convolution operation on the first half of the input data to obtain the first half of the result feature map, and perform an asymmetric attention operation on the second half of the input data to obtain the second half of the result feature map;

[0009] The first half of the result feature map and the second half of the result feature map are spliced ​​and then enhanced to obtain a result feature map;

[0010] The result feature map is divided into a template image of the result feature map and a search image of the result feature map, and the result feature map is input into the single-stage overall model. The steps to step 2 are repeated in an iterative manner until the specified number of times to obtain the final result feature map.

[0011] Step 3: Input the final result feature map into the online evaluation module to obtain the confidence score, input the final result feature map into the head corner module to obtain the prediction result, and determine whether the prediction result is used as the online template in the next stage of tracking process according to the confidence score;

[0012] Repeat steps 1 to 2 based on a large-scale dataset, and pre-train the single-stage overall model in combination with the online prediction loss function to obtain the trained single-stage overall model, and then use the trained single-stage overall model to track the target online.

[0013] The present invention also proposes a target tracking system based on a channel separation attention module, the system comprising:

[0014] Data augmentation module for:

[0015] Input the template image and the search image into the single-stage overall model, obtain the template token and the search token through the backbone feature extraction module and the feature enhancement module of the single-stage overall model, and then concatenate the template token and the search token to obtain the total input token;

[0016] Perform channel separation operation on the total input tokens to obtain the first half of the input data and the second half of the input data;

[0017] Feature map acquisition module, used for:

[0018] An adaptive convolution operation is performed on the first half of the input data to obtain the first half of the result feature map, and an asymmetric attention operation is performed on the second half of the input data to obtain the second half of the result feature map;

[0019] The first half of the result feature map and the second half of the result feature map are spliced ​​and then enhanced to obtain a result feature map;

[0020] The result feature map is divided into a template image of the result feature map and a search image of the result feature map, and the result feature map is input into a single-stage overall model, and the result feature map is obtained by repeating the iterative method for a specified number of times;

[0021] Pre-training and tracking modules for:

[0022] The final result feature map is input into the online evaluation module to obtain the confidence score, and the final result feature map is input into the head corner module to obtain the prediction result, and the confidence score is used to determine whether the prediction result is used as the online template in the next stage of tracking process;

[0023] Based on a large-scale dataset and combined with the online prediction loss function, the single-stage overall model is pre-trained to obtain the trained single-stage overall model, and then the trained single-stage overall model is used to track the target online.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention utilizes channel separation attention, which combines adaptive depth-separable convolution with asymmetric attention mechanism. The module can dynamically extract global and local information from input data to improve the tracker's ability to distinguish between targets and backgrounds.

[0026] 2. The present invention introduces an online evaluation module, through which the online template image can be corrected online according to the predicted score of the search image, which can cope with the changes in the appearance of the object in complex scenes, so that it can better handle the difficulties such as severe occlusion, scale changes and complex background in the tracking process, effectively capture time information and handle changes in object appearance, thereby improving the performance of target tracking.

[0027] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flow chart of the target tracking method based on the channel separation attention module proposed in the present invention;

[0029] Figure 2 This is a schematic diagram of the framework principle of the target tracking method based on the channel separation attention module proposed in the present invention;

[0030] Figure 3 This is a diagram of an adaptive convolution operation module of a target tracking method based on a channel separation attention module proposed in the present invention;

[0031] Figure 4 Schematic diagram of the asymmetric attention mechanism of the target tracking method based on the channel separation attention module proposed in the present invention;

[0032] Figure 5 Schematic diagram of the channel separation attention mechanism of the target tracking method based on the channel separation attention module proposed in the present invention;

[0033] Figure 6This is a module schematic diagram of the target tracking system based on the channel separation attention module proposed in the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0035] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0036] See also Figure 1 and Figure 2 The embodiment of the present invention proposes a target tracking method based on a channel separation attention module, the method comprising the following steps:

[0037] Step 1: Input the template image and the search image into the single-stage overall model, obtain the template token and the search token through the backbone feature extraction module and the feature enhancement module, and then concatenate the template token and the search token to obtain the total input token;

[0038] Perform channel separation operation on the total input tokens to obtain the first half of the input data and the second half of the input data;

[0039] In step 1, the template image and the search image are input into the single-stage overall model, and the template token and the search token are obtained through the backbone feature extraction module and the feature enhancement module of the single-stage overall model. Then, the template token and the search token are concatenated to obtain the total input token. The corresponding process has the following formula:

[0040] ;

[0041] in, Indicates that through the splicing function operation, represents the total input tokens, represents the initial template token, Both represent online template tokens. Represents a search token;

[0042] The total input tokens are channel-separated to obtain the first half of the input data and the second half of the input data. The corresponding process formula is:

[0043] ;

[0044] in, It means that it is processed by separation function. Represents the first half of the input data, Indicates the second half of the input data.

[0045] Step 2: Perform an adaptive convolution operation on the first half of the input data to obtain the first half of the result feature map, and perform an asymmetric attention operation on the second half of the input data to obtain the second half of the result feature map;

[0046] The first half of the result feature map and the second half of the result feature map are spliced ​​and then enhanced to obtain a result feature map;

[0047] The result feature map is divided into a template image of the result feature map and a search image of the result feature map, and the result feature map is input into the single-stage overall model. The steps to step 2 are repeated in an iterative manner until the specified number of times to obtain the final result feature map.

[0048] See also Figure 3 , Figure 4 and Figure 5 In step 2, an adaptive convolution operation is performed on the first half of the input data to obtain the first half of the result feature map. The specific operations are as follows:

[0049] The first half of the input data is used to perform adaptive pooling operations and convolution function operations in sequence to obtain the intermediate output result. The corresponding formula is:

[0050] ;

[0051] in, Indicates the intermediate output result, It means that through the convolution function operation, Represents an adaptive pooling operation;

[0052] The adaptive convolution kernel is obtained through the intermediate output result, and the formula for the corresponding process is:

[0053] ;

[0054] in, Represents temporary output for reshaping intermediate output results, Indicates that through the reshaping operation, represents the height of the convolution kernel, represents the width of the convolution kernel, represents the number of convolution kernel groups, Indicates the number of channels of the convolution kernel, represents the weight ratio of the convolution kernel, Indicates Group convolution kernel weight ratio, Represents the operation of calculating the weights of each dimension after the expanded dimension. represents the adaptive convolution kernel, Indicates learnable parameters;

[0055] The adaptive convolution kernel and the total input token are used to obtain the feature map of the first half of the result. The formula for the corresponding process is:

[0056] ;

[0057] in, Represents the feature map of the first half of the results, represents the parameter weight of the convolution kernel, Indicates the number of groups into which the convolution is divided, Indicates the number of divided convolution groups;

[0058] Perform an asymmetric attention operation on the second half of the input data to obtain the second half of the result feature map. The specific operations are as follows:

[0059] The corresponding query, key and value are generated by using template tokens and search tokens. The formula for the corresponding process is:

[0060] ;

[0061] in, A query representing a template token, A key representing a template token, Represents the value of a template token, a query representing a search token, a key representing the search token, represents the value of the search token, The convolution operation representing the query of the augmented token, The convolution operation representing the key of the augmented token, The convolution operation representing the value of the boost token, It represents the operation of reducing the channel dimension of the input data. represents the query for the overall token, a key representing the overall token, represents the value of the overall token, Represents a template token, Represents a search token;

[0062] Asymmetric attention calculation is performed on the query, key, and value corresponding to the template token and the search token to obtain the feature map of the second half of the result. The formula for the corresponding process is:

[0063] ;

[0064] in, represents the template token after self-attention calculation, represents the search token after cross-attention calculation, Represents the feature map of the second half of the result, represents the dimension of the key, represents the transpose operation of the matrix, Indicates the use Function to calculate the attention weight operation;

[0065] The result feature map of the first half is concatenated with the result feature map of the second half and then enhanced to obtain the result feature map. The formula for the corresponding process is as follows:

[0066] ;

[0067] in, represents the resulting feature map, Represents an operation enhanced by a function.

[0068] Step 3: Input the final result feature map into the online evaluation module to obtain the confidence score, input the final result feature map into the head corner module to obtain the prediction result, and determine whether the prediction result is used as the online template in the next stage of tracking process according to the confidence score;

[0069] Repeat steps 1 to 2 based on a large-scale data set, and pre-train the single-stage overall model in combination with the online prediction loss function to obtain a trained single-stage overall model, and then use the trained single-stage overall model to track the target online;

[0070] In step 3, the final result feature map is input into the online evaluation module to obtain the confidence score. The specific steps for obtaining the confidence score are as follows:

[0071] The query for searching the region of interest token is generated by using the score token. The corresponding process has the following relationship:

[0072] ;

[0073] in, A fractional token representing learning, represents the operation of query convolution on the learned score tokens, represents a query for searching for region of interest tokens;

[0074] The final result feature map is used to adaptively extract important regional features, and the corresponding keys and values ​​are generated through the important regional features. The relationship between the corresponding process is as follows:

[0075] ;

[0076] in, Represents the final result feature map, Keys representing important regional features, The values ​​representing important regional features, Indicates important regional features. represents a function for adaptively extracting important area features;

[0077] The attention weights are calculated using the keys and values ​​of the important regional features, and then the attention weights are sequentially operated through the multilayer perceptron and the Sigmoid activation function to obtain the confidence score. The corresponding relationship in the process is:

[0078] ;

[0079] in, represents the attention weight of the attention output, represents the activation function used to generate a score from 0 to 1, represents the confidence score, The transpose of the key representing the important regional features, Represents the operation through multiple perceptron functions;

[0080] The single-stage overall model is pre-trained in combination with the loss function of online prediction to obtain the trained single-stage overall model. The trained single-stage overall model is then used in combination with the loss function of the target object box to generate prediction results and track the target online. The loss function expression of online prediction is:

[0081] ;

[0082] in, represents the loss of online prediction, represents the true label;

[0083] In the process of using the trained single-stage overall model to generate prediction results and track the target online, the loss function of the target object frame is used to accurately predict the position of the target to be tracked in the image. The loss function expression of the target object frame is:

[0084] ;

[0085] in, Indicates the actual position of the target object frame, represents the position of the predicted target object box, represents the range loss, represents the GIOU box loss, express The weight value of range loss, represents the weight value of GIOU loss, Represents the loss of the target object box.

[0086] See also Figure 6 , an embodiment of the present invention further provides a target tracking system based on a channel separation attention module, the system comprising:

[0087] Data augmentation module for:

[0088] Input the template image and the search image into the single-stage overall model, obtain the template token and the search token through the backbone feature extraction module and the feature enhancement module of the single-stage overall model, and then concatenate the template token and the search token to obtain the total input token;

[0089] Perform channel separation operation on the total input tokens to obtain the first half of the input data and the second half of the input data;

[0090] Feature map acquisition module, used for:

[0091] An adaptive convolution operation is performed on the first half of the input data to obtain the first half of the result feature map, and an asymmetric attention operation is performed on the second half of the input data to obtain the second half of the result feature map;

[0092] The first half of the result feature map and the second half of the result feature map are spliced ​​and then enhanced to obtain a result feature map;

[0093] The result feature map is divided into a template image of the result feature map and a search image of the result feature map, and the result feature map is input into a single-stage overall model, and the result feature map is obtained by repeating the iterative method for a specified number of times;

[0094] Pre-training and tracking modules for:

[0095] The final result feature map is input into the online evaluation module to obtain the confidence score, and the final result feature map is input into the head corner module to obtain the prediction result, and the confidence score is used to determine whether the prediction result is used as the online template in the next stage of tracking process;

[0096] Based on a large-scale dataset and combined with the online prediction loss function, the single-stage overall model is pre-trained to obtain the trained single-stage overall model, and then the trained single-stage overall model is used to track the target online.

[0097] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0099] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A target tracking method based on a channel separation attention module, characterized in that: The method comprises the following steps: Step 1: Input the template image and the search image into the single-stage overall model, obtain the template token and the search token through the backbone feature extraction module and the feature enhancement module of the single-stage overall model, and then concatenate the template token and the search token to obtain the total input token; Perform channel separation operation on the total input tokens to obtain the first half of the input data and the second half of the input data; Step 2: Perform an adaptive convolution operation on the first half of the input data to obtain the first half of the result feature map, and perform an asymmetric attention operation on the second half of the input data to obtain the second half of the result feature map; The first half of the result feature map and the second half of the result feature map are spliced ​​and then enhanced to obtain a result feature map; The result feature map is divided into a template image of the result feature map and a search image of the result feature map, and the result feature map is input into the single-stage overall model. Steps 1 to 2 are repeated in an iterative manner until a specified number of times, and the final result feature map is obtained. Step 3: Input the final result feature map into the online evaluation module to obtain the confidence score, input the final result feature map into the head corner module to obtain the prediction result, and determine whether the prediction result is used as the online template in the next stage of tracking process according to the confidence score; Repeat steps 1 to 2 based on a large-scale data set, and pre-train the single-stage overall model in combination with the online prediction loss function to obtain a trained single-stage overall model, and then use the trained single-stage overall model to track the target online; In step 2, an adaptive convolution operation is performed on the first half of the input data to obtain the first half of the result feature map. The specific operation is as follows: The first half of the input data is used to perform adaptive pooling operations and convolution function operations in sequence to obtain the intermediate output result. The corresponding formula is: ； in, Indicates the intermediate output result, It means that through the convolution function operation, represents the adaptive pooling operation, Represents the first half of the input data; The adaptive convolution kernel is obtained through the intermediate output result, and the formula for the corresponding process is: ； in, Represents temporary output for reshaping intermediate output results, Indicates that through the reshaping operation, represents the height of the convolution kernel, represents the width of the convolution kernel, represents the number of convolution kernel groups, Indicates the number of channels of the convolution kernel, represents the weight ratio of the convolution kernel, Indicates Group convolution kernel weight ratio, Represents the operation of calculating the weights of each dimension after the expanded dimension. represents the adaptive convolution kernel, Indicates learnable parameters; The adaptive convolution kernel and the total input token are used to obtain the feature map of the first half of the result. The formula for the corresponding process is: ； in, Represents the feature map of the first half of the results, represents the parameter weight of the convolution kernel, Indicates the number of groups into which the convolution is divided, Represents the number of divided convolution groups, represents the total input tokens; Perform an asymmetric attention operation on the second half of the input data to obtain the second half of the result feature map. The specific operations are as follows: The corresponding query, key and value are generated by using template tokens and search tokens. The formula for the corresponding process is: ； in, A query representing a template token, A key representing a template token, Represents the value of a template token, a query representing a search token, a key representing the search token, represents the value of the search token, The convolution operation representing the query of the augmented token, The convolution operation representing the key of the augmented token, The convolution operation representing the value of the boost token, It represents the operation of reducing the channel dimension of the input data. represents the query of the overall token, a key representing the overall token, represents the value of the overall token, Represents a template token, Represents a search token; Asymmetric attention calculation is performed on the query, key, and value corresponding to the template token and the search token to obtain the feature map of the second half of the result. The formula for the corresponding process is: ； in, represents the template token after self-attention calculation, represents the search token after cross-attention calculation, Represents the feature map of the second half of the result, represents the dimension of the key, represents the transpose operation of the matrix, Indicates the use Function to calculate the attention weight operation; The result feature map of the first half is concatenated with the result feature map of the second half and then enhanced to obtain the result feature map. The formula for the corresponding process is as follows: ； in, represents the resulting feature map, Represents an operation enhanced by a function.

2. The target tracking method based on the channel separation attention module according to claim 1 is characterized in that: In step 1, the template image and the search image are input into the single-stage overall model, and the template token and the search token are obtained through the backbone feature extraction module and the feature enhancement module of the single-stage overall model, and then the template token and the search token are spliced ​​to obtain the total input token. The corresponding process has the following formula: ； in, Indicates that through the splicing function operation, represents the initial template token, Both represent online template tokens. Represents a search token.

3. The target tracking method based on the channel separation attention module according to claim 2 is characterized in that: In step 1, the total input tokens are subjected to channel separation operation to obtain the first half of the input data and the second half of the input data. The corresponding process has the following formula: ； in, It means that it is processed by separation function. Indicates the second half of the input data.

4. The target tracking method based on the channel separation attention module according to claim 1 is characterized in that: In step 3, the final result feature map is input into the online evaluation module to obtain a confidence score. The specific steps for obtaining the confidence score are as follows: The query for searching the region of interest token is generated by using the score token. The corresponding process has the following relationship: ； in, A fractional token representing learning, represents the operation of query convolution on the learned score tokens, represents a query for searching for region of interest tokens; The final result feature map is used to adaptively extract important regional features, and the corresponding keys and values ​​are generated through the important regional features. The relationship between the corresponding process is as follows: ； in, Represents the final result feature map, Keys representing important regional features, The values ​​representing important regional features, Indicates important regional features. represents a function for adaptively extracting important area features; The attention weights are calculated using the keys and values ​​of the important regional features, and then the attention weights are sequentially operated through the multilayer perceptron and the Sigmoid activation function to obtain the confidence score. The corresponding relationship in the process is: ； in, represents the attention weight of the attention output, represents the activation function used to generate a score from 0 to 1, represents the confidence score, The transpose of the key representing the important regional features, Represents the operation through multiple perceptron functions.

5. The target tracking method based on the channel separation attention module according to claim 4 is characterized in that: In step 3, the single-stage overall model is pre-trained in combination with the loss function of online prediction to obtain the trained single-stage overall model, and then the trained single-stage overall model is used to track the target online, wherein the loss function expression of online prediction is: ； in, represents the loss of online prediction, represents the true label; In the process of using the trained single-stage overall model to generate prediction results and track the target online, the loss function of the target object frame is used to accurately predict the position of the target to be tracked in the image. The loss function expression of the target object frame is: ； in, Indicates the actual position of the target object frame, represents the position of the predicted target object box, represents the range loss, represents the GIOU box loss, express The weight value of range loss, represents the weight value of GIOU loss, Represents the loss of the target object box.

6. The target tracking system based on the channel separation attention module is characterized by: The system applies the target tracking method based on the channel separation attention module as described in any one of claims 1 to 5 above, and the system comprises: Data augmentation module for: Input the template image and the search image into the single-stage overall model, obtain the template token and the search token through the backbone feature extraction module and the feature enhancement module of the single-stage overall model, and then concatenate the template token and the search token to obtain the total input token; Perform channel separation operation on the total input tokens to obtain the first half of the input data and the second half of the input data; Feature map acquisition module, used for: An adaptive convolution operation is performed on the first half of the input data to obtain the first half of the result feature map, and an asymmetric attention operation is performed on the second half of the input data to obtain the second half of the result feature map; The first half of the result feature map and the second half of the result feature map are spliced ​​and then enhanced to obtain a result feature map; The result feature map is divided into a template image of the result feature map and a search image of the result feature map, and the result feature map is input into a single-stage overall model, and the result feature map is obtained by repeating the iterative method for a specified number of times; Pre-training and tracking modules for: The final result feature map is input into the online evaluation module to obtain the confidence score, and the final result feature map is input into the head corner module to obtain the prediction result, and the confidence score is used to determine whether the prediction result is used as the online template in the next stage of tracking process; Based on a large-scale dataset and combined with the online prediction loss function, the single-stage overall model is pre-trained to obtain the trained single-stage overall model, and then the trained single-stage overall model is used to track the target online.

Images