A visual object tracking method based on multi-phase attention mechanism

Abstract

The present invention relates to the technical field of computer vision tracking, and in particular to a visual target tracking method based on a multi-phase attention mechanism, comprising the following steps: S1: normalizing the input images of the target template and the search area in the twin neural network; S2: inputting the image information into the feature extraction network, and outputting the last three layers of feature blocks of the extracted information; S3: hierarchically inputting the three layers of feature blocks into the dual-stage attention perception network; S4: uniformly adjusting the dimensions of the three feature blocks through multiple convolution operations of the adjustment layer; S5: hierarchically inputting the three feature blocks into the multi-layer context information decoder module; S6: merging and reducing the scale of the output three feature blocks through scale reduction operations respectively; S7: obtaining a response map; S8: obtaining a response map optimized by global perspective; S9: inputting the optimized response map into the classification regression prediction head to obtain a tracking result. The present invention can enhance the accuracy of tracking targets with different levels of features.

Description

Technical Field

[0001] The present invention relates to the field of computer vision tracking technology, and in particular to a visual target tracking method based on a multi-phase attention mechanism. Background Art

[0002] The target tracking task is a basic and challenging problem in the field of computer vision. Target tracking aims to determine the target's subsequent position and generate a predicted aiming frame given only the information of the target's initial position in the first frame. Initially, target tracking was a task that simulated human visual attention and cognitive mechanisms and gradually became an important task in the field of computer vision. Recently, benefiting from the rapid development of interdisciplinary subjects, target tracking has been widely used in intelligent video surveillance, drone applications, assisted autonomous driving and other fields.

[0003] The task of target tracking can be described as: by analyzing the video image sequence, matching the detected candidate target areas, and locating the coordinate positions of these targets in the image. The key links in the design of the tracking algorithm are divided into two parts: the design of the target appearance model and the design of the motion model. 1) The motion model can be established by the tracking strategy adopted by the algorithm. The tracking strategy adopted by the generative model tracking method such as particle filtering (based on probability, Bayesian) is usually: using Bayesian filtering theory to estimate the state of the target, that is, according to the prior knowledge and state equation of the target at the current moment, the state of the next moment is predicted and corrected in a recursive manner to achieve the estimation of the time-varying state of the target; 2) Establishing the corresponding appearance model through the appearance characteristics of the target, the target appearance modeling is divided into generative tracking and discriminative tracking. For generative tracking, the process is to use the established appearance model to directly match the template with the target category to achieve the purpose of tracking. For discriminative tracking, the algorithm uses a classification model to distinguish between the target and the background, and accurately determines the target position with the idea of ​​classification and regression.

[0004] Most of the existing traditional algorithms are based on generative models to achieve target tracking, which requires complex detection feature operators and uses manually designed features to perform template matching to complete the target tracking task. For example, the optical flow method for moving target tracking still divides the moving target tracking task into two steps. The first step is to calculate the optical flow information of two frames of images, and the second step is to select a suitable threshold to segment the optical flow map. It fails to integrate a suitable network framework, making it difficult to select the threshold. It is not applicable to aerial images with high platform freedom of movement and more complex movements, and there are problems such as slow speed, more missed detections, and inaccurate and incomplete moving target tracking.

[0005] In recent years, the optimal target tracking method based on twin neural networks has become a very efficient method to solve the tracking problem. While the performance of twin neural networks has been greatly improved, they have also achieved a balance between accuracy and real-time performance, and have become a hot research field based on deep learning methods. The core idea of ​​the twin neural network tracking method is to use two identical feature extraction network branches, which are used for feature extraction of target templates and search areas respectively, and transform the tracking problem into a similarity matching problem between the features of the two branches through the operation process of the cross-correlation layer network, and finally obtain the best matching search area through the subsequent classification regression network to complete the tracking. In recent years, the development trend of twin neural networks has shown that how to effectively utilize features at different levels is the key to improving the performance of the algorithm. One way is through linear multi-scale context fusion. For example, some works achieve feature fusion by directly summing or channel cascading the feature blocks extracted by the backbone extraction network, and other works enable the network to obtain richer global context information by designing efficient local encoding modeling layers or expanding the receptive field by decomposing the operation feature information structure.

[0006] Although the existing methods of utilizing features can achieve the use of context information to track targets of different scales, linear fusion or local modeling cannot fully utilize the feature information from a global perspective, and the pixel-level correlation between features at different levels is often ignored, which is also necessary for accurate tracking. At the same time, in order to achieve efficient and accurate tracking, we need to distinguish the two attributes of the target foreground and background and overcome various ambiguous interfering feature information. For example, in the scene where the target is partially occluded, the features of the car become insignificant and may only be detected as local features and become difficult to distinguish. In the scene of scale change, the multi-scale features of the car make successful tracking have high requirements for features at different levels. Therefore, it is necessary to design feature fusion and enhancement networks for features at different levels to improve the feature expression ability of pixel-level recognition. However, in the scenes of target occlusion, attitude scale change, small target tracking, etc. in aerial remote sensing scenes, the performance is poor, tracking drift is prone to occur, and the accuracy is low in challenging scenes. The reason is that the existing algorithms do not make perfect use of features.

[0007] Therefore, in order to solve the above problems and improve the performance of the existing twin neural network algorithm under the above challenging scenario characteristics, it is urgent to design a target tracking algorithm based on the twin neural network hierarchical attention strategy and multi-layer feature enhancement. Summary of the invention

[0008] In order to solve the above problems, the present invention provides a visual target tracking method based on a multi-phase attention mechanism, which can solve the problems that the existing methods are difficult to fully utilize the feature information under the global perspective and the pixel-level cross-correlation between features at different levels is ignored, and can improve the accuracy of visual target tracking and the success rate of tracking.

[0009] To achieve the above object, the present invention proposes the following technical solution: a visual target tracking method based on a multi-phase attention mechanism, comprising the following steps:

[0010] S1: Normalize the input images of the target template and search area branches in the twin neural network;

[0011] S2: Input the normalized image information into the feature extraction network and output the last three layers of feature blocks of the extracted information;

[0012] S3: The three-layer feature blocks obtained in S2 are hierarchically input into the two-stage attention perception network; the first input is a three-layer independent cascade decomposition encoder module structure;

[0013] S4: The three feature blocks output in S3 are uniformly adjusted in dimension through multiple convolution operations of the adjustment layer;

[0014] S5: The three feature blocks in S4 are hierarchically input into the second functional part of the two-stage attention perception network. The second functional part is a multi-layer context information decoder module. Both the target template and the search area branches include a multi-layer context information decoder module. The multi-layer context information decoder module contains four global feature modeling network modules.

[0015] S6: The three feature blocks of the target template output in S5 and the three feature blocks of the search area are respectively combined and scaled down by a scale reduction operation;

[0016] S7: Performing a deep separable cross-correlation operation on the target template obtained in S6 and the feature blocks of the two branches of the search area to obtain a response map;

[0017] S8: inputting the response map into the response map context information encoder module to obtain a global perspective optimized response map;

[0018] S9: Input the optimized response map into the classification regression prediction head to obtain the tracking result.

[0019] Furthermore, the three-layer feature blocks input in S3 are respectively used as the first functional area of ​​the two-stage attention perception, that is, the input of the three-level independent cascade decomposition encoder module, and the output is the optimized three-layer feature block of the same dimensional feature;

[0020] For the cascade decomposition encoder, the calculation process of the third feature block extracted by the backbone in the target template branch Z is taken as an example. Specifically, an input with 512 channels is decomposed into four 128-channel branches and one 512-input branch after four convolution operations and one pooling layer. For the first four branches, it is equivalent to dividing the input features into 4 subsets, and each subset channel has the same size, which are represented as ,in i Take a value from the set {1,2,3,4}, H, W and C Represents the shape of the input operation tensor data, H Indicates the number of channels, W Represents the height of each feature map, C represents the width of each feature map; the first subset of input features is passed to the 3 × 3 depth convolution, and the output is added to the next subset and used as the input of the next branch; the output of each branch is represented as F i , and then perform channel cascade, and add it with the output of the fifth branch as the final output of the module; in the fifth branch operation, the input passes through the average pooling layer to eliminate some noise interference, and finally passes through the Gamma function to continuously adjust during the training process and find the optimal fusion method of the network.

[0021] Furthermore, the calculation formula of the three feature blocks extracted by the backbone in the 6 cascade decomposition encoder parts of the target template branch Z and the search area branch X is:

[0022] ;

[0023] Get multi-scale feature map :

[0024] ;

[0025] in, x i It means that the input cascade decomposition encoder module is decomposed into 4 feature blocks with 128 channels after 1*1 convolution. F i It represents the feature block after the 3*3 convolution with a bias of 1. F i' Represents the feature block after cascading the first 4 output feature block channels to restore the number of input channels. F 5 represents the fifth output feature block, which is composed of the input feature block x(Z) The output results of the two branches after AvgPool average pooling and Gamma function are added together.

[0026] Furthermore, in S4, in the process of adjusting the channel dimensions of each feature block, a multi-layer convolutional layer dimension adjustment method is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading from the original [512, 1024, 2048] channels to [256, 256, 256] channels, and the dimension of the number of feature block channels is changed to 3*256.

[0027] Furthermore, in S5, a balance between deep and shallow features is achieved through the calculation of double-layer self-attention. Double-layer means that after the middle-layer feature block undergoes the optimization calculation of the first global feature modeling network module, its output is input into the second global feature modeling network module as the ternary input of the second global feature modeling network module. The final result of this path is the output of the second global feature modeling network module. The double-layer self-attention is specifically expressed as:

[0028]

[0029] in, T It represents the output result of the intermediate layer feature block after the optimization calculation of the first global feature modeling network module. L 4 represents the ternary input of the first global feature modeling network module. Represents the final output result after calculation by the second global feature modeling network module.

[0030] Furthermore, the global feature modeling network module in S5 uses an average pooling method at the input end of the attention calculation, uses the image's own encoding sequence information, and adopts a zero-filling method; the global feature modeling network module includes a multi-head attention module, a feedforward network, and a normalization layer; the calculation process of the global feature modeling network module is:

[0031]

[0032] in, Q Represents the query variable (Query) input in the ternary input of the global feature modeling network module; K Represents the key input in the key-value pair in the ternary input of the global feature modeling network module; V Represents the value input in the key-value pair in the ternary input of the global feature modeling network module; I It represents the input of the residual structure in the global feature modeling network module, which is an intermediate variable. Norm is the normalization layer, and the defined residual structure is Norm (I+MLP (I)). MLP is the fully connected layer, and MutiHead is the calculation process of the multi-head attention mechanism.

[0033] Furthermore, in S6, the number of channels and pixels of the feature map are reduced by multi-layer convolution; the feature block scales of the target template and search area branches are reduced to 7*7*256 and 31*31*256, respectively.

[0034] Furthermore, the response map in S7 is obtained in the following way: in the similarity matching network, by performing layered convolution on the corresponding layers of the three feature blocks of the target template and the search area, a depth-separable cross-correlation operation is adopted, the corresponding layer target template branch feature block is used as the convolution kernel, and the corresponding layer search area branch feature block is used as the convolution object, and the response map after each level of cross-correlation operation is calculated, which is specifically expressed by the following formula:

[0035]

[0036] in, represents the depthwise separable cross-correlation operation, X It represents the three-way feature block output by the two-stage attention perception network module in the search area branch after dimensionality reduction by the convolution layer. Z It represents the three-way feature block output by the two-stage attention perception network module in the target template branch after dimensionality reduction by the convolution layer. At this time, the number of channels of the output response map obtained is 256*3, and a 1*1 convolution layer is used to adjust the number of channels of the response map.

[0037] Furthermore, in S8, the optimized response graph is obtained in the following manner:

[0038] S81: The response graph feature information of the input module is first expanded (Unsqueeze) and reshaped (Reshape) to convert the feature expression into 4 dimensions through linear mapping;

[0039] S82: Through three 1*1*1 convolution processes, the input feature block is converted into three branches with half the number of channels. The three branches are θ Branch line, φ Branch line, g branch line;

[0040] S83: Flatten and transpose the feature block of the θ branch and multiply it with the flattened φ branch feature. Normalize the result through softmax to get the attention distribution. g After multiplying it on the branch feature block, the optimized feature R is obtained;

[0041] Steps S81-S83 are expressed from the perspective of theoretical modeling as follows:

[0042]

[0043] in,x represents the input feature map, i represents the spatiotemporal position index of the corresponding response feature, f Function Compute i and j The similarities between g Function calculation in j Characterization of the characteristic map of the position, the response factor C(x) Used to normalize the output to obtain the final output;

[0044] right R Features after transposition, flattening and dimension expansion through 1*1*1 convolutional layer Add them together and reset them to the same number of channels and size as the input branch operation features as the final output , specifically expressed as:

[0045]

[0046] Will The features of the initial input x Perform channel cascading, and then adjust the channel dimension to be consistent with the input channel dimension through a 1*1 convolution layer, and use it as the final output Y , specifically expressed as:

[0047] .

[0048] Furthermore, S9 adopts a classification and regression network based on the anchor-free mechanism, which directly classifies and regresses the candidate box at each pixel position through the classification and regression head network; and calculates the centrality score of the predicted box at each pixel position through the centrality head.

[0049] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0050] 1. First, the present invention proposes a new multi-phase perception attention twin network framework SiamMAN, which can enhance the network's ability to distinguish features for the task of target tracking and improve the accuracy of target tracking in scenarios with high requirements for features at different levels. The present invention uses the SiamMAN framework for target tracking tasks and introduces a multi-phase attention network that adapts to features at different depth levels, which can better obtain the correlation between features at different levels and improve the utilization of shallow spatial position features and deep semantic feature information.

[0051] 2. Secondly, the present invention proposes two attention networks adapted to different deep features, which can realize the dependency information between different features of the network from a global perspective. The rich context dependency information obtained by adaptive global modeling of features at different levels can significantly improve the tracking results of the network architecture;

[0052] The attention perception sub-network, the three feature blocks 3, 4, and 5 extracted by the backbone network are first passed through the proposed cascade decomposition encoder to break the limitation of the original network receptive field, and the channel correlation is established to achieve the update of each channel and the enhancement of feature significance. Then, the Transformer multi-layer enhanced decoder network with pooling operation is used to achieve the global modeling fusion enhancement of hierarchical features, so that the features include richer global context information;

[0053] The similarity matching sub-network designs a response graph enhancement network to enhance the cross-correlation dependency of deep features, better balance the semantic and spatial information of deep features, and improve the output effect of the classification and regression network.

[0054] 3. In summary, the present invention is a complete solution for tracking moving targets in aerial image sequences under complex scenes. The present invention is proposed to solve the problems that linear fusion or local modeling based on twin neural network trackers cannot fully utilize feature information from a global perspective and pixel-level cross-correlations between features at different levels are ignored in aerial video sequence target tracking, as well as tracking drift and low precision caused by target occlusion, scale and perspective changes, and complex backgrounds. The main difficulties of this task are:

[0055] 1) Noise interference: Noise interference is an unavoidable problem. Noise will affect both the imaging process and image transmission, and image quality will also be degraded due to lossy image compression. 2) Complex background: In air-to-ground scenes, the number of pixels in the target area is often low. At this time, the complex background is a difficult scene for tracking small target objects because it contains a lot of interference information. 3) Scale and perspective changes: In air-to-ground aerial image sequences, due to the movement of drones or airborne platforms, the images obtained often contain information about perspective changes and changes in distance to the ground, causing drastic scale changes and changes in observation angles of the target object, ultimately causing tracking drift and tracking failure. 4) Target occlusion: In air-to-ground motion tracking tasks, the target object area may be partially or completely occluded by objects in the background, causing the target tracking frame to drift or tracking failure. 5) Real-time tracking: Existing target tracking methods often have more model parameters while improving tracking performance. The resulting excessive amount of calculation makes it difficult to ensure the real-time performance of the tracking task. In the case of limited hardware resources, how to optimize the structural design and achieve better tracking performance with fewer parameters and computational complexity is a major difficulty in target tracking technology.

[0056] In the face of the above difficulties, the present invention designs a complete set of air-to-ground aerial image sequence target tracking technology, and proposes a new tracker SiamMAN based on twin neural networks to solve the problem that the linear fusion or local modeling adopted by the existing algorithm cannot fully utilize the feature information under the global perspective and the pixel-level correlation between features at different levels is ignored. Focus on the tracking drift and tracking failure problems in challenging aerial scenes, such as occlusion, aspect ratio changes, low resolution and scale changes. In this work, a new Siamese multi-phase attention tracker is proposed to achieve performance improvement of the algorithm in various challenging scenes, and a response graph context encoder is proposed to enable deep features to aggregate more contextual information and better balance deep semantic information. In addition, a two-stage perception attention module is proposed to aggregate more long-distance cross-channel spatiotemporal information to achieve global modeling, and enhance the tracker's ability to cope with complex scenes such as target occlusion and scale changes. In addition, the functional structure of the new multi-phase attention feature optimization is also efficiently integrated into the tracker architecture. It can greatly improve the accuracy and effect of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is an overall framework diagram of a visual target tracking model provided according to an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of the structure of a cascade decomposition encoder module provided according to an embodiment of the present invention;

[0059] Figure 3 is a schematic diagram of the structure of a response graph context information encoder module provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In the following, reference will be made to the Figure 1-3 Embodiments of the present invention are described. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following Figure 1-3 It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0062] A visual target tracking method based on a multi-phase attention mechanism includes the following steps:

[0063] S1: Normalize the input images of the target template and search area branches in the twin neural network; the normalization is performed by cropping or padding, with a unified resolution of 127*127 and 255*255.

[0064] S2: Input the normalized image information into the ResNet-50 feature extraction network and output the last three layers of feature blocks of the extracted information.

[0065] S3: The three-layer feature blocks obtained in S2 are hierarchically input into the two-stage attention perception network; the first input is a three-layer independent cascade decomposition encoder module structure, and the cascade decomposition encoder is CSE (CSE: Cascade SplitEncoder).

[0066] like Figure 1 As shown in the figure, the last three layers 3, 4, and 5 feature blocks output by the feature extraction backbone network are respectively used as the first functional area of ​​the dual-stage attention perception, that is, the input of the three-level independent cascade decomposition encoder (CSE) module, and the output is the optimized three-layer feature block of the same dimensional feature.

[0067] like Figure 2 As shown, for the cascade decomposition encoder (CSE), the calculation process of the third feature block extracted by the backbone in the target template branch Z is taken as an example. Specifically, an input with 512 channels is decomposed into four 128-channel branches and one 512-input branch after four convolution operations and one pooling layer. For the first four branches, it is equivalent to dividing the input features into 4 subsets, and each subset channel has the same size, which are represented as ,in i Take a value from the set {1,2,3,4}, H, W and C Represents the shape of the input operation tensor data, H Indicates the number of channels, W Represents the height of each feature map, C represents the width of each feature map; the first subset of input features is passed to the 3 × 3 depth convolution, and the output is added to the next subset and used as the input of the next branch; the output of each branch is represented as F i , and then perform channel cascade, and add it with the output of the fifth branch as the final output of the module; in the fifth branch operation, the input passes through the average pooling layer to eliminate some noise interference, and finally passes through the Gamma function to continuously adjust during the training process and find the optimal fusion method of the network to achieve better feature utilization.

[0068] The calculation formula of the three feature blocks extracted by backbone in the 6 cascade decomposition encoder parts of the target template branch Z and the search area branch X is:

[0069] ;

[0070] Get multi-scale feature map :

[0071] ;

[0072] in, x i It means that the input cascade decomposition encoder module is decomposed into 4 feature blocks with 128 channels after 1*1 convolution. F i It represents the feature block after the 3*3 convolution with a bias of 1. F i' Represents the feature block after cascading the first 4 output feature block channels to restore the number of input channels. F 5 represents the fifth output feature block, which is composed of the input feature block x(Z) The output results of the two branches after AvgPool average pooling and Gamma function are added together.

[0073] Compared with traditional convolution operations, the cascaded decomposition encoder can obtain rich long-range global context dependency information, break the limitation of the receptive field in the traditional CNN structure, and make full use of the global multi-scale features between features at different levels to enhance the recognition ability of the network.

[0074] S4: The three feature blocks output in S3 are uniformly adjusted to 256 channels through multiple convolution operations in the adjustment layer. In the process of adjusting the channel dimensions of each feature block in S4, the dimension adjustment method of multiple convolution layers is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading from the original [512, 1024, 2048] channels to [256, 256, 256] channels to reduce subsequent parameters and calculations. At the same time, the dimension of the number of feature block channels becomes 3*256.

[0075] S5: After dimensionality adjustment, the feature blocks of each layer are flattened into sequence information using convolution operations and used as the input of the second functional part of the subsequent two-stage attention-aware network module; the two-stage attention-aware network is TAN (TAN: Two-stage Attention-aware Network). The three feature blocks in S4 are hierarchically input into the second functional part of the two-stage attention-aware network, which is a multi-level contextual information decoder module, which is MCD (MCD: Multi-level Contextual Decoder); both the target template and the search area branches include a multi-level contextual information decoder module, which contains four global feature modeling network modules, and the global feature modeling network is GFM (GFM: Global Feature Modeling).

[0076] This embodiment designs a GFM network similar to the Transformer encoder, which globally models the global context information of the channel dimension and uses the self-attention mechanism to obtain the global dependency relationship between long-distance channels. The MCD modules of the target template and the search area each contain 4 proposed GFM modules. Specifically, the adjusted feature block L4 corresponding to the feature block of the fourth layer of the feature extraction backbone network is respectively used as the query variable Q input of the three-way GFM module. Through such an operation, the mutual attention mechanism is realized, the information exchange between different branches is realized, and the global dependency modeling of long-distance deep and shallow positions and semantic information is better realized. The key-value pair KV input of each path corresponds to the output features L3, L4 and L5 of the adjusted dimensions of each level. The result of the block with KV input as L4 is sent to another GFM module, and a better balance of deep and shallow features is achieved through the calculation of double-layer self-attention. Double-layer means that after the intermediate layer feature block corresponding to con4 undergoes the optimization calculation of the first GFM module, its output is input to the second GFM module as the Q, K, V ternary input of the second GFM module, and the final result of this path is the output of the second GFM module. The calculation of the double-layer self-attention is specifically expressed as:

[0077]

[0078] in, T It represents the output result of the intermediate layer feature block corresponding to con4 after the optimization calculation of the first GFM module; L 4 represents the Q, K, V triple input of the first GFM module, that is, the intermediate layer feature block corresponding to conv4, which is conv4_x 256 and conv4_z 256 in the overall diagram; Represents the final output result after calculation by the second GFM module.

[0079] For the proposed GFM module, specifically, different from the traditional Transformer encoder structure, we use the average pooling method at the input of the attention calculation as a preprocessing mechanism for the input data to optimize the K and V parameters. In order to further optimize and obtain a lighter Transformer-based structure for object tracking tasks, we replace the position encoding step in the traditional Transformer with the image's own encoded sequence information, and use a zero-padding strategy to ensure the integrity of the sequence information. The GFM module consists of a multi-head attention module, a feedforward network, and a normalization layer. Its core process is the process of processing the input ternary data, which is essentially the calculation process of the multi-head attention mechanism. The calculation process of the global feature modeling network module is:

[0080]

[0081] in, Q Represents the query variable (Query) input in the GFM module ternary input; K Represents the key input in the key-value pair in the GFM module ternary input; V Represents the value input in the key-value pair in the GFM module ternary input; I Represents the input of the residual structure in the GFM module, which is an intermediate variable in the calculation process. Norm is the normalization layer, the residual structure defined is Norm (I + MLP (I)), MLP is the fully connected layer, and MutiHead is the calculation process of the multi-head attention mechanism.

[0082] S6: The three feature blocks of the target template output in S5 and the three feature blocks of the search area are combined and scaled down by scale reduction operations. In S6, the number of channels and pixels of the feature map are reduced by multi-layer convolution; the scales of the feature blocks of the target template and the search area branches are reduced to 7*7*256 and 31*31*256 respectively.

[0083] S7: Perform a deep separable cross-correlation operation on the target template obtained in S6 and the feature blocks of the two branches of the search area to obtain a response map. The response map in S7 is obtained specifically in the following manner: In the similarity matching network, hierarchical convolution is performed on the corresponding layers of the three-layer feature blocks of the 3rd, 4th, and 5th layers of the corresponding target template and the search area after convolution dimension reduction of the TAN module output, and a deep separable cross-correlation operation is used. The corresponding layer target template branch feature block is used as the convolution kernel, and the corresponding layer search area branch feature block is used as the convolved object. The response map after each level of cross-correlation operation is calculated, thereby realizing the fusion of deep and shallow features in the output response map. It is specifically expressed by the following formula:

[0084]

[0085] in, represents the depthwise separable cross-correlation operation, X It represents the three-way feature block output by the TAN module in the search area branch after dimensionality reduction by the convolution layer. Z It represents the three-way feature block output by the TAN module in the target template branch after dimensionality reduction by the convolution layer. The number of channels of the output response map obtained at this time is 256*3. If it is directly used as the input of the subsequent classification and regression network, the amount of calculation is still too large. Therefore, a 1*1 convolution layer is used to adjust the number of channels of the response map, reduce the number of model parameters and increase the model derivation speed, so as to maximize the balance between model effect and calculation efficiency.

[0086] S8: After deep cross-correlation is performed to obtain the response map, in order to make full use of the deep response feature information and make it contain richer global dependency semantic information, a response map context information encoder RCE (RCE: Responsemap Context Encoder) is designed to break through the limitations of local modeling. Its structure is as follows Figure 3 The response map is input into the response map context information encoder module to obtain a global perspective optimized response map.

[0087] In S8, the optimized response graph is obtained in the following manner:

[0088] S81: The response graph feature information of the input module is first expanded (Unsqueeze) and reshaped (Reshape) to convert the feature expression into 4 dimensions through linear mapping; to adapt to the subsequent high-dimensional convolution pooling optimization process;

[0089] S82: Through three 1*1*1 convolution processes, the input feature block is converted into three branches with half the number of channels. The three branches are θ Branch line, φ Branch line, g branch line; θ Branch line, φ Branch line, g The branches are the three input features to be calculated next;

[0090] S83: Flatten and transpose the feature block of the θ branch and multiply it with the flattened φ branch feature. Normalize the result through softmax to get the attention distribution. g After multiplying it on the branch feature block, the optimized feature is obtained R .

[0091] Steps S81-S83 are expressed from the perspective of theoretical modeling as follows:

[0092]

[0093] in, x represents the input feature map, i represents the spatiotemporal position index of the corresponding response feature, f Function Compute i and j The similarities between g Function calculation in j Characterization of the characteristic map of the position, the response factor C(x) Used to normalize the output to get the final output.

[0094] right R Features after transposition, flattening and dimension expansion through 1*1*1 convolutional layer Add them together, and then reshape them to the same channel number and size as the input branch operation features as the final output , specifically expressed as:

[0095]

[0096] Will The features of the initial input x Perform channel cascading, and then adjust the channel dimension to be consistent with the input channel dimension through a 1*1 convolution layer, and use it as the final output Y , specifically expressed as:

[0097] .

[0098] Compared with continuously stacking convolution and RNN operators, non-local operations can quickly capture long-range dependencies by directly calculating the relationship between two spatiotemporal positions. High-dimensional global modeling of long-distance dependencies can effectively improve the feature expression ability of deep response graphs, achieve the effect of pixel-level deep and shallow information balance and semantic information optimization, and have higher computational efficiency.

[0099] S9: Input the optimized response map into the classification regression prediction head to obtain the tracking result.

[0100] In the prediction head part of S9, the response map generated after the cross-correlation operation can be regarded as the linear mapping of the input image of the original search area in the space under the target template as the weight. Each position in the response map can be mapped to the corresponding position in the original image space of the search area, and the key to the mapping lies in the size of the receptive field of the linear transformation. Different from the algorithm based on the region proposal network that uses a fixed multi-scale aiming frame for similarity matching to obtain multiple response maps, then maps its maximum response area back to the corresponding search area position to obtain the target prediction aiming frame center and outputs the prediction aiming frame by size regression. The present invention adopts a classification regression network based on an anchor-free frame mechanism, and directly classifies and regresses the candidate frame at each pixel position through a classification and regression head network, without setting prior information to initialize the anchor frame size, thereby breaking through the scale and receptive field limitations of the fixed aiming frame, and minimizing the limitations and interference of the set prior information on the classification regression network, avoiding the introduction of multi-layer hyperparameter information that is difficult to optimize and adjust, and ensuring the computational efficiency of the classification regression network. At the same time, the prediction frame center score of each pixel position is calculated by the center degree head to eliminate the interference of some abnormal pixels on the accuracy of the generated aiming frame.

[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual target tracking method based on a multi-phase attention mechanism, characterized in that: The steps include: S1: Normalize the input images of the target template and search area branches in the twin neural network; S2: Input the normalized image information into the feature extraction network and output the last three layers of feature blocks of the extracted information; S3: The three-layer feature blocks obtained in S2 are hierarchically input into the two-stage attention perception network; the first input is a three-layer independent cascade decomposition encoder module structure; The three-layer feature blocks input in S3 are respectively used as the first functional area of ​​the two-stage attention perception, that is, the input of the three-level independent cascade decomposition encoder module, and the output is the optimized three-layer feature block of the same dimensional feature; For the cascade decomposition encoder, the calculation process of the third feature block extracted by the backbone in the target template branch Z is taken as an example. Specifically, an input with 512 channels is decomposed into four 128-channel branches and one 512-input branch after four convolution operations and one pooling layer. For the first four branches, it is equivalent to dividing the input features into 4 subsets, and each subset channel has the same size, which are represented as ,in i Take a value from the set {1,2,3,4}, H, W and C Represents the shape of the input operation tensor data, H Indicates the number of channels, W Represents the height of each feature map, C represents the width of each feature map; the first subset of input features is passed to the 3 × 3 depth convolution, and the output is added to the next subset and used as the input of the next branch; the output of each branch is represented as F i , and then perform channel cascade, and add it with the output of the fifth branch as the final output of the module; in the fifth branch operation, the input passes through the average pooling layer to eliminate the interference of some noise, and finally passes through the Gamma function to continuously adjust during the training process and find the optimal fusion method of the network; The calculation formula of the three feature blocks extracted by backbone in the 6 cascade decomposition encoder parts of the target template branch Z and the search area branch X is: ； Get multi-scale feature map : ； in, x i It means that the input cascade decomposition encoder module is decomposed into 4 feature blocks with 128 channels after 1*1 convolution. F i It represents the feature block after the 3*3 convolution with a bias of 1. F i' Represents the feature block after cascading the first 4 output feature block channels to restore the number of input channels. F 5 represents the fifth output feature block, which is composed of the input feature block x(Z) The output of the two branches after AvgPool average pooling and Gamma function is added; S4: The three feature blocks output in S3 are uniformly adjusted in dimension through multiple convolution operations of the adjustment layer; S5: The three feature blocks in S4 are hierarchically input into the second functional part of the two-stage attention perception network. The second functional part is a multi-layer context information decoder module. Both the target template and the search area branches include a multi-layer context information decoder module. The multi-layer context information decoder module contains four global feature modeling network modules. S6: The three feature blocks of the target template output in S5 and the three feature blocks of the search area are respectively combined and scaled down by a scale reduction operation; S7: Performing a deep separable cross-correlation operation on the target template obtained in S6 and the feature blocks of the two branches of the search area to obtain a response map; S8: inputting the response map into the response map context information encoder module to obtain a global perspective optimized response map; S9: Input the optimized response map into the classification regression prediction head to obtain the tracking result.

2. The visual target tracking method based on the multi-phase attention mechanism according to claim 1, characterized in that: In S4, in the process of adjusting the channel dimensions of each feature block, a multi-layer convolutional layer dimension adjustment method is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading from the original [512, 1024, 2048] channels to [256, 256, 256] channels, and the dimension of the number of feature block channels is changed to 3*256.

3. The visual target tracking method based on the multi-phase attention mechanism according to claim 2, characterized in that: In S5, the balance between deep and shallow features is achieved through the calculation of double-layer self-attention. Double-layer means that after the middle-layer feature block undergoes the optimization calculation of the first global feature modeling network module, its output is input into the second global feature modeling network module as the ternary input of the second global feature modeling network module. The final result of this path is the output of the second global feature modeling network module. The double-layer self-attention calculation is specifically expressed as: in, T It represents the output result of the intermediate layer feature block after the optimization calculation of the first global feature modeling network module. L 4 represents the ternary input of the first global feature modeling network module. Represents the final output result after calculation by the second global feature modeling network module.

4. The visual target tracking method based on the multi-phase attention mechanism according to claim 3, characterized in that: The global feature modeling network module in S5 uses the average pooling method at the input end of the attention calculation, uses the image's own encoding sequence information, and adopts the zero-padding method; the global feature modeling network module includes a multi-head attention module, a feedforward network, and a normalization layer; the calculation process of the global feature modeling network module is: in, Q Represents the query variable (Query) input in the ternary input of the global feature modeling network module; K Represents the key input in the key-value pair in the ternary input of the global feature modeling network module; V Represents the value input in the key-value pair in the ternary input of the global feature modeling network module; I It represents the input of the residual structure in the global feature modeling network module, which is an intermediate variable. Norm is the normalization layer, and the defined residual structure is Norm (I+MLP (I)). MLP is the fully connected layer, and MutiHead is the calculation process of the multi-head attention mechanism.

5. The visual target tracking method based on the multi-phase attention mechanism according to claim 4, characterized in that: In S6, the number of channels and pixels of the feature map are reduced by multi-layer convolution; the feature block scales of the target template and search area branches are reduced to 7*7*256 and 31*31*256, respectively.

6. The visual target tracking method based on the multi-phase attention mechanism according to claim 5, characterized in that: S9 adopts a classification and regression network based on the anchor-free mechanism. The candidate box at each pixel position is directly classified and regressed through the classification and regression head network; the center score of the predicted box at each pixel position is calculated through the center head.

Images