Image target tracking method, device and apparatus based on hierarchical attention strategy

Abstract

The present invention relates to the field of computer vision, and specifically provides an image target tracking method, device, and equipment based on a hierarchical attention strategy. By introducing the HAS module composed of a channel global context attention mechanism and a Transformer multi-layer feature enhancement mechanism, the algorithm model can be applied to the advantages of deep semantic features and shallow spatial features from as many feature levels as possible, breaking through the limitations of the receptive field of traditional convolutional neural networks and achieving multi-layer feature enhancement. At the same time, it also solves the problem that other trackers based on twin neural networks do not fully utilize deep and shallow feature information at different levels, and makes balanced use of deep and shallow feature information. While effectively using shallow information to help determine the position of the target aiming frame, it enhances deep semantic information to help the network better complete the judgment of the foreground and background binary attributes of each pixel position. While achieving real-time detection, it greatly improves the progress and success rate of target tracking.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image target tracking method, device and equipment based on a hierarchical attention strategy. Background Art

[0002] As an important task in the field of computer vision, target tracking technology can be cross-disciplinary and has become a hot topic for researchers in many related fields. It is widely used in intelligent video surveillance, drone applications, assisted autonomous driving and other fields. The target tracking method based on twin neural networks pays attention to the background information of the target motion rather than being limited to the characteristics of the target itself, and has achieved good tracking results. However, it performs poorly in scenarios such as target occlusion, attitude scale changes, and small target tracking in aerial remote sensing scenes. It is prone to tracking drift and has low accuracy in challenging scenarios. The reason is that the existing algorithms do not make perfect use of features. The main difficulties of target tracking tasks are:

[0003] 1. Noise interference: Noise interference is an unavoidable problem. Noise will affect both the imaging process and the image transmission. In addition, lossy compression of the image will lead to a decrease in image quality.

[0004] 2. Complex background: In air-to-ground scenes, the number of pixels in the target area is often low. At this time, the complex background is a difficult scene for the task of tracking small target objects because it contains a lot of interference information;

[0005] 3. Scale and perspective changes: In air-to-ground aerial image sequences, due to the movement of drones or airborne platforms, the images obtained often contain information about perspective changes and changes in distance to the ground, causing drastic scale changes and changes in observation angles of target objects, ultimately causing tracking drift and tracking failure.

[0006] 4. Target occlusion: In air-to-ground motion tracking tasks, the target object area may be partially or completely occluded by objects in the background, causing the target tracking frame to drift or tracking failure;

[0007] 5. Real-time tracking: Existing target tracking methods often have more model parameters while improving tracking performance. The resulting excessive amount of calculation makes it difficult to ensure the real-time performance of the tracking task. With limited hardware resources, how to optimize the structural design and achieve better tracking performance with fewer parameters and calculations is a major difficulty in target tracking technology.

[0008] The task of target tracking can be described as: by analyzing the video image sequence, matching the detected candidate target areas, and locating the coordinate positions of these targets in the image. The key links in the design of the tracking algorithm are divided into two parts: the design of the target appearance model and the design of the motion model. The motion model can be established by the tracking strategy adopted by the algorithm. The tracking strategy adopted by the generative model tracking method such as particle filtering (based on probability, Bayesian) is: using Bayesian filtering theory to estimate the state of the target, that is, according to the prior knowledge and state equation of the target at the current moment, the state of the next moment is predicted and corrected in a recursive manner to achieve the estimation of the time-varying state of the target. The corresponding appearance model is established through the appearance characteristics of the target, and the target appearance modeling is divided into generative tracking and discriminative tracking. For generative tracking, the process is to use the established appearance model to directly match the template with the target category to achieve the purpose of tracking. For discriminative tracking, the algorithm distinguishes the target and the background through the classification model, and accurately determines the target position with the idea of ​​classification and regression.

[0009] Most of the existing traditional algorithms are based on generative models to achieve target tracking, which requires complex detection feature operators and uses manually designed features to perform template matching to complete the target tracking task. For example, the optical flow method for moving target tracking still divides the moving target tracking task into two steps. The first step is to calculate the optical flow information of the two frames of images, and the second step is to select a suitable threshold to segment the optical flow map. The appropriate network framework has not been integrated, resulting in difficulty in selecting the threshold. It is not applicable to aerial images with high platform freedom of movement and more complex movements, and there are problems such as slow speed, many missed detections, and inaccurate and incomplete moving target tracking. The analysis of the existing mainstream moving target tracking network based on deep learning shows the following defects. First, the current network mainly achieves target tracking by similarity matching based on the structure of the twin neural network, but does not take advantage of the advantages of the deep feature extraction network. Simply using a deeper feature extraction network cannot achieve better results in the target tracking task. While adopting deeper features, the advantages of comprehensive utilization of deep and shallow features are not balanced. Shallow features contain more spatial location information which is very important for determining the target location, while deep features contain more semantic information to help the network distinguish between foreground and background. The tracking accuracy and success rate of existing trackers still have a lot of room for improvement due to the failure to fully balance the utilization of features at different levels.

[0010] Among the existing solutions, for example, SiamCAR changes the process of regression network calculation into solving the distance between the point on the search area branch of the feature map and the four sides of the selected truth box. The position of the optimal target center point is determined by calculating the classification score map and the centrality score map. Then, based on the two data of the distance between the optimal target center point and the four sides of the truth box, the position of the obtained prediction box is finally determined to achieve tracking. It is an anchor-free strategy tracker that eliminates the influence of anchor parameters and makes the network faster. However, its feature fusion network does not deeply explore the advantages of deep and shallow features at different levels. For scenes such as target occlusion and scale changes that are common in aerial video sequences, tracking drift and low accuracy are prone to occur. There is still a lot of room for improvement in the feature processing strategy before fusion. Summary of the invention

[0011] In view of this, an embodiment of the present invention provides an image target tracking method, device, and equipment based on a hierarchical attention strategy, which effectively utilizes shallow information to help determine the target aiming frame position while enhancing deep semantic information to help the network better complete the judgment of the foreground and background binary attributes of each pixel position, thereby achieving real-time detection and greatly improving the target tracking progress and success rate.

[0012] In a first aspect, an embodiment of the present invention provides an image target tracking method based on a hierarchical attention strategy, comprising:

[0013] Normalizing the target template of the twin neural network and the input images of the two branches of the search area to obtain a normalized image;

[0014] Input the normalized image into the residual network for feature extraction to obtain feature blocks of the last three layers;

[0015] Inputting the feature blocks of the last three layers into a hierarchical attention module for channel adjustment to obtain three feature blocks with a target number of channels, wherein the hierarchical attention module has three independent channel context attention modules;

[0016] The three feature blocks are hierarchically input into the Transformer multi-layer feature enhancement mechanism module to obtain the target template and the three feature blocks corresponding to the search area branches, respectively merged and scaled down to obtain the target scale feature block;

[0017] Performing a depth-separable cross-correlation operation on the target scale feature block to obtain a response map;

[0018] The response map is input into the classification regression prediction head to obtain the target tracking result.

[0019] As an optional solution, the hierarchical attention module adopts a typical correlation analysis CCA module, and the three feature blocks with the target number of channels obtained by hierarchically inputting the feature blocks of the last three layers into the hierarchical attention module for channel adjustment include:

[0020] The feature blocks of the last three layers are hierarchically input into the CCA module. The CCA module reshapes the feature map A of the input feature block to obtain three new maps with dimensions of CxN, and transposes any of the three new maps to obtain A T , for the obtained A T Perform matrix multiplication with another reshaped feature map to obtain an attention map X with a dimension of C×C. The attention map X is passed through a softmax layer to obtain an attention score map.

[0021] By X T The process of multiplying the matrix with another feature map A of dimension CxN is to apply the attention scoring process to different channels according to different attention scores, and restore the result to the original dimension to obtain the attention optimized feature map O. The attention map involved in the calculation is expressed as (1):

[0022]

[0023] The output attention optimized feature map O and the feature map A are fused together through channel cascading, and the final output feature map E is obtained through a 1*1 convolution process. The process is expressed in equation (2):

[0024] express:

[0025] E = conv(concat(A,O)) (6)

[0026] The CCA module is used to globally model the global context information of the channel dimension, capture the channel dependency between any two channel maps using the self-attention mechanism, and update each channel map using the weighted sum of all channels, thereby completing the optimization of the channel dimension of the source input data;

[0027] In the process of adjusting the channel dimensions of each feature block, a multi-layer convolutional layer dimension adjustment strategy is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading to the target channel.

[0028] As an optional solution, the step of hierarchically inputting the three feature blocks into the output of the Transformer multi-layer feature enhancement mechanism module to obtain the target template and the three feature blocks corresponding to the search area branches are respectively merged and scaled down to obtain the target scale feature block, including:

[0029] After dimension adjustment, the feature blocks of each layer are flattened into sequence information using convolution operations and used as input to the subsequent Transformer multi-level feature enhancement module, wherein the multi-level feature enhancement module includes four TEP modules. The three layers of sequence information in the structure of the multi-level feature enhancement module are respectively used as QKV triplet inputs of the three TEP modules. The result of the TEP block with conv4 as input is sent to another TEP module, and the balance between deep features and shallow features is performed through the calculation of a double-layer self-attention mechanism.

[0030] Multi-head attention is used to make the model focus on information from different representation subspaces at different positions. The calculation process is expressed as formula (3):

[0031]

[0032] In the above formula (3), Q, K, and V represent the query variable, the key in the initial key-value pair, and the value in the initial key-value pair, d k Represents the number of channels, calculates the similarity between Q and K, multiplies V by the normalized attention distribution weight, and realizes the feature enhancement of V. The specific calculation process includes: initializing multiple sets of coefficient weight matrices, calculating the input of each set of multi-head attention, calculating the attention map of each group according to the self-attention calculation process, cascading on the channel dimension and mapping it to the original space through linear transformation, and the obtained attention output has the same dimension as the original input;

[0033] The data is refined through multiple convolutional layers to reduce the number of channels and pixels of the feature map.

[0034] As an optional solution, performing a depth-separable cross-correlation operation on the target scale feature block to obtain a response map includes:

[0035] By performing hierarchical convolution on the corresponding layers of the target template and the search area, a depth-separable cross-correlation operation is used to calculate the response map after each level of cross-correlation operation, and a fusion of deep features and shallow features in the output response map is output. The process is described as (4):

[0036]

[0037] in, Represents the depthwise separable cross-correlation operation.

[0038] As an optional solution, the step of inputting the response map into a classification regression prediction head to obtain a target tracking result includes:

[0039] A classification and regression network based on an anchor-free mechanism is used to directly classify and regress the candidate boxes at each position in the response map to obtain the target tracking result. The response map represents the linear mapping of the input image in the search area in space with the target template as the weight, and each position in the response map is mapped to the corresponding position in the original image space of the search area.

[0040] As an optional solution, the uniform resolution of the normalized image is 127*127 and 255*255, the target number of channels is 256 channels, and the scale of the target scale feature block is 7*7*256 and 31*31*256.

[0041] As an optional solution, in the process of adjusting the channel dimensions of each feature block, a multi-layer convolutional layer dimension adjustment strategy is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading to the target channel, including:

[0042] By adopting the dimension adjustment strategy of multi-layer convolutional layers, the number of channels of the three-layer feature blocks after channel cascading is uniformly adjusted from the original [512, 1024, 2048] channels to [256, 256, 256] channels, and the dimension of the number of channels of the feature block will become 3*256.

[0043] In a second aspect, an embodiment of the present invention provides an image target tracking device based on a hierarchical attention strategy, comprising:

[0044] A normalization module unit is used to normalize the input images of the target template and the two branches of the search area of ​​the twin neural network to obtain a normalized image;

[0045] A feature extraction unit, used for inputting the normalized image into a residual network for feature extraction to obtain feature blocks of the last three layers;

[0046] A channel adjustment unit, used for hierarchically inputting the feature blocks of the last three layers into a hierarchical attention module for channel adjustment to obtain three feature blocks with a target number of channels, wherein the hierarchical attention module has three independent channel context attention modules;

[0047] A scale adjustment unit is used to hierarchically input the three feature blocks into the output of the Transformer multi-layer feature enhancement mechanism module to obtain the three feature blocks corresponding to the two branches of the target template and the search area, and respectively merge and reduce the scale to obtain a target scale feature block;

[0048] A similarity matching network unit is used to perform a deep separable cross-correlation operation on the target scale feature block to obtain a response map;

[0049] The prediction unit is used to input the response map into the classification regression prediction head to obtain the target tracking result.

[0050] In a third aspect, an embodiment of the present invention provides a computer device, including:

[0051] at least one processor; and

[0052] a memory communicatively connected to the at least one processor; wherein,

[0053] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned image target tracking method based on the hierarchical attention strategy.

[0054] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above-mentioned image target tracking method based on the hierarchical attention strategy.

[0055] In the embodiments of the present invention, an image target tracking method, device, and apparatus based on a hierarchical attention strategy are proposed. By introducing the HAS module composed of the channel global context attention mechanism and the Transformer multi-layer feature enhancement mechanism, the algorithm model can be applied to the advantages of deep semantic features and shallow spatial features from as many feature levels as possible, breaking through the limitations of the receptive field of traditional convolutional neural networks and achieving multi-layer feature enhancement. At the same time, it also solves the problem that other trackers based on twin neural networks do not fully utilize the deep and shallow feature information at different levels, and makes balanced use of the deep and shallow feature information. While effectively using the shallow information to help determine the target aiming frame position, the deep semantic information is enhanced to help the network better complete the judgment of the foreground and background binary attributes of each pixel position. While achieving real-time detection, it greatly improves the target tracking progress and success rate, providing a feasible method for target tracking in aerial video sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of an image target tracking method based on a hierarchical attention strategy is provided in an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the overall framework of a model in an image target tracking method based on a hierarchical attention strategy is provided in an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the structure of a channel context attention mechanism module in an image target tracking method based on a hierarchical attention strategy is provided in an embodiment of the present invention;

[0059] Figure 4A schematic diagram of the structure of a TEP module in an image target tracking method based on a hierarchical attention strategy is provided in an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of a multi-layer feature enhancement module, a deep cross-correlation operation, and a scale reduction operation in an image target tracking method based on a hierarchical attention strategy is provided in an embodiment of the present invention;

[0061] Figure 6 A structural block diagram of an image target tracking device based on a hierarchical attention strategy is provided in an embodiment of the present invention;

[0062] Figure 7 A structural block diagram of a computer device is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0064] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] Combination Figure 1 As shown, an embodiment of the present invention provides an image target tracking method based on a hierarchical attention strategy, comprising:

[0066] S101, normalizing the target template of the twin neural network and the input images of the two branches of the search area to obtain a normalized image.

[0067] Specifically, the input images of the target template and search area branches of the twin neural network can be normalized by cropping or padding, with unified resolutions of 127*127 and 255*25.

[0068] S102: Input the normalized image into a residual network for feature extraction to obtain feature blocks of the last three layers.

[0069] Specifically, the normalized image information can be input into the residual network, and the feature blocks of the last three layers of the extracted information can be output. The residual network can adopt the ResNet-50 feature extraction network.

[0070] S103, hierarchically inputting the feature blocks of the last three layers into a hierarchical attention module for channel adjustment to obtain three feature blocks with a target number of channels, wherein the hierarchical attention module has three independent channel context attention modules.

[0071] Specifically, the acquired three-layer feature blocks can be hierarchically input into three independent channel context attention modules, and the three feature blocks output by the channel context attention modules can be uniformly adjusted to 256 channels.

[0072] S104, hierarchically input the three feature blocks into the output of the Transformer multi-layer feature enhancement mechanism module to obtain the three feature blocks corresponding to the two branches of the target template and the search area, and respectively merge and reduce the scale to obtain a target scale feature block.

[0073] Specifically, three feature blocks adjusted to 256 channels can be hierarchically input into the Transformer multi-layer feature enhancement mechanism module, and the three feature blocks of the output target template and search area branches are merged and scaled down to 7*7*256 and 31*31*256 respectively.

[0074] S105 , performing a depth-separable cross-correlation operation on the target-scale feature block to obtain a response graph.

[0075] The feature blocks of the two branches are subjected to a deeply separable cross-correlation operation to obtain a response map. The response map can be regarded as a linear mapping of the input image in the original search area in space with the target template as the weight. Each position in the response map can be mapped to the corresponding position in the original image space of the search area, and the key to the mapping lies in the size of the receptive field of the linear transformation.

[0076] S106: Input the response map into a classification regression prediction head to obtain a target tracking result.

[0077] The response map is input into the classification regression prediction head to obtain the tracking result. In this embodiment, a classification regression network based on the anchor-free mechanism is adopted to directly classify and regress the candidate frame at each position. There is no need to set prior information to initialize the anchor frame size, thereby breaking through the limitations of the scale and receptive field of the fixed aiming frame, minimizing the limitations and interference of the set prior information on the classification regression network, avoiding the introduction of multi-layer hyperparameter information that is difficult to optimize, and ensuring the computational efficiency of the classification regression network.

[0078] Combination Figure 2 As shown in the figure, the overall framework diagram of the model is shown. The model network results of this method mainly include the following four main modules: feature extraction backbone network, feature fusion and feature enhancement network, similarity matching network, and classification regression prediction head. The feature extraction backbone network can adopt ResNet-50 feature extraction network, and the feature fusion and feature enhancement network includes two hierarchical attention strategy (HAS) modules, which are defined as HAS module 1 and HAS module 2 respectively. Each HAS module is composed of a canonical correlation analysis (CCA, canonical correlation analysis) module and a Transformer multi-level feature enhancement TME module. Each TME module has four TEP modules. The classification regression prediction head includes a classification prediction head, a centrality comparison, and a regression prediction head.

[0079] In an embodiment of the present invention, an image target tracking method based on a hierarchical attention strategy is proposed. By introducing the HAS module composed of the channel global context attention mechanism and the Transformer multi-layer feature enhancement mechanism, the algorithm model can be applied to the advantages of deep semantic features and shallow spatial features from as many feature levels as possible, breaking through the limitations of the receptive field of traditional convolutional neural networks and achieving multi-layer feature enhancement. At the same time, it also solves the problem that other trackers based on twin neural networks do not fully utilize the deep and shallow feature information at different levels, and makes balanced use of the deep and shallow feature information. While effectively using the shallow information to help determine the target aiming frame position, the deep semantic information is enhanced to help the network better complete the judgment of the foreground and background binary attributes of each pixel position. While achieving real-time detection, it greatly improves the target tracking progress and success rate, providing a feasible method for target tracking in aerial video sequences.

[0080] In step S103, the hierarchical attention module adopts a typical correlation analysis CCA module, and the three feature blocks with the target number of channels obtained by hierarchically inputting the feature blocks of the last three layers into the hierarchical attention module for channel adjustment specifically include:

[0081] Combination Figure 3 As shown, the structure diagram of the channel contextual attention mechanism module and the structure diagram of the TEP module are shown. The last three layers of feature blocks output after the residual network extracts features are used as the QKV ternary input of the three-level typical association analysis CCA module, and the output is the optimized three-layer feature block of the same dimensional features. Among them, the CCA module is the proposed improved channel context-aware attention mechanism module applied to the feature fusion network, which attempts to apply the method of the global contextual attention mechanism to features at different levels, and realizes the mining process of feature information at different depths from the channel dimension. The channels containing the target features are modeled with the global context to obtain a larger attention score, thereby highlighting the information of these channels, preparing for the next step of Transformer multi-level feature enhancement, and completing the data preprocessing of the input data. Specifically, the CCA module reshapes the input feature map A to obtain three new maps of dimension CxN, and then transposes one of the maps to obtain A. T , for the obtained A T Perform matrix multiplication with another reshaped feature map to obtain an attention map X with a dimension of C×C, and then pass it through the softmax layer to obtain the attention score map. T The process of multiplying the matrix with another feature map A of dimension CxN can apply the attention scoring process to different channels according to different attention scores, and then restore the result to the original dimension to obtain the attention optimized feature map O. The attention map involved in the calculation in this process can be expressed as:

[0082]

[0083] Then the output attention optimized feature map O and the feature map A are fused together through channel cascade, and finally the final output feature map E is obtained through a 1*1 convolution process. This process can be expressed by the following equation.

[0084] E = conv(concat(A,O)) (10)

[0085] In summary, the proposed CCA module globally models the global context information in the channel dimension, uses the self-attention mechanism to capture the channel dependency between any two channel maps, and updates each channel map using the weighted sum of all channels, thus completing the optimization process of the channel dimension of the source input data.

[0086] In the process of adjusting the channel dimensions of each feature block, a multi-layer convolutional layer dimension adjustment strategy is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading from the original [512, 1024, 2048] channels to [256, 256, 256] channels to reduce subsequent parameters and calculations. The dimension of the number of channels of the feature block will become 3*256.

[0087] In step S104, the three feature blocks are hierarchically input into the Transformer multi-layer feature enhancement mechanism module output to obtain the target template and the three feature blocks corresponding to the search area branches, respectively merged and scaled down to obtain the target scale feature block, specifically including:

[0088] Combination Figure 4 As shown in the figure, the TEP module is a modified Transformer encoder, in which the average pooling strategy is introduced as a preprocessing mechanism for input data to optimize the K and V parameters. In order to further optimize and obtain a lighter Transformer-based structure for object tracking tasks, we replace the position encoding step in the traditional Transformer with the position information of the input sequence, and adopt a zero-padding strategy to ensure the integrity of the sequence information. The multi-layer neural network MLP layer in the network structure represents the fully connected layer in the network, which is used to integrate the feature information of each channel. The Norm layer maintains the stability of the input feature distribution by normalizing the input features, avoiding the gradient vanishing problem in the deep network. Specifically, the LayerNorm layer is used for normalization in the channel direction, and the Add&Norm layer is used for residual connection and layer normalization operations.

[0089] Combination Figure 5 As shown in the figure, after dimensionality adjustment, the feature blocks of each layer are flattened into sequence information using convolution operations and used as the input of the subsequent Transformer multi-level feature enhancement TME module. The three layers of sequence information in the structure of the TME module are used as QKV triples of three TEP blocks respectively. The proposed TME module contains a total of 4 TEP modules. The result of the TEP block with conv4 as input is sent to another TEP block. Through the calculation of the double-layer self-attention mechanism, a better balance between deep features and shallow features is achieved. The core process of the TEP block is the process of processing the input triple data, which is essentially the calculation process of the multi-head attention mechanism. The purpose of multi-head attention is to prevent the model from paying attention to only certain features and ignoring other features, so we can let the model have multiple attention layers with the same structure but different weights. Each head pays attention to different types of features, so the model as a whole will pay attention to more features. Therefore, if we execute the attention heads in parallel by linear projection, the dimension of the output value will be affected by the number of attention heads. Finally, we can get the final value through another linear projection process after channel cascading. Multi-head attention enables the model to jointly focus on information from different representation subspaces at different positions. Its calculation process can be expressed by the following formula:

[0090]

[0091] In the above formula, Q, K, and V represent the query variable, the key in the initial key-value pair, and the value in the initial key-value pair; d k Represents the number of channels. This module calculates the similarity between Q and K, multiplies V by the normalized attention distribution weight, and realizes feature enhancement of V. The specific calculation process includes: initializing multiple sets of coefficient weight matrices, calculating the input of each set of multi-head attention, and then calculating the attention map of each group according to the self-attention calculation process. After cascading in the channel dimension, it is mapped to the original space through linear transformation. The final attention output has the same dimension as the original input.

[0092] The process of refining the data through multiple convolutional layers further reduces the number of channels and pixels of the feature map, thereby greatly reducing the computational effort of the similarity matching network.

[0093] In step S105, a depth-separable cross-correlation operation is performed on the target scale feature block to obtain a response graph, including:

[0094] Combination Figure 5 As shown in the figure, the deep separable cross-correlation operation and the subsequent process of reducing the feature scale are shown. In the similarity matching network, by performing hierarchical convolution on the corresponding layers of the target template and the search area 3, 4, and 5, the deep separable cross-correlation operation is used to calculate the response map of each level of cross-correlation operation, thereby realizing the fusion of deep features and shallow features in the output response map. The process can be described as:

[0095]

[0096] in, Represents a deep separable cross-correlation operation. The reason why the feature scale needs to be reduced is that the number of channels of the output response map obtained at this time is 256*3. If it is directly used as the input of the subsequent classification and regression network, the amount of calculation is still too large. Therefore, we use a 1*1 convolution layer to adjust the number of channels of the response map, reduce the number of model parameters and increase the model derivation speed, to maximize the balance between model effect and calculation efficiency.

[0097] In step S106, the response map is input into the classification regression prediction head to obtain the target tracking result, including:

[0098] A classification and regression network based on an anchor-free mechanism is used to directly classify and regress the candidate boxes at each position in the response map to obtain the target tracking result. The response map represents the linear mapping of the input image in the search area in space with the target template as the weight, and each position in the response map is mapped to the corresponding position in the original image space of the search area.

[0099] In some embodiments, the uniform resolution of the normalized image is 127*127 and 255*255, the target number of channels is 256 channels, and the scale of the target scale feature block is 7*7*256 and 31*31*256.

[0100] In some embodiments, in the process of adjusting the channel dimensions of each feature block, a dimensionality adjustment strategy of a multi-layer convolutional layer is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading to the target channel, including: adopting a dimensionality adjustment strategy of a multi-layer convolutional layer to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading from the original [512, 1024, 2048] channels to [256, 256, 256] channels, and the dimension of the number of channels of the feature block will become 3*256.

[0101] In the embodiment of the present invention, an image target tracking device based on a hierarchical attention strategy is provided, and a tracker SiamHAS based on a twin neural network is used to solve the tracking drift and tracking failure problems in challenging aerial photography scenes, such as occlusion, aspect ratio change, low resolution and scale change. The proposed HAS module adopts a hierarchical attention strategy, which can apply the advantages of deep semantic features and shallow spatial features from as many feature levels as possible, break through the limitations of the receptive field of traditional convolutional neural networks, and achieve multi-layer feature enhancement. In the part of the channel context attention mechanism, we have achieved global feature enhancement in the channel dimension. Through the process of channel context attention modeling, the model adaptively realizes the allocation of attention scores of each channel on the basis of global modeling, so that there is information exchange between different channels, including more global context information, and better feature optimization is achieved. In addition, the self-attention enhancement of deep information is realized through multiple parallel multi-head attention networks, which enhances the ability of the model to distinguish foreground and background while balancing the deep and shallow feature information. While greatly improving the algorithm to cope with challenging scenes, the real-time performance of the algorithm is guaranteed, so that the network can achieve real-time tracking. From the perspective of network structure, the present invention aims to address the problem that the existing algorithms fail to effectively utilize the characteristics of features at different levels in the data enhancement strategy before feature fusion to improve the tracking effect in air-to-ground scenarios. In the feature fusion subnetwork, this paper introduces a feature enhancement method that integrates a hierarchical attention strategy. Through the channel context mechanism, global feature information can be utilized between different channels so that the features of each channel contain more spatial context information, thereby realizing feature enhancement in the channel dimension and effectively improving the utilization of shallow feature information. The introduction of a multi-layer feature enhancement strategy of the multiple improved Transformer encoder improves the significance of the deep features of each branch, thereby effectively improving the performance of the algorithm.

[0102] Combination Figure 6 As shown, accordingly, an embodiment of the present invention provides an image target tracking device based on a hierarchical attention strategy, comprising:

[0103] A normalization module unit 601 is used to normalize the input images of the target template and the two branches of the search area of ​​the twin neural network to obtain a normalized image;

[0104] A feature extraction unit 602 is used to input the normalized image into a residual network for feature extraction to obtain feature blocks of the last three layers;

[0105] A channel adjustment unit 603 is used to hierarchically input the feature blocks of the last three layers into a hierarchical attention module for channel adjustment to obtain three feature blocks with a target number of channels, wherein the hierarchical attention module has three independent channel context attention modules;

[0106] The scale adjustment unit 604 is used to hierarchically input the three feature blocks into the output of the Transformer multi-layer feature enhancement mechanism module to obtain the three feature blocks corresponding to the target template and the search area branches, respectively merge and reduce the scale to obtain a target scale feature block;

[0107] A similarity matching network unit 605 is used to perform a depth-separable cross-correlation operation on the target scale feature block to obtain a response map;

[0108] The prediction unit 606 is used to input the response map into the classification regression prediction head to obtain the target tracking result.

[0109] In an embodiment of the present invention, an image target tracking device based on a hierarchical attention strategy is proposed. By introducing the HAS module composed of the channel global context attention mechanism and the Transformer multi-layer feature enhancement mechanism, the algorithm model can be applied to the advantages of deep semantic features and shallow spatial features from as many feature levels as possible, breaking through the limitations of the receptive field of traditional convolutional neural networks and achieving multi-layer feature enhancement. At the same time, it also solves the problem that other trackers based on twin neural networks do not fully utilize the deep and shallow feature information at different levels, and makes balanced use of the deep and shallow feature information. While effectively using the shallow information to help determine the target aiming frame position, the deep semantic information is enhanced to help the network better complete the judgment of the foreground and background binary attributes of each pixel position. While achieving real-time detection, it greatly improves the target tracking progress and success rate, providing a feasible method for aerial video sequence target tracking.

[0110] Accordingly, according to an embodiment of the present invention, the present invention also provides a computer device, a readable storage medium and a computer program product.

[0111] Figure 7 FIG. 1 is a schematic diagram of the structure of a computer device 12 provided in an embodiment of the present invention. Figure 7A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 7 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0112] like Figure 7 As shown, computer device 12 is in the form of a general-purpose computing device. Computer device 12 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0113] Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16 , a system memory 28 , and a bus 18 that connects various system components including system memory 28 and processing unit 16 .

[0114] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0115] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0116] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0117] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0118] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the computer device 12, and / or may communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the image target tracking method based on the hierarchical attention strategy provided in the embodiment of the present invention.

[0120] An embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein when the program is executed by a processor, the image target tracking method based on a hierarchical attention strategy provided in all the inventive embodiments of the present application is implemented.

[0121] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, an apparatus, or a device.

[0122] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0123] The program code included in the computer-readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages ​​such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect to the Internet).

[0124] An embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned image target tracking method based on the hierarchical attention strategy.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document is not limited here.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image target tracking method based on a hierarchical attention strategy, characterized in that: include: Normalizing the target template of the twin neural network and the input images of the two branches of the search area to obtain a normalized image; Input the normalized image into the residual network for feature extraction to obtain feature blocks of the last three layers; Inputting the feature blocks of the last three layers into a hierarchical attention module for channel adjustment to obtain three feature blocks with a target number of channels, wherein the hierarchical attention module has three independent channel context attention modules; The hierarchical attention module adopts a typical correlation analysis CCA module, and the three feature blocks with a target number of channels obtained by hierarchically inputting the feature blocks of the last three layers into the hierarchical attention module for channel adjustment include: The feature blocks of the last three layers are hierarchically input into the CCA module. The CCA module reshapes the feature map A of the input feature block to obtain three new maps with dimensions of CxN, and transposes any one of the three new maps to obtain A T , for the obtained A T Perform matrix multiplication with another reshaped feature map to obtain an attention map X with a dimension of C×C. The attention map X is passed through a softmax layer to obtain an attention score map. By X T The process of multiplying the feature map A matrix with another dimension of CxN is to apply the attention scoring process to different channels according to different attention scores, and restore the result to the original dimension to obtain the attention optimized feature map O. The attention map involved in the calculation is expressed as (1): (1) The output attention optimized feature map O and the feature map A are fused together through channel cascading, and the final output feature map E is obtained through a 1*1 convolution process. The process is expressed by equation (2): (2) The CCA module is used to globally model the global context information of the channel dimension, capture the channel dependency between any two channel maps using the self-attention mechanism, and update each channel map using the weighted sum of all channels, thereby completing the optimization of the channel dimension of the source input data; In the process of adjusting the channel dimensions of each feature block, a multi-layer convolutional layer dimension adjustment strategy is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading to the target channel; The three feature blocks are hierarchically input into the Transformer multi-layer feature enhancement mechanism module to obtain the target template and the three feature blocks corresponding to the search area branches, respectively merged and scaled down to obtain the target scale feature block; Performing a depth-separable cross-correlation operation on the target scale feature block to obtain a response map; The response map is input into the classification regression prediction head to obtain the target tracking result.

2. The image target tracking method based on the hierarchical attention strategy according to claim 1 is characterized in that: The step of inputting the three feature blocks into the output of the Transformer multi-layer feature enhancement mechanism module to obtain the target template and the three feature blocks corresponding to the search area branches are merged and scaled down to obtain the target scale feature block, including: After dimension adjustment, the feature blocks of each layer are flattened into sequence information using convolution operations and used as input to the subsequent Transformer multi-level feature enhancement module, wherein the multi-level feature enhancement module includes four pooling strategy encoder TEP modules. The three layers of sequence information in the structure of the multi-level feature enhancement module are respectively used as QKV triplet inputs of the four TEP modules. The result of the TEP block with conv4 as input is sent to another TEP module, and the balance between deep features and shallow features is performed through the calculation of a double-layer self-attention mechanism. Multi-head attention is used to make the model focus on information from different representation subspaces at different positions. The calculation process is expressed as formula (3): (3) In the above formula (3), Q, K, and V represent the query variable, the key in the initial key-value pair, and the value in the initial key-value pair, d k Represents the number of channels, calculates the similarity between Q and K, multiplies V by the normalized attention distribution weight, and realizes the feature enhancement of V. The specific calculation process includes: initializing multiple sets of coefficient weight matrices, calculating the input of each set of multi-head attention, calculating the attention map of each group according to the self-attention calculation process, cascading on the channel dimension and mapping it to the original space through linear transformation, and the obtained attention output has the same dimension as the original input; The data is refined through multiple convolutional layers to reduce the number of channels and pixels of the feature map.

3. The image target tracking method based on the hierarchical attention strategy according to claim 2 is characterized in that: The step of performing a depth-separable cross-correlation operation on the target scale feature block to obtain a response graph includes: By performing hierarchical convolution on the corresponding layers of the target template and the search area, a depth-separable cross-correlation operation is used to calculate the response map of each level of cross-correlation operation, and a fusion of deep features and shallow features in the output response map. The process is described as (4): (4) in, Represents the depthwise separable cross-correlation operation.

4. The image target tracking method based on the hierarchical attention strategy according to claim 3 is characterized in that: The step of inputting the response graph into a classification regression prediction head to obtain a target tracking result includes: A classification and regression network based on an anchor-free mechanism is used to directly classify and regress the candidate boxes at each position in the response map to obtain the target tracking result. The response map represents the linear mapping of the input image in the search area in space with the target template as the weight, and each position in the response map is mapped to the corresponding position in the original image space of the search area.

5. The image target tracking method based on a hierarchical attention strategy according to any one of claims 1 to 4, characterized in that: The uniform resolution of the normalized image is 127*127 and 255*255, the target number of channels is 256 channels, and the scales of the target scale feature blocks are 7*7*256 and 31*31*256.

6. The image target tracking method based on the hierarchical attention strategy according to claim 1, characterized in that: In the process of adjusting the channel dimensions of each feature block, a dimension adjustment strategy of a multi-layer convolutional layer is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading to the target channel, including: By adopting the dimension adjustment strategy of multi-layer convolutional layers, the number of channels of the three-layer feature blocks after channel cascading is uniformly adjusted from the original [512, 1024, 2048] channels to [256, 256, 256] channels, and the dimension of the number of channels of the feature block will become 3*256.

7. An image target tracking device based on a hierarchical attention strategy, characterized in that: A normalization module unit is used to normalize the input images of the target template and the two branches of the search area of ​​the twin neural network to obtain a normalized image; A feature extraction unit, used for inputting the normalized image into a residual network for feature extraction to obtain feature blocks of the last three layers; A channel adjustment unit, used for hierarchically inputting the feature blocks of the last three layers into a hierarchical attention module for channel adjustment to obtain three feature blocks with a target number of channels, wherein the hierarchical attention module has three independent channel context attention modules; The hierarchical attention module adopts a typical correlation analysis CCA module, and the three feature blocks with a target number of channels obtained by hierarchically inputting the feature blocks of the last three layers into the hierarchical attention module for channel adjustment include: The feature blocks of the last three layers are hierarchically input into the CCA module. The CCA module reshapes the feature map A of the input feature block to obtain three new maps with dimensions of CxN, and transposes any one of the three new maps to obtain A T , for the obtained A T Perform matrix multiplication with another reshaped feature map to obtain an attention map X with a dimension of C×C. The attention map X is passed through a softmax layer to obtain an attention score map. By X T The process of multiplying the feature map A matrix with another dimension of CxN is to apply the attention scoring process to different channels according to different attention scores, and restore the result to the original dimension to obtain the attention optimized feature map O. The attention map involved in the calculation is expressed as (1): (1) The output attention optimized feature map O and the feature map A are fused together through channel cascading, and the final output feature map E is obtained through a 1*1 convolution process. The process is expressed by equation (2): (2) The CCA module is used to globally model the global context information of the channel dimension, capture the channel dependency between any two channel maps using the self-attention mechanism, and update each channel map using the weighted sum of all channels, thereby completing the optimization of the channel dimension of the source input data; In the process of adjusting the channel dimensions of each feature block, a multi-layer convolutional layer dimension adjustment strategy is adopted to uniformly adjust the number of channels of the three-layer feature blocks after channel cascading to the target channel; A scale adjustment unit is used to hierarchically input the three feature blocks into the output of the Transformer multi-layer feature enhancement mechanism module to obtain the three feature blocks corresponding to the two branches of the target template and the search area, and respectively merge and reduce the scale to obtain a target scale feature block; A similarity matching network unit is used to perform a deep separable cross-correlation operation on the target scale feature block to obtain a response map; The prediction unit is used to input the response map into the classification regression prediction head to obtain the target tracking result.

8. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the image target tracking method based on a hierarchical attention strategy as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the image target tracking method based on a hierarchical attention strategy as described in any one of claims 1 to 6.

Images