Unmanned aerial vehicle target tracking method based on feature fusion and sparse attention

Abstract

The invention discloses an unmanned aerial vehicle target tracking method based on feature fusion and sparse attention, and the method comprises the following steps: selecting improved Res2NeXt to obtain multi-scale features in order to enable a tracker to directly pay attention to an important space region and avoid the increase of the calculation amount, and carrying out the recognition of the multi-scale features through the improved Res2NeXt. Channel interdependence and space interdependence of target features are learned, self-adaption fusion is carried out, and feature representation is enhanced. A self-context enhancement module and a cross feature module are used, so that useful semantic context information can be adaptively focused to perform feature enhancement, and feature maps of each branch can be accepted to perform fusion. And introducing a regression and classification model as a prediction head to predict a target template. The method can solve the problems of target shielding and small target of the unmanned aerial vehicle, improves the local feature processing capability of the model, and effectively improves the accuracy of a tracking algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle video processing, and particularly designs an efficient unmanned aerial vehicle video target tracking algorithm based on feature fusion and sparse attention by optimizing feature extraction, fusion and classification regression mechanisms. Background Art

[0002] Drones are small, highly concealed, and capable of capturing images, and have been widely used in civil and military fields. Target tracking technology is one of the important technologies in drone applications, which can improve the autonomous flight and monitoring capabilities of drones and achieve real-time monitoring and tracking of targets. However, drones need to constantly adjust their flight angles and altitudes during tracking. Classic target tracking algorithms are subject to occlusion, deformation, and scale changes during tracking, and face interference from similar objects, backgrounds, and other factors, resulting in poor tracking results. Therefore, designing an effective and accurate target tracking algorithm to address the impact of various interference factors is the main content of current research.

[0003] During the flight of a drone, the target may be severely obscured, and changes in illumination can easily cause the target to be lost; the geometric deformation, rotation, and attitude change of the target also bring great challenges to tracking. At present, the target tracking algorithm is mainly based on correlation filtering and deep learning. The CSK (Circulant Structure with Kernels) filter algorithm combines the characteristics of the kernel method and the circulant matrix to improve the tracking accuracy. The KCF (Kernel Correlation Filter) filter algorithm introduces multidimensional features and Gaussian kernels on the basis of CSK to further improve the accuracy of the algorithm. Based on the traditional target tracking method, it is necessary to manually extract features, which is not ideal for the actual application of drones in complex backgrounds. Deep learning technology uses deep features instead of manually extracted features and has good feature expression capabilities. Therefore, researchers use deep learning technology in various fields of computer vision. SiameseFC uses a twin network structure to learn the similarity between the target and the background, thereby realizing the tracking of the target and solving the problem of insufficient feature expression of the correlation filter algorithm. Siamese series networks have appeared one after another. Although the Siamese series tracking algorithms have achieved significant improvements in tracking success rate and accuracy, drift is still prone to occur when the target is deformed or occluded, resulting in reduced algorithm accuracy.

[0004] In summary, in order to solve the problem that the target in the drone video accounts for a small proportion and is easily disturbed by complex background, a drone tracking algorithm based on adaptive multi-scale feature fusion and sparse attention is proposed. First, in order to enable the tracker to focus directly on important spatial areas while avoiding increasing the amount of calculation, the improved Res2NeXt is selected to obtain multi-scale features, learn the channel interdependence and spatial interdependence of target features, and adaptively fuse them to enhance the feature representation. Secondly, the self-context enhancement module and the cross-feature module are used to adaptively focus on useful semantic context information for feature enhancement and accept the feature map of each branch for fusion. Finally, the regression and classification models are introduced as prediction heads to predict the target template. The simulation experimental results show that the algorithm can solve the problems of drone target occlusion and small targets, improve the model's ability to process local features, and effectively improve the accuracy of the tracking algorithm.

[0005] In recent years, researchers have proposed a variety of attention mechanisms to enhance features, and introduced the Transformer structure into the visual target tracking algorithm to achieve the acquisition of global feature information and achieved good results. TransT, Stark, Trdimp, Trtr, Mixformer and other algorithms introduce Transformer into the twin network, which improves the extraction and fusion effects of features and enhances the fusion ability of the algorithm. Although the above Transformer-based target tracking algorithms have made good progress, they are mainly for better fusion or enhancement of target and search area features extracted from deep convolutional networks, ignoring the ability of the visual Transformer itself in feature extraction and decoding prediction. Although Transformer performs well in processing long-distance dependencies, it does not fully utilize local feature relationships, and the problem of processing small targets needs to be improved. The above algorithms improve the target tracking accuracy and reduce the impact of appearance changes on the algorithm performance, but it is still difficult to meet the actual needs of drone target tracking in complex scenarios. Summary of the invention

[0006] In the process of performing tasks, the UAV is affected by its own flight posture, which may cause the target to be lost during the tracking process, affecting the tracking accuracy. In order to ensure that the UAV can track the target with high precision, the overall network model of the tracking algorithm proposed in this paper based on the twin network and Transformer structure mainly includes the following parts: The first part is the Res2NeXt feature extractor with adaptive multi-scale feature fusion, which adaptively fuses the acquired multi-scale features to enhance the feature expression ability. The second part is the improved sparse attention algorithm, which uses sparse attention to enhance the feature maps from the two branches. The third part is the feature fusion and regression branch, which accepts the feature maps of each branch for fusion, and inputs the reorganized feature response map into the classification regression branch to obtain the final tracking position.

[0007] To achieve the above object, the present invention proposes a UAV video adaptive target tracking algorithm based on feature fusion and sparse attention, which includes the following steps:

[0008] S1: Select a large number of image sequences related to the target to be tracked collected by drones and pre-train the Am-Res2Next network. The training method is as follows:

[0009] S1.1: Thirty-two Am-Res2Net modules and one residual connection to construct the Am-Res2Next network;

[0010] S1.2: First, the input is divided into 32 branches for group convolution. The first layer uses a 1x1 convolution kernel to reduce the number of channels to 4. In the second convolution layer, the input 32 branches are divided into x according to the number of channels. i (1, 2, ..., 8) small branches are used as the input of the Am-Res2Net module. The third layer performs a 3×3 convolution operation on each small branch and outputs it as y i (1, 2, ..., 8);

[0011] S1.3: Change y i The features are quickly normalized to calculate the feature weight fusion and obtain the fusion target feature map y;

[0012] S2: Designed a feature enhancement and fusion branch of a Siamese structure composed of a Sparse-Transformer network, constructed an effective loss function, distinguished positive and negative samples at the location points of the response map, and designed a loss function for each point;

[0013] S3: The loss mean of all points is used as the overall loss of the response graph, and the input labeled image is used to calculate the loss using the loss mean and use the SGD algorithm to optimize the network weights;

[0014] S4: convert the video into image frames and input them into the system, and read the first frame;

[0015] S5: Use the template branch to extract the first frame target image feature f exemplar ;

[0016] S6: Read the next frame and use the search branch to extract the target feature f of the current frame instance ;

[0017] S7: f exemplar and f instance The features are sent to the Sparse-Transformer network, and the features of the two branches are enhanced through the self-context module and the cross-feature module;

[0018] S8: The feature map in the feature enhancement dual branch is sent to the cross feature module again for feature enhancement. By repeating the process of feature mutual enhancement and aggregation network multiple times, more discriminative features can be obtained;

[0019] S9: In each frame image, take n windows around the target position of the previous frame image of the frame image for multi-scale transformation, feature extraction, and dimensionality reduction, and use the classifier trained by the previous frame image for classification. The window with the largest classification score is the tracking target;

[0020] S10: Repeat S4-S9 until all drone video frames are processed, and obtain and output the video tracking results.

[0021] Beneficial Effects

[0022] According to the present invention, in view of the problem that the target in the drone video accounts for a small proportion and is easily disturbed by complex background, it is necessary to extract the features of small targets. A network model based on feature fusion and sparse attention is constructed by combining Siamese structure and Transformer network. Combining the multi-scale feature expression of Res2neXt and the gradient optimization characteristics of the residual network structure, the rich environmental background and scene information in the drone tracking task are effectively obtained, and an adaptive multi-scale fusion convolutional neural network Am-Res2Net is constructed, which can effectively extract the features of small targets and adaptively fuse the deep and shallow layer information according to the contribution. A sparse attention feature enhancement and fusion strategy is proposed, which can improve the resolution of the foreground background and solve the problem of complex background interference. Simulation experiments show that the proposed algorithm has good performance in scenes such as scale change, beyond the field of view, occlusion, deformation, etc., and can process these videos well. For test videos in other scenes, the performance of the proposed algorithm is also good, which fully verifies that the proposed algorithm has excellent overall performance in tracking accuracy, stability and robustness. The tracking accuracy of the feature fusion and enhancement methods proposed in this paper reached 0.849, which is 2.6% and 2.8% higher than the original residual network method and the original Transformer for target tracking, respectively; the tracking success rate reached 0.798, which is 3.2% and 2.3% higher than the original residual network method and the original Transformer for target tracking, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 Flow chart of a Transformer-based drone video target tracking algorithm method according to an embodiment of the present invention;

[0025] Figure 2 This is a diagram of the Am-Res2net residual block network structure of an embodiment of the present invention;

[0026] Figure 3 It is a schematic diagram of a network model based on feature fusion and feature enhancement according to an embodiment of the present invention;

[0027] Figure 4 Schematic diagram of the self-context attention network model and the cross-feature network model of the present invention.

[0028] Figure 5 This is a schematic diagram of a sparse attention network model of the present invention. DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or components having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0030] According to the present invention, the UAV video target tracking method based on feature fusion and sparse attention includes the following steps:

[0031] S1: Select a large number of image sequences related to the target to be tracked collected by drones and pre-train the Am-Res2NeXt network. The training method is as follows:

[0032] S1.1: The Am-Res2Next network is constructed by thirty-two Am-Res2Net modules and one residual connection;

[0033] The Am-Res2NeXt residual network has 5 layers, F = W2σ(W1x), y = F(x,{W i})+x, where σ represents the nonlinear function ReLU, and then through a shortcut and the second ReLU, the output y is obtained. Considering the computational cost, the residual block is optimized. For the first convolution layer, a 1x1 convolution kernel is used to reduce the number of channels to 4. Then, in the second convolution layer, each of the 32 branches is divided into x according to the number of channels. i (i=1,2,…,s) is used as the input of the filter bank. Then the processed features are concatenated and input to the third convolutional layer and the dimension is increased to 256. Finally, the output features of these 32 branches are added together and then the output result is added to the input feature map through a short connection to obtain the final output result y.

[0034] S1.2: In the fusion part of Am-Res2Net, in order to optimize the fusion method of deep and shallow feature information, a fusion method that distinguishes different input features is designed, and the bitwise addition operation of features is adopted. In general, the bitwise addition between features will cause mutual contamination of feature information. In order to optimize the method of deep and shallow feature information fusion, this paper adopts a fast normalization method to calculate feature weights to reduce mutual interference between information. In order to solve the problem of network model degradation caused by gradient disappearance or explosion, the Am-Res2Net residual network structure is proposed. The shortcut connection method effectively reduces the number of layers that the network gradient passes through during propagation, so that the problem of explosion and disappearance of loss value is alleviated, the network model training speed is accelerated, and the expression ability of the network model is improved. Therefore, combining the deep and shallow information and the multi-scale optimization characteristics of Res2NeXt, the Am-Res2NeXt network is constructed to extract image features of the tracking target.

[0035] S2: Designed a feature enhancement and fusion branch of a Siamese structure composed of a Sparse-Transformer network, constructed an effective loss function, distinguished positive and negative samples at the location points of the response map, and designed a loss function for each point;

[0036] S3: The loss mean of all points is used as the overall loss for the Siamese structural response map, and the input labeled image is used to calculate the loss using the loss mean and use the SGD algorithm to optimize the network weights;

[0037] The network has two branches, such as Figure 3 As shown in Figure 2, the upper part of the network is the template branch, and the lower part is the search branch. The similarity function f(z,x) is learned in space. Compare the target template image z and the current frame image x, and find the sample that is most similar to the target template image as the prediction target. The expression is Where: * indicates that the two feature map matrices are cross-correlated; b is a bias and is the same at each position. In order to construct an effective loss function, the loss function for each point in the response map is l(y,v)=log(1+exp(-yv)). Where v is the true value of each point, and y∈{+1,-1} is the label corresponding to this point.

[0038] For the overall loss of the corresponding graph, the mean loss of all points is used, that is: Where u∈D is the position in the response graph. y[u] is expressed as: In the formula, k is the network step size, c is the center point, and R is the radius of the search area. The parameters θ of the convolutional network are optimized using the SGD (Stochastic Gradient Descent) optimization algorithm to calculate the loss: The network is divided into 2 inputs, one input target image, size 127×127×3, the other input current frame search area image, size 255×255×3, the search area is 4 times the target size of the previous frame, and a cosine window is added to the response map to penalize the maximum displacement;

[0039] S4: convert the video into image frames and input them into the system, and read the first frame;

[0040] S5: Use the template branch to extract the first frame target image feature f exemplar ;

[0041] S6: Read the next frame and use the template branch to extract the target feature f of the current frame instance ;

[0042] S7: For feature fexemplar and f instance The Sparse-Transformer is used to perform feature enhancement and fusion processing, and the features of the two branches are enhanced through the self-context module and the cross-feature module. Figure 4 As shown in Figure 2, the self-context module uses sparse multi-head self-attention in the form of residuals to adaptively integrate information from different positions of the feature map. The attention mechanism cannot distinguish the position information of the input feature sequence. Therefore, this paper introduces the spatial position encoding process into Then a sine function is used to generate the spatial position encoding, and the calculation expression is as follows: EnC =X+SMultiHead(X+P X , X+P X , X), the cross-feature module fuses the feature vectors of the two inputs in the form of sparse multi-head cross-attention residuals. Similar to the self-context module, spatial position encoding is also used in the cross-feature module, and the expression is as follows:

[0043] In order to improve the resolution of foreground and background and reduce the ambiguity of foreground edge areas, the sparse multi-head self-attention algorithm is used. By selectively allocating attention, the sparse attention mechanism can focus most of the attention on the key areas of the target. In the original multi-head attention calculation, each pixel value of the attention feature is calculated by all the pixel values ​​of the input feature, which makes the edge area of ​​the foreground blurred. In SMultiHead, each pixel value of the attention feature is determined only by the K most similar pixel values, such as Figure 5 As shown in the figure, unlike the original Transformer, only the Softmax function is used here to normalize the K largest elements of each row of the similarity matrix. The other elements are set to 0;

[0044] S8: In the feature fusion part, the cross-feature modules can enhance the features of the two branches respectively, thereby constructing a feature mutual enhancement and aggregation network. By repeating the process of feature mutual enhancement and aggregation network many times, more discriminative features can be obtained. At the same time, the cross-feature module can also measure the similarity between the two branches and obtain a response map. Using feature mutual enhancement and aggregation network can greatly reduce the amount of calculation;

[0045] S9: The prediction head structure consists of two branches: classification branch and regression branch. Each branch consists of a three-layer perceptron with a hidden layer dimension of d and a ReLU activation function. These head branches are used to perform two main tasks for each vector in the feature map: classification prediction of the foreground and background and coordinate prediction of the search area size;

[0046] S10: Repeat S5-S9 until all drone video frames are processed, and obtain and output the video tracking results.

[0047] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A UAV target tracking method based on feature fusion and sparse attention, characterized by: The method comprises the following steps, S1: Select an image sequence related to the target to be tracked collected by the drone and pre-train the Am-Res2Next network; the pre-training method is as follows: S1.1: Thirty-two Am-Res2Net modules and one residual connection to construct the Am-Res2Next network; S1.2: First, the input is divided into 32 branches for group convolution; the first layer uses a 1x1 convolution kernel to reduce the number of channels to 4. In the second convolution layer, the input 32 branches are divided into x small branches according to the number of channels, and each x i As the input of the Am-Res2Net module; the third layer performs a 3×3 convolution operation on each small branch and outputs it as y i ; S1.3: Change y i The features are quickly normalized to calculate the feature weight fusion and obtain the fusion target feature map y; S2: Design the feature enhancement and fusion branch of the Siamese structure composed of the Sparse-Transformer network, construct the loss function, distinguish the positive and negative samples at the position points of the response map, and design the loss function for each point; S3: The loss mean of all points is used as the overall loss of the response graph, and the input labeled image is used to calculate the loss using the loss mean and use the SGD algorithm to optimize the network weights; S4: convert the video into image frames after preprocessing and input the system, and read the first frame; S5: Use the Am-Res2Next backbone network of the template branch to extract the target image features of the first frame, and fuse the outputs of the last three layers to obtain the template features f exemplar ; S6: Read the next frame and use the search branch to extract the target feature f of the current frame search ; S7: respectively convert the current frame target features f exemplar and the template feature f instance The features are sent to the Sparse-Transformer network, and the features of the two branches are enhanced through the self-context module and the cross-feature module; S8: The feature map in the feature enhancement dual branch is sent to the cross feature module again for feature enhancement. By repeating the process of feature mutual enhancement and aggregation network multiple times, more discriminative features are obtained; S9: Use the prediction head to classify and regress the feature image, and the window with the largest score is the tracking target; S10: Repeat S5-S9 until all drone video frames are processed, and obtain and output the video tracking results.

2. The target tracking method using feature fusion and sparse attention network according to claim 1, characterized in that: In S1, the Am-Res2Next network is constructed by thirty-two Am-Res2Net modules and a residual connection. The Am-Res2Next residual network has five layers. For the first convolutional layer, a 1x1 convolution kernel is used to reduce the number of channels to 4. Then, in the second convolutional layer, the 32 branches are divided into x branches according to the number of channels, and each x branch is i As the input of the filter group; then the processed features are concatenated and input into the third convolution layer and the dimension is increased to 256. Finally, the output features of the 32 branches are added and then the output result is added to the input feature map through a short connection to obtain the final output result; In the fusion part of Am-Res2Net, a fusion method that distinguishes different input features is designed, and the bitwise addition operation of features is adopted. The bitwise addition between features will cause mutual contamination of feature information. In order to optimize the method of fusion of deep and shallow feature information, a fast normalization method is used to calculate the feature weights to reduce mutual interference between information; the Am-Res2Net residual network structure is proposed, and the number of layers passed by the network gradient propagation is reduced through shortcut connection. Combining the deep and shallow layer information and the multi-scale optimization characteristics of Res2NeXt, the Am-Res2NeXt network is constructed to extract the image features of the tracking target.

3. The target tracking method using feature fusion and sparse attention network according to claim 1, characterized in that: There are two branches in S2; the upper part of the network is the template branch, and the lower part is the search branch, and it learns the similarity function in space Compare the target template image z and the current frame image x, and find the sample that is most similar to the target template image as the prediction target; to construct an effective loss function, the loss function for each point in the response graph is l(y,v)=log(1+exp(-yv)); where v is the true value of each point, and y∈{+1,-1} is the label corresponding to this point; for the overall loss of the corresponding graph, the loss mean of all points is used, that is: Where u∈D is the position in the response graph; y[u] is expressed as: Where k is the network step size, c is the center point, and R is the search area radius.

4. The target tracking method using feature fusion and sparse attention network according to claim 1, characterized in that: In S7, exemplar and f instance The Sparse-Transformer is used to perform feature enhancement and fusion processing. The features of the two branches are enhanced through the self-context module and the cross-feature module. The self-context module uses sparse multi-head self-attention in the form of residual to adaptively integrate information from different positions of the feature map. The attention mechanism cannot distinguish the position information of the input feature sequence; the spatial position encoding process is introduced into the input X; then a sine function is used to generate the spatial position encoding, which is expressed as follows: X EnC =X+SMultiHead(X+P X , X+P X , X), where P X is the spatial position encoding information calculated using the sine function, X EnC is the output of the self-context enhancement module, SMultiHead is the sparse multi-head attention mechanism; the cross-feature module uses the residual form of sparse multi-head cross-attention to fuse the feature vectors of the two inputs; spatial position encoding is used for the cross-feature module: Where X q is the input of the module branch, P q For the corresponding X q The spatial position encoding of X kv is the input from another branch; P kv For X kv Spatial encoding of coordinates; X CF is the output of the self-context enhancement module.

5. The target tracking method using feature fusion and sparse attention network according to claim 4, characterized in that: Using the sparse multi-head self-attention algorithm, by selectively allocating attention, the sparse attention mechanism can focus most of the attention on the key areas of the target. In the original multi-head attention calculation, each pixel value of the attention feature is calculated by all pixel values ​​of the input feature, which makes the edge area of ​​the foreground blurred; in SMultiHead, each pixel value of the attention feature is determined only by the K most similar pixel values, and the Softmax function is used to normalize the K largest elements in each row of the similarity matrix; for other elements, they are set to 0.

6. The target tracking method using feature fusion and sparse attention network according to claim 1, characterized in that: In S8, in the feature fusion part, the cross-feature modules can enhance the features of the two branches respectively, and build a feature mutual enhancement and aggregation network; by repeating the process of feature mutual enhancement and aggregation network multiple times, discriminative features are obtained; The cross-feature module can also measure the similarity between two branches and obtain a response map; feature mutual enhancement and aggregation networks are used to reduce the amount of computation.

7. The target tracking method using feature fusion and sparse attention network according to claim 1, characterized in that: In S9, the prediction head structure contains two branches: a classification branch and a regression branch; each branch consists of a three-layer perceptron with a hidden layer dimension of d and a ReLU activation function; the prediction head structure branch is used to perform two main tasks for each vector in the feature map: foreground and background classification prediction and coordinate prediction of the search area size.

Images