Unmanned aerial vehicle target tracking method based on feature fusion and online template updating

Abstract

The invention discloses an unmanned aerial vehicle target tracking method based on feature fusion and online template updating, and the method comprises the following steps: constructing a deep network model based on a Resnet residual network and cavity convolution, and effectively extracting target features and enhancing the effective receptive field of the features; meanwhile, detail features of a shallow network and semantic features of a deep network are efficiently fused, and the expression ability of the features is enhanced; a template branch and a detection branch of a Siamese structure are formed by a ResNet-50 network and a target fusion network, two branch feature maps are sent to a cascade cross-correlation module, and a target position is determined; and constructing a template library, calculating a response graph score, and carrying out online updating on template branches according to a threshold value. According to the method, the tracking success rate and the tracking accuracy both reach a high level, and the target tracking performance of the unmanned aerial vehicle is effectively improved.

Description

Technical Field

[0001] The present invention relates to a video target tracking method, which integrates advanced technologies in many fields such as image processing, artificial intelligence and computer, and in particular to a UAV video target tracking algorithm based on feature fusion and online template updating. Background Art

[0002] Drones are widely used in many fields, such as disaster relief, power inspection, military reconnaissance and environmental monitoring, due to their small size, high flexibility, good cost-effectiveness and no risk of casualties. With the continuous development of science and technology, as one of the main technologies in the application of drones, target tracking technology is also making continuous progress. However, due to the continuous changes in the flight posture and shooting angle of drones during flight, problems such as target occlusion, background clutter, and changes in the appearance of the tracked target occur, which in turn affects the performance of the tracking algorithm. Therefore, improving the target feature expression ability and solving the occlusion problem in target tracking technology are the key to enhancing the performance of drones. In recent years, deep networks have shown their powerful performance in the field of computer vision, and target tracking algorithms based on deep learning have emerged, which have significantly improved tracking accuracy and tracking success rate. The SiamFC algorithm based on the twin network proposed by Bertinetto has been widely studied because of its simple structure and easy expansion, but its robustness to target occlusion and target appearance changes is not good. SiamRPN proposed by Bo et al. adopts the anchor-based idea and introduces the regional proposal network idea in the field of target detection into target tracking, which can improve the network recognition accuracy, but the tracking effect is not ideal for scenes with overlapping targets and cluttered backgrounds. Although the above-mentioned deep learning-based target tracking algorithm has improved the target tracking performance overall, its processing capabilities for target appearance changes and target occlusion characteristics need to be improved, making it difficult to be effectively applied in UAV vision tasks.

[0003] In summary, this paper proposes a feature fusion and online template updating UAV target tracking algorithm based on Siamese structure. First, combined with the gradient optimization characteristics of the Resnet-50 residual network (Residual Network, ResNet) structure, the dilated convolution is applied to the network layer to increase the receptive field while keeping the resolution of the feature map output by the last three layers of the network consistent, and the output of the last three layers is spliced ​​together to fuse the deep and shallow features of the neural network to achieve efficient fusion of deep semantic features and shallow detail features, which can effectively extract target features and enhance their feature expression. Then the template branch and search branch feature map output by the network are input into the cascade cross-correlation module, and the feature maps of the two branches are used to perform deep and fine-grained positioning of the tracking target through two cross-correlation operations, and then the obtained score response map is sent to the classification branch and regression branch to obtain the final tracking result. Finally, a template memory is established, and the weight is calculated according to the score of the template response map to obtain a template that meets the current target state, and the tracking task of the next frame is performed at the same time until the tracking task is completed. The simulation experiment results show that the algorithm has achieved a high level in terms of success rate and accuracy, and can effectively improve the tracking performance of UAVs in the face of target occlusion and target deformation.

[0004] The shortcomings of existing methods: On the one hand, the tracking algorithm based on deep learning has greatly improved the tracking accuracy in simple scenes, but in complex environments, the UAV’s ability to express the characteristics of the tracking target is not very strong, which can easily reduce the performance of UAV target tracking. On the other hand, the classic tracking algorithm has a good tracking effect on specific targets and can achieve real-time performance, but it is not generalizable and cannot be applied to various occasions. It is not good at solving the problems of long-term tracking, interference from similar objects, and cluttered backgrounds, which can easily lead to tracking failures. Summary of the invention

[0005] Aiming at the problems that targets in drone videos are prone to morphological changes, occlusion, and cluttered backgrounds, the present invention designs a drone target tracking network based on the Siamese structure, and aims at the problems that targets in drone videos are prone to morphological changes, occlusion, and cluttered backgrounds. In combination with the Siamese structure, a network model based on feature fusion and template updating is constructed, and multi-layer features extracted by the network model are fused at multiple scales to obtain target features with stronger expression capabilities. At the same time, a template library is established to update the templates, which greatly improves the adaptability of the network to target changes, improves the prediction accuracy and precision, and reduces the influence of the surrounding background on the network performance.

[0006] To achieve the above object, the present invention proposes a UAV video target tracking algorithm based on feature fusion and template update network, which includes the following steps:

[0007] S1: Select a large number of image sequences related to the target to be tracked collected by drones and pre-train the feature fusion network. The training method is as follows:

[0008] S1.1: Construct a five-layer ResNet-50 residual network; remove the downsampling operation of the last two convolution blocks of the deep network layer, add dilated convolutions to the network to improve the receptive field. Set the stride in the fourth and fifth convolution blocks to 1, and set the expansion rate of the fourth convolution block to 2 and the fifth to 4;

[0009] S1.2: The feature maps C3, C4, and C5 with different expansion rates output by the last three layers of the Resnet-50 deep network are fused together according to the channel dimension to fuse the fine-grained information of the shallow features with the abstract semantic information of the deep features;

[0010] S2: Designed a template branch and search branch consisting of a Siamese structure consisting of Resnet-50 and a target fusion network, trained the network with a drone target tracking dataset, and constructed an effective loss function to distinguish between positive and negative samples at the location points of the response map;

[0011] S3: The mean loss of all points is used as the overall loss for the Siamese structural response diagram, the overall loss of the network is calculated using the loss mean, and the SGD algorithm is used to optimize the network weights according to this mean;

[0012] S4: convert the video into image frames after preprocessing and input the system, and read the first frame;

[0013] S5: Use the Resnet-50 backbone network of the template branch to extract the features of the target image in the first frame, and fuse the outputs of the last three layers to obtain the template branch features f exemplar ;

[0014] S6: Read the next frame and use the search branch to extract the target features of the current frame to obtain the search branch feature f instance ;

[0015] S7: The template branch feature and the search branch feature f are respectively exemplar and f instance The features of the two branches are fused through the cascade cross-correlation module. The target response map is obtained through two cross-correlation calculations and sent to the classification branch and regression branch to determine the target position.

[0016] S8: In each frame of the video stream, take n windows around the target position predicted by the previous frame, perform classification and regression calculation using the trained classifier through feature extraction, dimensionality reduction, multi-scale transformation, etc. The window with the largest score is the tracking target;

[0017] S9: Build a template library, calculate the weight according to the score of the template response graph and obtain the template that meets the current target state, and determine whether to replace the template graph of the template branch according to the threshold. If the score is higher than the threshold, replace it, otherwise do not update the template;

[0018] S10: Repeat S4-S8 until all drone video frames are processed, and obtain and output the video tracking results.

[0019] Compared with the prior art, the present invention aims at the problems that the targets in drone videos are prone to morphological changes, easy to be occluded, and have cluttered backgrounds. It is necessary to fully extract the target features, and a network model based on feature fusion and template updating is constructed in combination with the Siamese structure. Combining the gradient optimization characteristics of the residual network structure and the receptive field enhancement characteristics of the target features with the Resnet deep network and the dilated convolution, the target features are effectively extracted and their semantic and detail information is enhanced. A cascaded cross-correlation feature fusion module is proposed, which can efficiently fuse the features of the two branches, and an online template update strategy is proposed to establish a template library to enhance the target recognition ability in occluded scenes. Simulation experiments show that the proposed algorithm has good performance in scenes such as perspective changes, occlusions, deformations, and scale changes, and can process these videos well. For test videos in other scenes, the performance of the proposed algorithm is also good, which fully verifies that the proposed algorithm has excellent overall performance in tracking accuracy, stability, and robustness. The tracking accuracy of the feature fusion and template updating methods proposed in this paper reached 0.831, which was 2.3% and 3.4% higher than the accuracy of target tracking directly using Conv5 features and not using the template updating method, respectively; the tracking success rate reached 0.625, which was 2.1% and 2.3% higher than the accuracy of target tracking directly using Conv5 features and not using the template updating method, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 Flow chart of a UAV video target tracking algorithm method based on Siamese structure according to an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a network model based on feature fusion and template updating according to an embodiment of the present invention;

[0023] Figure 3 This is a structural diagram of a cascaded cross-correlation module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or components having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0025] According to the present invention, the UAV video target tracking method based on feature fusion and template updating network includes the following steps:

[0026] S1: Select a large number of image sequences related to the target to be tracked collected by drones and pre-train the Resnet-50 network. The training method is as follows:

[0027] S1.1: Construct a Siamese network with five layers of ResNet residual network and dilated convolution operations; remove the downsampling operations of the last two convolution blocks in the deep network layer, add dilated convolutions to the network to improve the receptive field. Set the stride in the fourth and fifth convolution blocks to 1, and set the expansion rate of the fourth convolution block to 2 and the fifth to 4;

[0028] ResNet residual network has five layers, F = W2σ(W1x), y = F(x,{W i})+x, where σ represents the nonlinear function ReLU, and then through a shortcut and the second ReLU, the output y is obtained. Considering the computational cost, the residual block is optimized. The first convolution layer is 1x1, the middle convolution layer is 3x3, and the third convolution layer is 1x1, which not only maintains accuracy but also reduces the amount of computation. Considering that ResNet-50 using continuous convolution steps can learn more and more abstract feature representations, but at the same time reduces the feature resolution, the downsampling operation is removed from the last two convolution blocks, and atrous convolution is used to improve the receptive field.

[0029] The network adopts the shortcut structure in ResNet, so that the gradient can be well transferred to the shallow layer, reducing the training burden of the deep network. Dilated Convolution adds a new parameter in the standard convolution layer - dilation rate, which determines the distance between pixels when the convolution kernel processes data, expands the convolution kernel to a specified scale, and fills the unoccupied pixel area in the original convolution kernel with 0, so that the receptive field of the feature map can be increased without increasing the amount of additional calculations. In order to solve the problem of network model degradation caused by gradient disappearance or explosion, the ResNet residual network structure is proposed. The shortcut connection method effectively reduces the number of layers that the network gradient passes through during propagation, so that the problem of loss value explosion and disappearance is alleviated, the network model training speed is accelerated, and the network model expression ability is improved. Therefore, combining the gradient optimization characteristics of ResNet and the receptive field enhancement characteristics of dilated convolution, a Resnet-50 deep network is constructed to extract image features of tracking targets.

[0030] S1.2: The feature maps C3, C4, and C5 with different expansion rates output by the last three layers of the Resnet-50 deep network are fused together according to the channel dimension to fuse the fine-grained information of the shallow features with the abstract semantic information of the deep features;

[0031] S2: Designed a template branch and search branch consisting of a Siamese structure consisting of Resnet-50 and a target fusion network, trained the network with a drone target tracking dataset, and constructed an effective loss function to distinguish between positive and negative samples at the location points of the response map;

[0032] S3: The mean loss of all points is used as the overall loss for the Siamese structural response map, and the labeled image is input. The loss is calculated using the mean loss and the SGD algorithm is used to optimize the network weights.

[0033] The network has two branches, such as Figure 2 As shown in Figure 2, the upper part of the network is the template branch, and the lower part is the search branch. The similarity function f(z,x) is learned in space. Compare the target template image z and the current frame image x, and find the sample that is most similar to the target template image as the prediction target.

[0034] In order to construct an effective loss function, the loss function for each point in the response graph is l(y,v)=log(1+exp(-yv)). In the formula, v is the true value of each point, and y∈{+1,-1} is the label corresponding to this point. For the overall loss of the corresponding graph, the loss mean of all points is used, that is: Where u∈D is the position in the response graph. y[u] is expressed as: In the formula, k is the network step size, c is the center point, and R is the search area radius.

[0035] The parameters θ of the convolutional network are optimized using the SGD (Stochastic Gradient Descent) algorithm. The network is divided into two inputs, one input target image, size 127×127×3, and the other input current frame search area image, size 255×255×3, the search area is 4 times the size of the previous frame target, and a cosine window is added to the response map to penalize the maximum displacement.

[0036] S4: convert the video into image frames after preprocessing and input the system, and read the first frame;

[0037] S5: Use the Resnet-50 backbone network of the template branch to extract the features of the target image in the first frame, and fuse the outputs of the last three layers to obtain the template features f exemplar ;

[0038] S6: Read the next frame and use the search branch to extract the target feature f of the current frame instance ;

[0039] S7: Separately set the template branch and search branch features f exemplar and f instance The features of the two branches are fused through the cascade cross-correlation module. The target response map is obtained through two cross-correlation calculations and sent to the classification branch and regression branch to determine the target position.

[0040] Cascaded cross-correlation modules such as Figure 3 As shown in the figure. The deep cross-correlation operation is performed on each channel of the feature map to obtain different semantic information between channels and achieve the first rough positioning of the target. Then the target features are refined, different feature information is learned, and a more fine-grained feature representation is obtained. In the second cross-correlation operation, the pixel-level global cross-correlation operation is used to calculate the similarity of each pixel in the search feature and the global template feature to obtain high-quality spatial and channel feature information representation, which to a certain extent compensates for the algorithm performance degradation caused by the loss of spatial information, and has a strong ability to suppress background interference, which can achieve more accurate target positioning, thereby improving the performance of the drone target tracking algorithm.

[0041] S8: In each frame of the video stream, take n windows around the target position predicted by the previous frame, perform classification and regression calculation using the trained classifier through feature extraction, dimensionality reduction, multi-scale transformation, etc. The window with the largest score is the tracking target;

[0042] S9: Build a template library, calculate the weight according to the score of the template response graph and obtain the template that meets the current target state, and determine whether to replace the template graph of the template branch according to the threshold. If the score is higher than the threshold, replace it, otherwise do not update the template;

[0043] According to the score response graph, the average peak correlation energy (APCE) and the maximum response value Fmax of the current frame are calculated. The calculation expression is as follows: Among them, F max Indicates the maximum value of the response graph, F min Indicates the minimum value of the response graph, F w,h Represents the response value of the wth row and the hth column. The template update strategy is executed every five frames. The templates of the three consecutive frames before the current frame are extracted from the template memory library, and different weights are assigned to each frame according to its APCE value. The updated template is fused with the template of the first frame. The fusion formula is: in, is the weight coefficient calculated based on the APCE value of the frame. x0 is the first frame template, x j is the j-th frame template in three consecutive frames, and τ0 is a constant coefficient.

[0044] S10: Repeat S5-S9 until all drone video frames are processed, and obtain and output the video tracking results.

[0045] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A UAV video target tracking method based on feature fusion and online template updating network, characterized by: The method comprises the following steps, S1: Select the image sequence related to the target to be tracked collected by the drone and pre-train the feature fusion network. The training method is as follows: S2: Design a template branch and a search branch composed of a Siamese structure consisting of Resnet-50 and a target fusion network, and construct an effective loss function to distinguish positive and negative samples at the location points of the response map; S3: The mean loss of all points is used as the overall loss for the Siamese structural response graph, the loss is calculated using the mean loss, and the SGD method is used to optimize the network weights; S4: convert the video into image frames after preprocessing and input the system, and read the first frame; S5: Use the Resnet-50 backbone network of the template branch to extract the features of the target image in the first frame, and fuse the outputs of the last three layers to obtain the template branch features f exemplar ; S6: Read the next frame and use the search branch to extract the target features of the current frame to obtain the search branch feature f search ; S7: The template branch feature and the search branch feature f are respectively exemplar and f search The features of the two branches are fused through the cascade cross-correlation module. The target response map is obtained through two cross-correlation calculations and sent to the classification branch and regression branch to determine the target position. S8: In each frame of the video stream, take n windows around the target position predicted by the previous frame, perform classification and regression calculation using the trained classifier through feature extraction, dimensionality reduction, and multi-scale transformation. The window with the largest score is the tracking target. S9: Build a template library, calculate the weight according to the score of the template response graph and obtain the template that meets the current target state, and determine whether to replace the template graph of the template branch according to the threshold. If the score is higher than the threshold, replace it, otherwise do not update the template; S10: Repeat S5-S9 until all drone video frames are processed, and obtain and output the video tracking results.

2. The target tracking method based on feature fusion and online template updating network according to claim 1 is characterized in that: S1 includes: S1.1: Construct a Siamese network with five layers of ResNet residual network and dilated convolution operations; remove the downsampling operations of the last two convolution blocks in the deep network layer, add dilated convolutions to the network to improve the receptive field; set the stride in the fourth and fifth convolution blocks to 1, and set the expansion rate of the fourth convolution block to 2 and the fifth to 4; The ResNet residual network has five layers, among which ReLU represents the nonlinear function, and then a shortcut and the second ReLU are used to obtain the output y. Considering the computational cost, the residual block is optimized. The first convolution layer is 1x1, the middle convolution layer is 3x3, and the third convolution layer is 1x1, which not only maintains the accuracy but also reduces the amount of computation. The downsampling operation is removed from the last two convolution blocks, and atrous convolution is used to improve the receptive field. The network adopts the direct connection structure in ResNet; the dilated convolution adds a new parameter - the dilation rate - to the standard convolution layer. The dilation determines the distance between pixels when the convolution kernel processes data, expands the convolution kernel to a specified scale, and fills the unoccupied pixel area in the original convolution kernel with 0; the ResNet residual network structure is proposed, which reduces the number of layers passed by the network gradient propagation through shortcut connections.

3. The target tracking method based on feature fusion and online template updating network according to claim 1 is characterized in that: The network in S3 has 2 branches; The upper part of the network is the template branch, and the lower part is the search branch. Compare the target template image z and the current frame image x, and find the sample that is most similar to the target template image as the prediction target; To construct the loss function, the loss function for each point in the response graph is l9y,v)=log(1+exp(-yv)); where v is the true value of each point, and y∈{+1,-1} is the label corresponding to this point; the overall loss of the corresponding graph is the mean loss of all points, that is: Where u∈D is the position in the response map, D is the response map area; y[u] is expressed as: In the formula, k is the network step size, c is the center point, and R is the search area radius; The parameters θ of the convolutional network are optimized using the SGD (stochastic gradient descent) optimization method to calculate the loss; the network is divided into 2 inputs, one input target image with a size of 127×127×3, and the other input current frame search area image with a size of 255×255×3. The search area is 4 times the target size of the previous frame, and a cosine window is added to the response map to penalize the maximum displacement.

4. The target tracking method based on feature fusion and online template updating network according to claim 1 is characterized in that: In S7, the cascaded cross-correlation module consists of two cross-correlation operations; the deep cross-correlation operation is performed on each channel of the feature map to obtain different semantic information between channels and achieve the first rough positioning of the target; then the target features are refined, different feature information is learned, and fine-grained feature representation is obtained; in the second cross-correlation operation, the pixel-level global cross-correlation operation is used to calculate the similarity of each pixel in the search feature and the global template feature to obtain high-quality spatial and channel feature information representation.

5. The target tracking method based on feature fusion and online template updating network according to claim 1 is characterized in that: In S9, the average peak correlation energy APCE and the maximum response value Fmax of the current frame are calculated according to the score response graph, and the calculation expression is as follows: Among them, F max Indicates the maximum value of the response graph, F min Indicates the minimum value of the response graph, F w,h represents the response value of the wth row and the hth column; the template update strategy is executed every five frames; the templates of the three consecutive frames before the current frame are extracted from the template memory library, and different weights are given to each frame according to its APCE value, and the updated template is obtained by fusion with the template of the first frame. The fusion formula is: in, is the weight coefficient calculated based on the APCE value of the frame, x0 is the first frame template, x j is the j-th frame template in three consecutive frames, and τ0 is a constant coefficient.

Images