Unmanned aerial vehicle target tracking method based on feature fusion and online template updating

Abstract

The application discloses a UAV target tracking method based on feature fusion and online template updating, and comprises the following steps: a deep network model is constructed based on a Resnet residual network and a hollow convolution, target features can be effectively extracted, the effective receptive field of the features is enhanced, the detailed features of a shallow network and the semantic features of a deep network are efficiently fused, and the expression capability of the features is enhanced; a template branch and a detection branch in a Siamese structure are composed of a ResNet-50 network and a target fusion network, feature maps of the two branches are sent into a cascaded cross-correlation module, and the target position is determined; a template library is constructed, a response map score is calculated, and the template branch is updated online according to a threshold value. The application reaches a high level in terms of tracking success rate and accuracy, and effectively improves the UAV target tracking performance.

Description

Technical Field

[0001] The present invention relates to a video target tracking method, which integrates advanced technologies in many fields such as image processing, artificial intelligence and computers, and in particular to a UAV video target tracking algorithm based on feature fusion and online template updating. Background Art

[0002] Due to their small size, high flexibility, cost-effectiveness, and lack of risk of casualties, drones are widely used in many fields, such as disaster relief, power inspections, military reconnaissance, and environmental monitoring. With the continuous advancement of technology, target tracking, a key technology in drone applications, has also seen continuous progress. However, due to the constant changes in flight attitude and camera angle during flight, drones can experience issues such as target occlusion, background clutter, and changes in the appearance of the tracked target, which in turn affect the performance of tracking algorithms. Therefore, improving target feature representation and resolving occlusion issues in target tracking technology are key to enhancing drone performance. In recent years, deep networks have demonstrated their powerful performance in computer vision, leading to the emergence of deep learning-based target tracking algorithms, which have significantly improved tracking accuracy and success rate. The SiamFC algorithm based on a Siamese network proposed by Bertinetto has been widely studied due to its simple structure and ease of scalability. However, it is not robust to target occlusion and appearance changes. The SiamRPN proposed by Bo et al. uses an anchor-based approach, incorporating the concept of region proposal networks (RPNs) from the field of target detection into target tracking. This improves network recognition accuracy, but the tracking effect is less than ideal in scenes with overlapping targets and cluttered backgrounds. While these deep learning-based target tracking algorithms have improved overall target tracking performance, their ability to handle changes in target appearance and occlusion remains to be improved, making them difficult to effectively apply to drone vision tasks.

[0003] In summary, this paper proposes a Siamese-based feature fusion and online template updating algorithm for drone target tracking. First, leveraging the gradient optimization characteristics of the ResNet-50 residual network (ResNet) architecture, dilated convolutions are applied to the network layers. This increases the receptive field while maintaining consistent resolution of the feature maps output by the last three layers. The outputs of these three layers are then concatenated, fusing deep and shallow neural network features. This effectively integrates deep semantic features with shallow detail features, effectively extracting target features and enhancing their representation. The template and search branch feature maps output by the network are then fed into a cascaded cross-correlation module. Two cross-correlation operations are performed on the feature maps of these two branches to perform deep, fine-grained localization of the tracked target. The resulting score response map is then fed into the classification and regression branches to obtain the final tracking result. Finally, a template memory is established, and weights are calculated based on the scores of the template response maps to obtain a template that matches the current target state. Tracking is then performed on the next frame until the tracking task is completed. The simulation experiment results show that this algorithm has achieved a high level in terms of success rate and accuracy, and can effectively improve the tracking performance of UAVs in the face of target occlusion and target deformation.

[0004] Existing methods have shortcomings: On the one hand, deep learning-based tracking algorithms have greatly improved tracking accuracy in simple scenarios. However, in complex environments, drones are not very good at expressing the characteristics of the tracked target, which can easily reduce the performance of drone target tracking. On the other hand, classic tracking algorithms are very effective for tracking specific targets and can achieve real-time performance, but they lack generalization and cannot be applied to various situations. They also have difficulty in solving problems such as long-term tracking, interference from similar objects, and cluttered backgrounds, which can easily lead to tracking failures. Summary of the Invention

[0005] To address the problems of targets in drone videos being prone to morphological changes, being easily occluded, and having cluttered backgrounds, the present invention designs a drone target tracking network based on the Siamese structure. Furthermore, in combination with the Siamese structure, a network model based on feature fusion and template updating is constructed. The multi-layer features extracted by the network model are fused at multiple scales to obtain target features with stronger expressive power. At the same time, a template library is established to update the templates, which greatly improves the network's adaptability to target changes, enhances prediction accuracy and precision, and reduces the impact of the surrounding background on network performance.

[0006] To achieve the above objectives, the present invention proposes a UAV video target tracking algorithm based on feature fusion and template update network, which includes the following steps:

[0007] S1: Select a large number of image sequences related to the target to be tracked collected by drones and pre-train the feature fusion network. The training method is as follows:

[0008] S1.1: Build a five-layer ResNet-50 residual network; remove the downsampling operations of the last two convolutional blocks in the deep network layer and add dilated convolutions to the network to improve the receptive field. Set the stride of the fourth and fifth convolutional blocks to 1, and the expansion rate of the fourth convolutional block to 2 and the fifth to 4.

[0009] S1.2: The feature maps C3, C4, and C5 with different expansion rates output by the last three layers of the Resnet-50 deep network are fused together along the channel dimension to fuse the fine-grained information of shallow features with the abstract semantic information of deep features.

[0010] S2: We designed a Siamese-structured template branch and search branch consisting of a Resnet-50 and target fusion network. We trained the network using a drone target tracking dataset and constructed an effective loss function to distinguish between positive and negative samples at the response map locations.

[0011] S3: Use the loss mean of all points as the overall loss for the Siamese structure response diagram, use the loss mean to calculate the overall loss of the network and use the SGD algorithm to optimize the network weights based on this mean;

[0012] S4: Convert the video into image frames after preprocessing and input the system, and read the first frame;

[0013] S5: Use the Resnet-50 backbone network of the template branch to extract the target image features of the first frame, and fuse the outputs of the last three layers to obtain the template branch features f exemplar ;

[0014] S6: Read the next frame and use the search branch to extract the target features of the current frame to obtain the search branch feature f instance ;

[0015] S7: Separately transform the template branch feature and the search branch feature f exemplar and f instance The features of the two branches are fed into the cascade cross-correlation module to fuse them. The target response map is obtained through two cross-correlation calculations and fed into the classification branch and regression branch to determine the target position.

[0016] S8: In each frame of the video stream, n windows are taken around the target position predicted by the previous frame. Through feature extraction, dimensionality reduction, multi-scale transformation, etc., the trained classifier is used for classification and regression calculation. The window with the largest score is the tracking target.

[0017] S9: Build a template library, calculate the weight according to the score of the template response graph and obtain the template that meets the current target state, and determine whether to replace the template graph of the template branch according to the threshold. If the score is higher than the threshold, replace it, otherwise do not update the template;

[0018] S10: Repeat S4-S8 until all drone video frames are processed, and obtain and output the video tracking results.

[0019] Compared with the existing technology, the present invention addresses the problems of targets in drone videos being prone to morphological changes, being easily occluded, and having cluttered backgrounds. This requires sufficient extraction of target features, and combines the Siamese structure to construct a network model based on feature fusion and template updating. Combining the gradient optimization characteristics of the residual network structure with the dilated convolution of the Resnet deep network and the receptive field enhancement characteristics of the target features, the target features are effectively extracted and their semantic and detail information is enhanced. A cascaded cross-correlation feature fusion module is proposed, which can efficiently fuse the features of the two branches. An online template update strategy is also proposed, and a template library is established to enhance the target recognition capability in occluded scenes. Simulation experiments show that the proposed algorithm has good performance in scenes with perspective changes, occlusions, deformations, scale changes, etc., and can process these videos well. The performance of the proposed algorithm is also good for test videos in other scenes, which fully verifies that the proposed algorithm has excellent overall performance in tracking accuracy, stability, and robustness. The tracking accuracy of the feature fusion and template updating methods proposed in this paper reached 0.831, which is 2.3% and 3.4% higher than the accuracy of target tracking directly using Conv5 features and not using the template updating method, respectively; the tracking success rate reached 0.625, which is 2.1% and 2.3% higher than the accuracy of target tracking directly using Conv5 features and not using the template updating method, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 Flowchart of the UAV video target tracking algorithm method based on Siamese structure according to an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a network model based on feature fusion and template updating according to an embodiment of the present invention;

[0023] Figure 3 This is a structural diagram of a cascaded cross-correlation module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or components having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0025] The method for tracking a target in a UAV video based on a feature fusion and template updating network according to the present invention includes the following steps:

[0026] S1: Select a large number of image sequences related to the target to be tracked collected by drones and pre-train the Resnet-50 network. The training method is as follows:

[0027] S1.1: Construct a Siamese network using a five-layer ResNet residual network and dilated convolution operations. Remove the downsampling operations from the last two convolution blocks in the deep network layer and add dilated convolutions to the network to improve the receptive field. Set the stride in the fourth and fifth convolution blocks to 1, and the expansion rate of the fourth convolution block to 2 and the fifth to 4.

[0028] ResNet residual network has five layers, F = W2σ(W1x), y = F(x, {W i})+x, where σ represents the nonlinear function ReLU. Then, through a shortcut and a second ReLU, the output y is obtained. Considering computational cost, the residual blocks were optimized: the first convolutional layer is 1x1, the middle convolutional layer is 3x3, and the third convolutional layer is 1x1. This maintains accuracy while reducing computational effort. Considering that ResNet-50 using a continuous convolutional stride can learn increasingly abstract feature representations but also reduces feature resolution, the downsampling operation in the last two convolutional blocks was removed, and atrous convolution was used to improve the receptive field.

[0029] The network utilizes the shortcut architecture of ResNet, enabling gradients to be efficiently propagated to shallow layers, reducing the training burden of deeper networks. Dilated convolution adds a new parameter—the dilation rate—to the standard convolutional layer. This parameter determines the distance between pixels in the convolution kernel when processing data. This parameter expands the convolution kernel to a specified scale and fills the unoccupied pixel areas of the original convolution kernel with zeros, thereby increasing the receptive field of the feature map without increasing the amount of additional computation. To address the problem of network model degradation caused by vanishing or exploding gradients, the ResNet residual network architecture was proposed. By using shortcut connections, it effectively reduces the number of layers through which the network gradient propagates, alleviating the problems of exploding and vanishing loss values, accelerating network model training, and improving the network's expressiveness. Therefore, combining the gradient optimization characteristics of ResNet with the receptive field enhancement properties of dilated convolution, a ResNet-50 deep network was constructed to extract image features for tracking targets.

[0030] S1.2: The feature maps C3, C4, and C5 with different expansion rates output by the last three layers of the Resnet-50 deep network are fused together along the channel dimension to fuse the fine-grained information of shallow features with the abstract semantic information of deep features.

[0031] S2: We designed a Siamese-structured template branch and search branch consisting of a Resnet-50 and target fusion network. We trained the network using a drone target tracking dataset and constructed an effective loss function to distinguish between positive and negative samples at the response map locations.

[0032] S3: Use the loss mean of all points as the overall loss for the Siamese structure response map, input the labeled image, use the loss mean to calculate the loss, and use the SGD algorithm to optimize the network weights;

[0033] The network has two branches, such as Figure 2 As shown in the figure. The upper part of the network is the template branch, and the lower part is the search branch. It learns the similarity function f(z,x) in space Compare the target template image z and the current frame image x, and find the sample that is most similar to the target template image as the prediction target.

[0034] To construct an effective loss function, the loss function for each point in the response graph is l(y,v)=log(1+exp(-yv)). In the formula, v is the true value of each point, and y∈{+1,-1} is the label corresponding to this point. The overall loss of the corresponding graph is the mean loss of all points, that is: Where u∈D is the position in the response graph. y[u] is expressed as: Where k is the network step size, c is the center point, and R is the search area radius.

[0035] The convolutional network parameters θ are optimized using the stochastic gradient descent (SGD) optimization algorithm. The network takes two inputs: a target image of size 127×127×3 and a search region image of the current frame of size 255×255×3. The search region is four times the size of the target image in the previous frame. A cosine window is added to the response image to penalize maximum displacement.

[0036] S4: Convert the video into image frames after preprocessing and input the system, and read the first frame;

[0037] S5: Use the Resnet-50 backbone network of the template branch to extract the target image features of the first frame, and fuse the outputs of the last three layers to obtain the template features f exemplar ;

[0038] S6: Read the next frame and use the search branch to extract the target feature f of the current frame instance ;

[0039] S7: Separately set the template branch and search branch features f exemplar and f instance The features of the two branches are fed into the cascade cross-correlation module to fuse them. The target response map is obtained through two cross-correlation calculations and fed into the classification branch and regression branch to determine the target position.

[0040] Cascaded cross-correlation modules such as Figure 3 As shown in the figure. The deep cross-correlation operation is performed on each channel of the feature map to obtain different semantic information between channels, achieving the first rough positioning of the target. The target features are then refined, different feature information is learned, and a more fine-grained feature representation is obtained. In the second cross-correlation operation, a pixel-level global cross-correlation operation is used to calculate the similarity of each pixel in the search feature and the global template feature, obtaining high-quality spatial and channel feature information representation. This to some extent compensates for the algorithm performance degradation caused by the loss of spatial information, and has a strong ability to suppress background interference, which can achieve more accurate target positioning, thereby improving the performance of the UAV target tracking algorithm.

[0041] S8: In each frame of the video stream, n windows are taken around the target position predicted by the previous frame. Through feature extraction, dimensionality reduction, multi-scale transformation, etc., the trained classifier is used for classification and regression calculation. The window with the largest score is the tracking target.

[0042] S9: Build a template library, calculate the weight according to the score of the template response graph and obtain the template that meets the current target state, and determine whether to replace the template graph of the template branch according to the threshold. If the score is higher than the threshold, replace it, otherwise do not update the template;

[0043] According to the score response graph, the average peak correlation energy (APCE) and the maximum response value Fmax of the current frame are calculated. The calculation expressions are as follows: Among them, F max Indicates the maximum value of the response graph, F min Indicates the minimum value of the response graph, F w,h Represents the response value of the wth row and hth column. The template update strategy is executed every five frames. The templates of the three consecutive frames before the current frame are extracted from the template memory library. Different weights are assigned to each frame according to its APCE value. The updated template is fused with the template of the first frame. The fusion formula is: in, The weight coefficient x0 is calculated based on the APCE value of the frame. It is the first frame template, x j is the j-th frame template in three consecutive frames, and τ0 is a constant coefficient.

[0044] S10: Repeat S5-S9 until all drone video frames are processed, and obtain and output the video tracking results.

[0045] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A UAV video target tracking method based on feature fusion and online template updating network, characterized by: The method comprises the following steps, S1: Select the image sequence related to the target to be tracked collected by the UAV and pre-train the feature fusion network. The training method is as follows: S2: Design a template branch and a search branch composed of a Siamese structure consisting of Resnet-50 and a target fusion network, and construct an effective loss function to distinguish between positive and negative samples at the response map position points; S3: Use the loss mean of all points as the overall loss for the Siamese structure response graph, use the loss mean to calculate the loss and use the SGD method to optimize the network weights; S4: convert the video into image frames after preprocessing and input the system, and read the first frame; S5: Use the Resnet-50 backbone network of the template branch to extract the target image features of the first frame, and fuse the outputs of the last three layers to obtain the template branch features f exemplar ; S6: Read the next frame and use the search branch to extract the target features of the current frame to obtain the search branch feature f search ; S7: Separately transform the template branch feature and the search branch feature f exemplar and f search The features of the two branches are fed into the cascade cross-correlation module to fuse them. The target response map is obtained through two cross-correlation calculations and fed into the classification branch and regression branch to determine the target position. S8: In each frame of the video stream, n windows are taken around the target position predicted by the previous frame. Through feature extraction, dimensionality reduction, and multi-scale transformation, classification and regression calculation are performed using the trained classifier. The window with the largest score is the tracking target. S9: Build a template library, calculate the weight based on the score of the template response graph and obtain a template that meets the current target state. Determine whether to replace the template graph of the template branch based on the threshold. If the score is higher than the threshold, replace it; otherwise, do not update the template. In S9, calculate the average peak correlation energy APCE and maximum response value Fmax of the current frame based on the score response graph. The calculation expression is as follows: Among them, F max Indicates the maximum value of the response graph, F min Indicates the minimum value of the response graph, F w,h represents the response value of the wth row and the hth column; the template update strategy is executed every five frames; the templates of the three consecutive frames before the current frame are extracted from the template memory library, and different weights are assigned to each frame according to its APCE value. The updated template is fused with the template of the first frame. The fusion formula is: in, is the weight coefficient calculated based on the APCE value of the frame, x0 is the first frame template, x j is the j-th frame template in three consecutive frames, τ0 is a constant coefficient; S10: Repeat S5-S9 until all drone video frames are processed, and obtain and output the video tracking results.

2. The method for tracking target in UAV video based on feature fusion and online template updating network according to claim 1 is characterized in that: S1 includes: S1.1: Construct a Siamese network using a five-layer ResNet residual network and dilated convolution operations. Remove the downsampling operations from the last two convolution blocks in the deep network layer and add dilated convolutions to the network to improve the receptive field. Set the stride in the fourth and fifth convolution blocks to 1, and the expansion rate of the fourth convolution block to 2 and the fifth to 4. The ResNet residual network has five layers, one of which represents the nonlinear function ReLU. The output y is then obtained through a shortcut and a second ReLU. Considering the computational cost, the residual block is optimized. The first convolution layer is 1x1, the middle convolution layer is 3x3, and the third convolution layer is 1x1, which not only maintains accuracy but also reduces the amount of computation. The downsampling operation is removed from the last two convolution blocks, and atrous convolution is used to improve the receptive field. The network adopts the direct connection structure in ResNet; the dilated convolution adds a new parameter - the dilation rate - to the standard convolution layer. The dilation determines the distance between pixels when the convolution kernel processes data, expands the convolution kernel to a specified scale, and fills the unoccupied pixel area in the original convolution kernel with 0; the ResNet residual network structure is proposed, which reduces the number of layers passed by the network gradient through shortcut connections.

3. The method for tracking target in UAV video based on feature fusion and online template updating network according to claim 1 is characterized in that: The network in S3 has two branches; The upper part of the network is the template branch, and the lower part is the search branch. Compare the target template image z and the current frame image x, and find the sample that is most similar to the target template image as the prediction target; To construct the loss function, the loss function for each point in the response graph is l(y,v)=log(1+exp(-yv)); where v is the true value of each point and y∈{+1,-1} is the label corresponding to this point. The overall loss of the corresponding graph is the mean loss of all points, that is: Where u∈D is the position in the response map, D is the response map area; y[u] is expressed as: Where k is the network step size, c is the center point, and R is the search area radius; The parameters θ of the convolutional network are optimized using the stochastic gradient descent optimization method (SGD). The network is divided into two inputs: one input target image with a size of 127×127×3, and the other input current frame search area image with a size of 255×255×3. The search area is 4 times the size of the target in the previous frame, and a cosine window is added to the response image to penalize the maximum displacement.

4. The method for tracking target in UAV video based on feature fusion and online template updating network according to claim 1 is characterized in that: In S7, the cascaded cross-correlation module consists of two cross-correlation operations. The deep cross-correlation operation is performed on each channel of the feature map to obtain different semantic information between channels and achieve the first rough positioning of the target. The target features are then refined, different feature information is learned, and fine-grained feature representation is obtained. In the second cross-correlation operation, the pixel-level global cross-correlation operation is used to calculate the similarity of each pixel in the search feature and the global template feature to obtain high-quality spatial and channel feature information representation.

Images