A Small Target Tracking Method for UAV Video Images Based on Siamese Networks

Abstract

This invention provides a method for tracking small targets in UAV video images based on Siamese networks. The method utilizes an improved VGG16 network to extract features from the template region and search region, obtaining feature maps for these regions and improving their expressive power. The template region feature map is then passed through channel attention and spatial attention channels to increase the proportion of effective features, resulting in an enhanced template region feature map. The enhanced template region feature map and the search region feature map are input into an RPN network, where convolution operations are performed in the classification and regression branches to obtain the target's classification response map score and regression response map score. Finally, the target's position information in the next frame is generated according to the score, achieving target tracking. This invention enhances the target feature representation capability and can adapt to situations where small targets in video images exhibit pose variations, lighting changes, and similar background interference, enabling effective and stable tracking of small targets in UAV video images.

Description

Technical Field

[0001] This invention belongs to the field of visual target tracking technology, and in particular, it is a method for tracking small targets in UAV video images based on twin networks. Technical Background

[0002] When drones fly at high altitudes, the video images they capture often contain small targets, which are easily affected by background and lighting changes. Tracking and locating these small targets using traditional manual feature extraction methods is not ideal because the target occupies a small pixel block with limited surface-level semantic information. Deep learning-based target tracking methods can extract deeper feature information through neural networks, thus advancing video target tracking technology from the drone's perspective.

[0003] With the development of deep learning technology, convolutional neural networks (CNNs) have been widely used in target tracking due to their superior feature extraction capabilities. Jie H et al. proposed an SENet structure that compresses feature maps along channel directions, learns the relationships between channels, and assigns different weight parameters to different channels, enhancing the key information of the feature maps along channel directions. Liu Jiangjiang et al. proposed the SCNet network, which enhances the diversity of output features by expanding the convolutional receptive field, helping CNNs generate more discriminative feature representations and enriching the features generated by the network. Misra D et al. proposed a TripletAttention structure that establishes dependencies between different dimensions of features through rotation operations and residual transformations, emphasizing multi-dimensional interaction without reducing the importance of dimensions, and redistributing attention weights without increasing module complexity. In recent years, Siamese fully convolutional networks have attracted much attention due to their end-to-end training method and good real-time tracking performance. SiamFC uses two identical neural network structures to extract features from the input and output templates, outputting corresponding feature vectors. A loss function is used to calculate the similarity between the two regions, and the location with the highest similarity is the target location. While SiamFC offers fast tracking speed, it performs poorly with varying target scales. The SiamRPN algorithm addresses this by incorporating the RPN network from Faster R-CNN for bounding box regression, saving the time consumed by multi-scale testing in SiamFC. Although Siamese tracking networks determine target locations through similarity comparison between template and search branches, improving tracking speed and success rate to some extent, this approach is not effective for tracking small targets from a UAV perspective. This is because when the target is in a complex scene with varying lighting, interference from similar targets, or deformation, the network cannot extract sufficient information to achieve target tracking due to the target's small size. Summary of the Invention

[0004] The purpose of this invention is to provide a method for tracking small targets in UAV video images based on Siamese networks. This method can enhance the target feature representation capability, reduce the interference of attitude and illumination changes and similar backgrounds, and effectively improve the tracking capability of small targets in UAV video images.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for tracking small targets in UAV video images based on Siamese networks, comprising:

[0007] Step 1: Select video images of small targets under complex conditions, including illumination changes, interference from similar targets, and deformations; input the initial frame image and the next frame image to mark the target to be tracked.

[0008] Step 2: Extract a 127×127 region centered on the target to be tracked from the initial frame image as the template region {x T};

[0009] Step 3, in the next frame image, use the template region {x T A 255×255 region centered at {x} is selected as the search region. S};

[0010] Step four, set {x T}、{x S The input is fed into an improved VGG16 network for feature extraction, generating template region feature maps {F}. T} and search region feature map {f S};

[0011] Step 5, analyze the feature map {F} of the template region. T Further processing involves feature enhancement via channel attention and spatial attention modules, resulting in an enhanced template region feature map {f}. T};

[0012] Step 6, using the template region feature map {f T} represents the convolution kernel pair for the feature map of the search region {f S Perform convolution operations to generate classification response maps. Regression response plot

[0013] Step 7: Regress the predicted position values ​​based on the magnitude of the response values ​​on the generated response map to generate the position information PRO = {(x Pro y pro w pro h pro )};

[0014] Step 8: Determine if this is the last frame. If no, increment the video frame count and return to Step 2. If yes, identify the small target to be tracked in the drone video image.

[0015] The present invention has the following advantages:

[0016] 1. This invention addresses the problem of shallow feature extraction by shallow backbone networks in target tracking by employing an improved VGG16 network as the backbone network for the two branches of the Siamese network. It utilizes a deeper convolutional network to extract target features, thereby improving the feature representation capability.

[0017] 2. This invention addresses the feature weight distribution problem in template branches by employing channel attention and spatial attention to redistribute the feature weights of the feature information extracted from the template branches, thereby increasing the proportion of effective features and enhancing the feature extraction effect.

[0018] Figure and Table Description

[0019] Figure 1 This is a block diagram of the present invention.

[0020] Figure 2 To compare the visualization effects of heatmaps with and without the method of this invention.

[0021] Figure 3 This invention compares the tracking success rate and accuracy of the method with other methods under different attributes.

[0022] Figure 4 This invention compares the tracking performance of its method with other methods in complex scenarios. Detailed Implementation

[0023] The technical solution of the present invention will be described in detail with reference to the accompanying drawings and tables.

[0024] like Figure 1 As shown, the present invention provides a method for tracking small targets in UAV video images based on twin networks, which specifically includes the following steps:

[0025] Step 1: Select video images of small targets under complex conditions, including illumination changes, interference from similar targets, and deformation. Input the initial frame image and the next frame image of the target to be tracked.

[0026] Step 2: Extract a 127×127 region centered on the target to be tracked from the initial frame image as the template region {x T}

[0027] Step 3, in the next frame image, use the template region {x T A 255×255 region centered at {x} is selected as the search region. S}

[0028] Step four: To make the VGG16 network suitable for small target tracking tasks, its network structure is improved by dividing it into four groups of convolutional blocks. The first group of convolutional blocks includes two convolutional layers, Conv1_1 and Conv1_2, and a max pooling layer, maxpool_1. The second group of convolutional blocks includes two convolutional layers, Conv2_1 and Conv2_2, and a max pooling layer, maxpool_2. The third group of convolutional blocks includes three convolutional layers, Conv3_1, Conv3_2, and Conv3_3, and a max pooling layer, maxpool_3. The fourth group of convolutional blocks includes three convolutional layers, Conv4_1, Conv4_2, and Conv4_3.

[0029] Step 5: Set the convolutional layer parameters of the improved VGG16 network. The size of the convolutional kernel is 3×3. The number of convolutional kernels increases with the number of groups, namely 64, 128, 256 and 512 respectively. The sliding stride is 2. In order not to destroy the translation invariance of the network, the padding of the traditional VGG16 network feature map is cancelled, that is, the scale padding is 0. The pooling kernel size is 2×2.

[0030] Step six, set {x T}、{x S The input is fed into the improved VGG16 network, and after passing through 4 groups of 10 convolutional layers and 3 max pooling layers, template region feature maps {F} are generated. T} and search region feature map {F S}

[0031] Step 7, analyze the feature map F of the template region. T Feature enhancement is achieved by using a channel attention module to redistribute attention across channels of the feature map, resulting in the feature map F after channel attention processing. T The specific process is as follows:

[0032] (7.1) For the input feature map F T Spatial information of the feature map is extracted using max pooling and mean pooling respectively, and a max pooling feature map is generated. and mean pooling feature map

[0033] (7.2) Input the two obtained feature maps into a perceptron model consisting of two convolutional layers and one ReLU activation layer to aggregate feature information from different channels;

[0034] (7.3) The two feature vectors that aggregate channel information are nonlinearly mapped to generate channel attention weights M. c (F T );

[0035] (7.4) The weight M c (F T ) and input feature map F T Multiplying them together yields the feature map F obtained after channel attention processing. T ′, where the channel attention weight M c (F T The calculation formula is:

[0036]

[0037] In the formula, σ(·) represents the sigmoid activation function, and W0 and W1 represent the parameters of the two convolutional layers of the perceptron, respectively. These represent the max pooling feature map and the average pooling feature map in the channel attention model, respectively.

[0038] Step 8, transfer the feature map F T Feature enhancement is performed using a spatial attention module to obtain spatial attention weights M. s (F′ T This involves performing attention redistribution across spatial channels on the features to obtain the enhanced template region and ultimately generate the feature map f. T The specific process is as follows:

[0039] (8.1) The feature map F obtained after channel attention processing T Perform max pooling and mean pooling operations respectively to generate max pooling feature maps. and mean pooling feature map

[0040] (8.2) The feature maps are added at the channel level using the Concat(·) operation, and then the channel feature maps are fused using a 3×3 convolution kernel;

[0041] (8.3) The fused feature map is activated by the sigmoid function to obtain the spatial attention weights M. s (F T ′);

[0042] (8.4) The weight M s (F T ′) and feature map F T Multiplying these results in the template region feature map {f} after spatial attention enhancement. T}. Wherein, the spatial attention weight M s (F T The calculation formula is:

[0043]

[0044] In the formula, FT ′ represents the feature map obtained after channel attention processing, σ(·) represents the sigmoid function, and f 3×3 This represents a 3×3 convolution kernel. Concat(·) indicates the feature connection method, which stacks features at the channel level. AvgPool(·) and MaxPool(·) represent average pooling and max pooling operations, respectively. These represent the mean pooling feature map and the max pooling feature map extracted by the spatial attention module, respectively.

[0045] Step 9: In the RPN network, using the template region feature map {f T} represents the convolution kernel pair for the feature map of the search region {f S After passing through two 3×3 convolution kernels, template frame features for the classification branch are generated respectively. and detection frame features Template frame features of regression branch and detection frame features

[0046] Step 10: In the classification branch, use template frame features As a convolution kernel, it is used to detect frame features. Perform convolution operations to generate classification response maps. In the regression branch, template frame features are used. As a convolution kernel, for the detection frame Perform convolution operations to generate a regression response plot. Specifically:

[0047]

[0048]

[0049] Where w and h represent the length and width of the response map, k represents the number of anchor boxes generated, and * represents the convolution operation.

[0050] Step 11, from the classification response map Obtain classified output information From the regression response diagram Obtain regression output information Let p represent the predicted offset at position (i, j), where p ∈ [0, k).

[0051] Step 12: By regressing the predicted position offset values, the position information of the target to be tracked is obtained. Specifically:

[0052]

[0053] Where i∈[0, w), j∈[0, h), l∈[0, 2k), This indicates the anchor's coordinate position information.

[0054] Step 13: Determine if this is the last frame. If no, increment the video frame count and return to Step 2. If yes, identify the small target to be tracked in the drone video image.

[0055] To verify the feasibility and effectiveness of the present invention, the present invention will be further described in detail below with reference to examples.

[0056] This verification was conducted on a desktop computer equipped with an Intel(R) Core(TM) i5-9400F CPU@2.9GHz, 6GB of memory, and an NVIDIA GeForce GTX 1660 graphics card. It was implemented using PyCharm, CUDA 9.0, and CUDNN 7.0. The experiment verified the results from three aspects: a comparison of the heatmap visualization effects of the method of this invention with those without the method of this invention, a comparison of the tracking performance of the method of this invention with other methods, and the tracking effect of the method of this invention in complex scenarios.

[0057] Figure 2 To compare the heatmap visualization effects of incorporating the method of this invention with those not incorporating the method of this invention. The first image in each comparison group is the original input image, the second is the heatmap visualization result after convolution processing, and the third is the heatmap visualization result after applying the convolution features. Figure 2 The result of valueization. (From...) Figure 2 As can be seen from (a), before the method of the present invention was added, the target features extracted by the neural network contained a lot of background interference information, such as zebra crossings and sidewalks. As can be seen from the binarized image, although the target body to be tracked can be located, it is easy to have problems such as inaccurate positioning. Figure 2 Image (b) shows the effect of incorporating the method of this invention. As can be seen from the image, the feature enhancement effect on the target region to be tracked is significant, strengthening the expressive power of the features. The binarized image shows that the improved algorithm identifies the target region more accurately. Therefore, the method of this invention can perform feature enhancement operations on the target region to be tracked, helping the convolutional neural network to better locate the target and achieve target tracking.

[0058] Figure 3The figure shows a comparison of the tracking success rate and accuracy of the method of this invention with other methods under different attributes. As can be seen from the tracking accuracy curve, when the target is moving rapidly, the tracking accuracy of this invention is 1.8% higher than SiamDWfc; when the target is deformed, the accuracy of this invention is 6.7% higher than SiamRPN. In the tracking success rate curve, under the three different attributes, the tracking success rate of this invention is improved by 2.7%, 1.2%, and 5.4% respectively compared to the SiamRPN algorithm. The comparison results show that the method of this invention can adapt well to target tracking tasks in different environments. Good tracking accuracy ensures accurate estimation of the target position, and a high success rate ensures correct positioning of the target, effectively improving the tracking drift that occurs during the tracking process.

[0059] Figure 4 The diagram shows a comparison of the tracking performance of the method of this invention with other methods in complex scenarios. Red represents the method of this invention, blue represents the SiamRPN_Alex method, and green represents the SiamRPN_ResNet50 method. Figure 4 In (a) and (d), the vehicles and pedestrians in the bike1 and group1 video sequences undergo significant pose changes. Specifically, the SiamRPN_ResNet50 tracking box in frame 269 of bike1 shows a noticeable shift. The method of this invention further extracts object features from the template frame, achieving accurate target localization without tracking box drift. Figure 4 In (b), the boat4 video sequence exhibits significant illumination changes. Due to interference from these changes, the tracking results of both SiamRPN_Alex and SiamRPN_ResNet50 show some errors in frame 367, leading to deviations in target scale estimation. The method of this invention, however, can achieve better target tracking under varying illumination conditions. Figure 4 In both (c) and (d) of the 4th generation, the car1 and group1 video sequences contain interfering objects similar to the target. For example, in car1, a similar target appears near the vehicle's location in frame 1248, and there is some occlusion. At this time, both SiamRPN_Alex and SiamRPN_ResNet50 include the interfering target. In group1, in frame 2823, due to the influence of a pedestrian, SiamRPN_Alex and SiamRPN_ResNet50 have gradually deviated from the target position. The method of this invention uses an attention mechanism to selectively enhance the features of the target to be tracked, ensuring that the maximum response position of the target is obtained during the similarity comparison process, thus achieving more accurate tracking.

Claims

1. A method for tracking small targets in UAV video images based on Siamese networks, characterized in that, Includes the following steps: Step 1: Select video images of small targets under conditions including illumination changes, interference from similar targets, and complex deformations; input the initial frame image and the next frame image to mark the target to be tracked. Step two, take 127*127 region as template region {x T} in initial frame image with the target to be tracked as the center Step three, in the next frame image, take 255*255 region as search region {x T} with template region {x S} as center. Step four, set {x T }、{x S The input is fed into an improved VGG16 network for feature extraction, generating template region feature maps {F}. T } and search region feature map {f S }; Step 5, analyze the feature map {F} of the template region. T Further processing involves feature enhancement via channel attention and spatial attention modules, resulting in an enhanced template region feature map {f}. T }; Step 6, using the template region feature map {f T } represents the convolution kernel pair for the feature map of the search region {f S Perform convolution operations to generate classification response maps. Regression response plot Step 7: Regress the predicted position values ​​based on the response values ​​on the generated response map to generate the position information PRO = {(x pro y pro w pro h pro )}; Step 8: Determine if this is the last frame. If no, increment the video frame count and return to Step 2. If yes, identify the small target to be tracked in the drone video image.

2. The method for tracking small targets in UAV video images based on Siamese networks according to claim 1, characterized in that, Step four will {x T }、{x S The input is fed into an improved VGG16 network for feature extraction, generating template region feature maps {F}. T } and search region feature map {f S Specifically: (2.1) In order to make the VGG16 network suitable for small target tracking tasks, its network structure is adjusted and divided into four groups of convolutional blocks. The first group of convolutional blocks includes two convolutional layers, Conv1_1 and Conv1_2, and a max pooling layer, maxpool_1. The second group of convolutional blocks includes two convolutional layers, Conv2_1 and Conv2_2, and a max pooling layer, maxpool_2. The third group of convolutional blocks includes three convolutional layers, Conv3_1, Conv3_2, and Conv3_3, and a max pooling layer, maxpool_3. The fourth group of convolutional blocks includes three convolutional layers, Conv4_1, Conv4_2, and Conv4_3. (2.2) Set the convolutional layer parameters of the improved VGG16 network, where the size of the convolutional kernel is 3×3, the number of convolutional kernels increases with the number of groups, namely 64, 128, 256 and 512 respectively, the sliding stride is 2, in order not to destroy the translation invariance of the network, the padding of the feature map of the traditional VGG16 network is cancelled, that is, the scale padding is 0, and the pooling kernel size is 2×2. (2.3) Set {x T }、{x S The input is fed into the improved VGG16 network, passing through 4 groups of 10 convolutional layers and 3 max-pooling layers to generate template region feature maps {F}. T } and search region feature map {f S } 3. The method for tracking small targets in UAV video images based on twin networks according to claim 1, characterized in that, In step five, the template region feature map {F T Further processing involves feature enhancement via channel attention and spatial attention modules, resulting in an enhanced template region feature map {f}. T Specifically: (3.1) For the template region feature map F T Use the channel attention module to enhance features, achieve attention reallocation on the channels of the feature map, and obtain the feature map F T ′. The specific process is as follows: First, the input feature map F T Spatial information of the feature map is extracted using max pooling and mean pooling respectively, and a max pooling feature map is generated. and mean pooling feature map Secondly, the two feature maps obtained are input into a perceptron model consisting of two convolutional layers and one ReLU activation layer to aggregate feature information from different channels; Then, the two feature vectors that aggregate channel information are nonlinearly mapped to generate channel attention weights M. c (F T ); Finally, the weight M c (F T ) and input feature map F T Multiplying them together yields the feature map F obtained after channel attention processing. T ′, where the channel attention weight M c (F T This can be represented as: In the formula, σ(·) represents the sigmoid activation function, and W0 and W1 represent the parameters of the two convolutional layers of the perceptron, respectively. These represent the max pooling feature map and the average pooling feature map in the channel attention model, respectively. (3.2) Transfer the feature map F T Feature enhancement is performed using a spatial attention module to obtain spatial attention weights M. s (F T This involves performing attention redistribution across spatial channels on the feature map to obtain the enhanced template region feature map {f}. T The specific process is as follows: First, the feature map F obtained after channel attention processing... T Perform max pooling and mean pooling operations respectively to generate max pooling feature maps. and mean pooling feature map Secondly, the feature maps are added at the channel level using the Concat(·) operation, and then the channel feature maps are fused using a 3×3 convolution kernel; Then, the fused feature map is activated by the sigmoid function to obtain the spatial attention weights M. s (F′ T ); Finally, the weight M s (F T ′) and feature map F T Multiplying these results in the template region feature map {f} after spatial attention enhancement. T }, where the spatial attention weight M s (F T ′) can be represented as: In the formula, F T ′ represents the feature map obtained after channel attention processing, σ(·) represents the sigmoid function, and f 3×3 This represents a 3×3 convolution kernel. Concat(·) indicates the feature connection method, which stacks features at the channel level. AvgPool(·) and MaxPool(·) represent average pooling and max pooling operations, respectively. These represent the mean pooling feature map and the max pooling feature map extracted by the spatial attention module, respectively.

Images