Unmanned aerial vehicle target tracking method and system based on transformer multi-layer feature fusion

Abstract

The present application belongs to the technical field of computer vision and image processing, and particularly relates to a UAV target tracking method and system based on transformer multi-layer feature fusion, comprising: using a Mobilenetv3 network to perform feature extraction on target regions in search pictures and template pictures, and obtaining three feature maps of different scales respectively; using a multi-layer feature fusion method to perform feature fusion on the three feature maps of different scales, and obtaining a fused search feature map and a fused template feature map; using a transformer-based feature matching method to perform deep fusion on the fused search feature map and the fused template feature map, and obtaining a response map; and inputting the response map into a multi-head attention module to obtain a target bounding box. The present application develops a lightweight UAV target tracking algorithm based on transformer multi-layer feature fusion by using a transformer architecture, and realizes more robust and rapid prediction of a target bounding box for a UAV.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and in particular relates to a method and system for UAV target tracking based on transformer multi-layer feature fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, unmanned aerial vehicles (UAVs) have been widely used in target tracking tasks such as terrain reconnaissance and geographical surveys due to their advantages of small size, maneuverability, and ease of operation. However, the rapid movement of UAVs and target occlusion pose significant challenges to establishing a fast and robust UAV target tracker. Currently, mainstream trackers are mainly divided into two categories: target trackers based on correlation filtering and those based on Siamese networks. Trackers based on correlation filtering utilize online regression to update the tracker. Although the regression algorithm and image sampling methods are constantly evolving, the problem of tracking performance shifting with target scale changes remains unresolved. Furthermore, the hand-crafted features such as color and grayscale extracted by these trackers are relatively coarse, often resulting in low tracking accuracy. Target trackers based on Siamese networks achieve a better balance between tracking performance and inference speed. They employ convolutional neural networks for feature extraction, effectively compensating for the shortcomings of correlation filtering algorithms in feature extraction. However, due to the limitations of onboard computing power in UAVs, Siamese networks with deep convolutional networks as the backbone cannot meet the real-time inference speed requirements of the tracker. For example, SiamRPN++ uses ResNet50 as the feature extraction network to improve target tracking accuracy, which results in a large number of parameters that reduce the onboard inference speed of the UAV.

[0004] Shallow networks cannot effectively capture higher semantic information, resulting in reduced tracking performance. The introduction of attention mechanisms improves the model's focus on important features and enhances its ability to distinguish targets from complex backgrounds, thus compensating for the shortcomings of shallow network feature extraction. However, convolution-based attention mechanisms cannot establish global dependencies on image features, therefore the tracker cannot effectively track targets with large deformations or occlusions. Summary of the Invention

[0005] To address the issue of low accuracy in lightweight trackers, this invention provides a UAV target tracking method and system based on transformer-based multi-layer feature fusion. This invention proposes a transformer-based multi-layer feature fusion module, enabling the model to learn richer semantic and global features. Furthermore, the TrTr tracker proposes a tracking architecture based on transformer-based encoder and decoder structures, employing a multi-head attention mechanism for feature fusion between the search image and the template image, replacing the traditional cross-correlation operation of linear matching on the search image using the template as a sliding window. This invention improves upon the TrTr tracking architecture, obtaining a richer and more centralized response map.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention provides a UAV target tracking method based on transformer multi-layer feature fusion, comprising:

[0008] The Mobilenetv3 network was used to extract features from the target regions in the search images and template images, resulting in feature maps at three different scales.

[0009] A multi-layer feature fusion method is used to fuse feature maps of three different scales to obtain a fused search feature map and a fused template feature map.

[0010] A transformer-based feature matching method is used to perform deep fusion of the fusion search feature map and the fusion template feature map to finally obtain the response map.

[0011] The response map is input into the multi-head attention module to obtain the final target bounding box.

[0012] A second aspect of the present invention provides a UAV target tracking system based on transformer multi-layer feature fusion, comprising:

[0013] The feature extraction module is configured to use the Mobilenetv3 network to extract features from the target regions in the search image and the template image, and obtain feature maps at three different scales.

[0014] The multi-layer feature fusion module is configured to use a multi-layer feature fusion method to fuse feature maps at three different scales respectively, and obtain a fusion search feature map and a fusion template feature map.

[0015] The feature deep fusion module is configured to use a transformer-based feature matching method to perform deep fusion of the fusion search feature map and the fusion template feature map to finally obtain the response map.

[0016] The target tracking module is configured to input the response map into the multi-head attention module to obtain the final target bounding box.

[0017] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a UAV target tracking method based on transformer multi-layer feature fusion as described in the first aspect of the present invention.

[0018] The fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a UAV target tracking method based on transformer multi-layer feature fusion as described in the first aspect of the present invention.

[0019] The above one or more technical solutions have the following beneficial effects:

[0020] (1) This invention employs a transformer architecture to develop a lightweight UAV target tracking algorithm based on transformer multi-layer feature fusion, enabling UAVs to predict target bounding boxes more robustly and quickly. Considering the limitations of UAV computing resources, this invention uses the Mobilenetv3 network as the feature extraction network of the tracker, which can greatly reduce the number of convolution kernel channels, realize a lightweight UAV tracker design, and thus quickly predict target bounding boxes, meeting the requirements of UAV tracking speed.

[0021] (2) Considering the insufficient tracking accuracy of shallow feature extraction networks, in order to solve the problem of feature extraction being localized due to convolution operations, this invention designs a lightweight multi-layer feature fusion module based on encoder, which enhances the tracker's attention to different semantic information, realizes the effective combination of different semantic information, and enables the tracker to extract more discriminative target features.

[0022] (2) This invention improves the transformer feature fusion mechanism of the TrTr tracker, achieving deep fusion of acquired images and target features to obtain richer and more concentrated response maps. This improves the learning efficiency of the tracker's classification and regression tasks, playing a significant role in enabling efficient target tracking in complex UAV environments. This invention is of great significance for subsequent applications of UAVs in tracking and surveillance, urban pursuit, and close-range battlefield reconnaissance.

[0023] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0025] Figure 1 This is a flowchart of a UAV target tracking method based on transformer multi-layer feature fusion, as shown in the first embodiment.

[0026] Figure 2 This is a flowchart of the encoder-based multi-layer feature fusion method in the first embodiment.

[0027] Figure 3 This is a flowchart of the feature deep fusion method based on transformer in the first embodiment.

[0028] Figure 4 (a), (b), (c), and (d) in the figure are the visualization results of the first embodiment on the DTB70 dataset, respectively.

[0029] Figure 5 (a), (b), (c), and (d) in the figure are the visualization results of the first embodiment on the UAV123 dataset, respectively. Detailed Implementation

[0030] Example 1

[0031] Inspired by the TrTr tracker, this invention proposes a UAV tracking algorithm based on transformer-based multi-feature information fusion. It should be noted that this invention uses a lightweight MobileNetv3 network to replace the parameter-intensive ResNet50 network, improving the UAV's processing speed; it designs feature fusion at different semantic levels based on the transformer encoder structure, increasing the tracker's focus on important features and thus enhancing its feature extraction capabilities; it introduces and improves the feature matching method of the TrTr tracker's search image and template image, obtaining richer and more accurate response maps, effectively improving the tracker's tracking performance under conditions of intense aerial target movement and deformation.

[0032] It should be noted that the search images are each frame captured by the drone, while the template image refers to the first frame of the target captured by the drone.

[0033] like Figure 1As shown, this embodiment discloses a UAV target tracking method based on transformer multi-layer feature fusion, including:

[0034] Step 1: Wait for the drone system to unlock and initialize its parameters, including: initializing the parameters of the monocular camera and LiDAR; loading the target tracking pre-trained model; and waiting for the drone to unlock command.

[0035] Step 2: After unlocking, the drone performs target tracking tasks.

[0036] The specific steps are as follows:

[0037] Step 201: Determine whether the target box has been manually selected;

[0038] Step 202: The model extracts image features from the template region;

[0039] Step 203: The monocular camera acquires image data in real time;

[0040] Step 204: Extract features from the acquired image information.

[0041] Step 3: To meet the real-time requirements of UAV target tracking, Mobilenetv3 is used as the feature extraction network for the tracker. The template region image and the search region image are respectively input into the shared backbone network Mobilenetv3 to obtain feature information and perform feature fusion operations. The algorithm flowchart is shown below. Figure 2 As shown. The specific operation is as follows: the output feature maps of the sixth, eighth and eleventh layers are subjected to a 1×1 convolution operation to obtain three feature maps of different scales, M3, M4 and M5.

[0042] Step 4: Use a multi-layer feature fusion method to fuse the feature maps at three different scales to obtain a fused search feature map and a fused template feature map;

[0043] Step 5: Use a transformer-based feature matching method to perform deep fusion of the two fused feature maps to obtain a richer response map;

[0044] Step 6: Input the richer response map into the multi-head attention module to obtain the final target bounding box.

[0045] In step 4, the constraints affecting the accuracy and speed of target detection are analyzed. This invention designs a multi-feature information fusion module based on the transformer encoder structure, which can enhance the feature extraction capability of the model and realize global dependence of different semantic information, thereby enabling the model to extract more general and important feature information.

[0046] Specifically, it includes:

[0047] Step 401: Perform 1×1 convolution operations on feature maps of different scales respectively.

[0048] Step 402: Perform position self-learning encoding on the processed feature maps.

[0049] Step 403: Perform a Flatten operation on the encoded feature maps at different scales and stack the channels to obtain key values ​​Q (query), K (key), and evaluation value V (value). Then, operate on Q, K, and V based on the attention function MultiHead(Q,K,V). The operation can be represented as MultiHead(Q,K,V) = Concat(H1,H2,…,H…). N W o Finally, the search region features X and the template region features Z are obtained.

[0050] Attention function formula:

[0051]

[0052] Where C is the normalization parameter, W o ∈R C×C The linear transformation parameters are represented by N, where N is the number of attention heads, and d is the number of attention heads. head This indicates that the dimension of each attention head is equal to

[0053] Step 404: The fused token is entered into the transformer's encoder structure, thereby achieving feature fusion at different scales;

[0054] Specifically, the template region features Z and the search region features X are fed into the feedforward neural network FFN to obtain fused feature maps f(Z) = Norm(Linear(ReLu(Lionear(Z)))+Z) and f(X) = Norm(Linear(ReLu(Linear(X)))+X).

[0055] Here, the Linear function represents a fully connected operation, ReLU is an activation function that makes the model more non-linear, and Norm represents a summation and averaging operation.

[0056] Step 405: Perform a split operation on the fused feature map to obtain a fused search feature map X with the same size as the output prediction feature map of the eighth layer. f and fusion template feature map T f .

[0057] In step 5, to obtain richer and more accurate foreground semantic information, this invention designs an encoder-decoder structure based on TrTr that can extract richer foreground semantic information, realizing global contextual dependency between the search and template regions, and improving the robustness of the model. Specifically, as follows:

[0058] Step 501: Analyze the fusion template feature map T f Perform position encoding and Flatten operation.

[0059] Step 502: Input the output template feature map into the encoder module to obtain the depth template feature map T. f1 .

[0060] Step 503: Search for the fusion search feature map X of the image. f Perform position encoding and Flatten operation.

[0061] Step 504: Output the search feature map by inputting it into the decoder module to obtain the depth search feature map X. f1 .

[0062] Step 505, Depth Template Feature Map T f1 With deep search feature map X f1 Perform deep feature fusion to obtain intermediate feature map X f2 .

[0063] Step 506, Flattened depth template feature map T f1 With intermediate feature map X f2 After channel splicing, the signal is input into the encoder module to obtain the response graph.

[0064] Step 507: To eliminate interference and enrich semantic information, this invention performs channel concatenation of the response map g and the search region feature map X to obtain g1, and then performs self-attention mechanism and fully connected operation to finally obtain a richer response map Response_map, as shown in the following formula:

[0065] Response_map=Norm(Linear(ReLu(Linear(g2)))+g2) (2)

[0066] g2 = MultiHead(g1, g1, g1) (3)

[0067] In step 6, in order to improve the accuracy of the Siamese network in object classification and bounding box prediction, the present invention introduces a multi-head attention module, which includes a fully connected layer module and a convolutional layer module, used for classification and bounding box regression, respectively.

[0068] In order to fully utilize the advantages of fully connected prediction head fc-head and convolutional prediction head conv-head, this invention introduces the Double-Head method, which goes beyond individual fc-head and conv-head, so that each head does not only focus on a single classification task or regression task, but performs both tasks simultaneously.

[0069] First, a confidence map and a regression map are predicted based on a Double-Head network. Then, the pixels (proposals) in the confidence map are scored and ranked using a cosine window and scale change penalty. The pixel with the highest score is selected as the anchor box, which corresponds to the center position of the target. Finally, the boundary position in the regression map is combined to obtain the final target bounding box.

[0070] Specifically as follows:

[0071] Step 601: The response map Response_map obtains intermediate feature maps fc_branchmap and conv_branch map through the convolutional layer module, respectively.

[0072] Step 602: fc_branch map obtains the first classification map cls_fc and the first regression map bbox_fc through two fully connected layers.

[0073] Step 603: The conv_branch map obtains the second classification map cls_conv and the second regression map bbox_conv through the convolutional layer.

[0074] Step 604: Perform a weighted average operation on the first and second classification maps to obtain the final classification map, i.e., the confidence map:

[0075] ls_score_final=cls_fc+cls_conv*(1-cls_fc).

[0076] The final regression plot is obtained by performing a weighted average operation on the first and second regression plots.

[0077] Step 605: The position corresponding to the maximum value of the pixel in the confidence map is the center position of the target box. The value corresponding to the position in the regression map is the top, bottom, left, and right position of the target box. Use a cosine window and scale change penalty to sort the pixels in the confidence map, select the anchor box with the highest confidence, and then combine it with the final regression map to obtain the final target prediction box.

[0078] It should be noted that, because most anchor boxes generated during model learning include background information, the distribution of positive and negative samples is uneven. To effectively alleviate the problems of uneven distribution of positive and negative samples and low discrimination of difficult samples, this invention adopts the FocalLoss loss function FL(p t )=-α t (1-p t ) γ log(p t ).in, α t Weights help handle class imbalance, where p is the value of each pixel in the classification map, (1-p) t ) γ It is a regulation factor, and γ>0 is an adjustable focusing parameter. It makes the model more focused on training on difficult samples, greatly improving the model training efficiency.

[0079] This invention adopts the Mobilenetv3 pre-trained model proposed by Andrew G. Howard, and introduces a transformer tracking structure and a self-developed transformer-based multi-layer feature fusion module. Finally, it is trained on the VID, COCO, GOT10K and Lasot datasets, and the performance of lightweight and heavyweight trackers is compared on the UAV123 and DTB70 datasets, respectively, as shown in Table 1 and Table 2.

[0080] Table 1 Performance comparison of the lightweight model on the DTB70 and UAV123 datasets

[0081]

[0082] Table 2 Evaluation of heavyweight models on the DTB70 and UAV123 datasets

[0083]

[0084] The DTB70 dataset contains 70 scenes of intense motion, posing a challenge for drone target tracking. Figure 4In (a), (b), (c), and (d), this benchmark is used to evaluate the effectiveness of the tracker of the present invention in handling motion. Specifically, the performance of the tracker using AlexNet as the backbone is compared, as shown in Table 1. The experimental data shows that the tracker of the present invention achieves a frame rate (FPS) of approximately 130 fps and achieves the best success rate. Compared with the SiamCAR tracker, the inference speed is similar, the accuracy exceeds 0.01, and the success rate exceeds 0.026. Therefore, the tracker of the present invention, as a lightweight tracker, achieves a good balance between speed and accuracy. Furthermore, the performance of the tracker using ResNet50 as the backbone is compared, as shown in Table 2. The tracker of the present invention achieves the highest speed and the second-best success rate. Compared with the SiamBAN model, the success rate is lower than 0.001, but the inference speed is three times that of the SiamBAN tracker.

[0085] An aerial tracking perspective better reflects the timeliness of UAV target tracking in flight environments. The UAV123 dataset is a large-scale aerial tracking dataset. Figure 5 In (a), (b), (c), and (d), this benchmark is used to evaluate the effectiveness of the tracker of the present invention in tracking targets during aerial operations. Compared with lightweight trackers, the tracker of the present invention achieves optimal success rate and accuracy with similar inference speed, as shown in Table 1. Compared with trackers using ResNet50 as the backbone, it has the highest inference speed and a success rate comparable to SiamBAN, both reaching the highest levels.

[0086] Therefore, this invention introduces a self-developed feature fusion module and a transformer tracking architecture based on the improved TrTr, achieving a good balance between target tracking speed and tracking accuracy. Compared with existing models, this invention is more advanced.

[0087] Example 2

[0088] This embodiment discloses a UAV target tracking system based on transformer multi-layer feature fusion, including:

[0089] The feature extraction module is configured to use the Mobilenetv3 network to extract features from the target regions in the search image and the template image, and obtain feature maps at three different scales.

[0090] The multi-layer feature fusion module is configured to use a multi-layer feature fusion method to fuse feature maps at three different scales respectively, and obtain a fusion search feature map and a fusion template feature map.

[0091] The feature deep fusion module is configured to use a transformer-based feature matching method to perform deep fusion of the fusion search feature map and the fusion template feature map to finally obtain the response map.

[0092] The target tracking module is configured to input the response map into the multi-head attention module to obtain the final target bounding box.

[0093] Example 3

[0094] The purpose of this embodiment is to provide a computer-readable storage medium.

[0095] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps in a UAV target tracking method based on transformer multi-layer feature fusion as described in Embodiment 1 of this disclosure.

[0096] Example 4

[0097] The purpose of this embodiment is to provide an electronic device.

[0098] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in a UAV target tracking method based on transformer multi-layer feature fusion as described in Embodiment 1 of this disclosure.

[0099] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0100] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0101] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A UAV target tracking method based on transformer multi-layer feature fusion, characterized in that, include: The Mobilenetv3 network was used to extract features from the target regions in the search images and template images, resulting in feature maps at three different scales. A multi-layer feature fusion method is used to fuse feature maps of three different scales to obtain a fused search feature map and a fused template feature map. A transformer-based feature matching method is used to perform deep fusion of the fusion search feature map and the fusion template feature map to finally obtain the response map. The response map is input into the multi-head attention module to obtain the final target bounding box; The method of using a multi-layer feature fusion approach to fuse feature maps at three different scales includes: Convolution operations are performed on feature maps of three different scales respectively; the convolutional feature maps are then encoded using positional self-learning; and the encoded feature maps of different scales are then flattened and their channels are superimposed. The superimposed features are input into the transformer's encoder structure to achieve feature fusion at different scales; the fused feature map is then split to obtain a fused feature map with the same output size as the intermediate layer. The method of deep fusion of the fusion search feature map and the fusion template feature map using a transformer-based feature matching method includes: performing position encoding and flattening operations on the fusion template feature map, and then inputting it into an encoder module to obtain a deep template feature map; performing position encoding and flattening operations on the fusion search feature map, and then inputting it into a decoder module to obtain a deep search feature map; performing deep feature fusion on the deep template feature map and the deep search feature map to obtain an intermediate feature map; concatenating the flattened deep template feature map with the intermediate feature map by channel, and then inputting it into an encoder module to obtain a response map; concatenating the response map with the search region feature map by channel, and then performing self-attention mechanism and fully connected operation to finally obtain a richer response map; The process of fusing deep features between the deep template feature map and the deep search feature map includes performing cross-attention operations on the deep template feature map and the deep search feature map.

2. The UAV target tracking method based on transformer multi-layer feature fusion as described in claim 1, characterized in that, The Flatten operation and channel stacking of the encoded feature maps at different scales include: The encoded feature maps at different scales are flattened to obtain key values ​​Q, K and evaluation value V; The search region feature map and the template region feature map are obtained by operating on Q, K and V based on the attention function.

3. The UAV target tracking method based on transformer multi-layer feature fusion as described in claim 1, characterized in that, The step of inputting the response map into the multi-head attention module to obtain the final target bounding box includes: First, a confidence map and a regression map are predicted based on the Head network. Then, the pixels in the confidence map are scored and ranked using a cosine window and a scale change penalty. The pixel with the highest score is selected, and this pixel corresponds to the center position of the target. Finally, the boundary position in the regression map is combined to obtain the final target bounding box.

4. The UAV target tracking method based on transformer multi-layer feature fusion as described in claim 3, characterized in that, The first step, predicting the confidence map and regression map based on the Head network, includes: The Head network includes parallel fully connected layer modules and convolutional layer modules; The response maps are input into the fully connected layer module and the convolutional layer module respectively to obtain the first classification map, the first regression map, the second classification map, and the second regression map. A weighted average of the first and second classification maps is performed to obtain the confidence map. The final regression plot is obtained by performing a weighted average operation on the first and second regression plots.

5. A UAV target tracking system based on transformer multi-layer feature fusion, employing the UAV target tracking method based on transformer multi-layer feature fusion as described in any one of claims 1-4, characterized in that, include: The feature extraction module is configured to use the Mobilenetv3 network to extract features from the target regions in the search image and the template image, and obtain feature maps at three different scales. The multi-layer feature fusion module is configured to use a multi-layer feature fusion method to fuse feature maps at three different scales respectively, and obtain a fusion search feature map and a fusion template feature map. The feature deep fusion module is configured to use a transformer-based feature matching method to perform deep fusion of the fusion search feature map and the fusion template feature map to finally obtain the response map. The target bounding box tracking module is configured to input the response map into the multi-head attention module to obtain the final target bounding box.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the UAV target tracking method based on transformer multi-layer feature fusion as described in any one of claims 1-4.

7. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the UAV target tracking method based on transformer multi-layer feature fusion as described in any one of claims 1-4.

Images