Unmanned aerial vehicle target tracking method and device

Abstract

The invention relates to an unmanned aerial vehicle target tracking method and device. The method comprises the following steps: acquiring a flight video of an unmanned aerial vehicle by using a color camera; judging whether the current frame is a first frame of the tracked video; if yes, manually marking an unmanned aerial vehicle target, and acquiring a template region image; if not, determining a search area image of target tracking of the unmanned aerial vehicle according to the tracking result of the previous frame; inputting the obtained template area and search area images into an unmanned aerial vehicle target tracking network; the tracking result output by the network is displayed on a display module; and acquiring the next frame of data acquired by the camera, and repeating the steps until the video processing is finished. The method aims at capturing long-distance dependence, reserving accurate position information and effectively extracting target features from a complex background. Meanwhile, a multi-frequency-domain target positioning method is adopted, so that the network has adaptability to targets of different scales, and finally the accuracy and success rate of target tracking of the unmanned aerial vehicle are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and in particular relates to a method and device for tracking a target of an unmanned aerial vehicle. Background Art

[0002] With the rapid development of deep learning and automation technology, drones have been widely used in many fields such as aerial photography and environmental monitoring. However, with the popularity of drones, their potential threat to public safety has become increasingly apparent. Therefore, being able to monitor and track the location information of drones in real time is crucial to maintaining public safety. However, due to the scale changes, motion mutations, and complex flight environments during drone flight, there are still many challenges to achieving accurate drone tracking.

[0003] In the field of target tracking, due to the lack of sufficient drone-specific datasets, research on tracking algorithms for drones is relatively limited. Traditional general target tracking algorithms, such as SiamFC (Fully-Convolutional Siamese Networks for Object Tracking), SiamRPN (High Performance Visual Tracking with Siamese Region Proposal Network), SiamMask (High Performance Visual Tracking with Siamese Region Proposal Network), etc., usually use linear correlation operations to match the target with the search area. This method limits the nonlinear interaction between the template and the search area, resulting in insufficient representation capabilities of the model. In recent years, with the development of large models, Transformer-based tracking algorithms such as STARK (Learning Spatio-Temporal Transformer for Visual Tracking) and ODTrack (Online Dense Temporal Token Learning for Visual Tracking) have gradually emerged, but these algorithms still mainly rely on the training of large targets in general datasets, and have poor adaptability to small targets or small-scale datasets.

[0004] In summary, the existing technologies that can be used for drone target tracking have the following problems:

[0005] In view of the complex flying environment of UAV targets, existing technologies extract features by establishing long-distance dependencies, but ignore the position information, which is crucial for the results of target tracking. At the same time, current target tracking algorithms are mostly trained on large targets in general data sets, which has poor robustness for tracking UAV targets with constantly changing scales. Summary of the invention

[0006] The present invention aims to solve the technical problems in the prior art and provide a method and device for tracking drone targets. The present invention adopts a spatial position perception module to capture long-distance dependencies while retaining accurate position information, effectively extracting target features from complex backgrounds. At the same time, a multi-frequency domain target positioning method is adopted to make the network adaptable to targets of different scales, ultimately improving the accuracy and success rate of drone target tracking.

[0007] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0008] A drone target tracking device, comprising: a drone target tracking network structure;

[0009] The UAV target tracking network structure includes: a multi-frequency domain feature fusion module, and two spatial position perception modules connected to the multi-frequency domain feature fusion module, each spatial position perception module is also connected to a feature extraction network; the multi-frequency domain feature fusion module is also connected to a classification branch and a regression branch;

[0010] The feature extraction network is used for feature extraction of template images and search images;

[0011] The spatial position perception module is used to retain the precise position information of the features while capturing long-range dependencies, enhancing the object features, and better extracting the drone target from the complex background;

[0012] The multi-frequency domain feature fusion module is used to fuse the features extracted by the feature extraction network and the spatial position perception module in sequence using cross attention;

[0013] The classification branch is used to distinguish between the target area and the non-target area in tracking, that is, to determine where the target exists in the search area;

[0014] The regression branch is used in tracking to fine-tune the coordinates, size, and aspect ratio of the bounding box at the location where the target may exist, thereby generating an accurate target box;

[0015] The tracking result output by the UAV target tracking network structure is displayed on the display module.

[0016] A drone target tracking method applicable to the drone target tracking device comprises the following steps:

[0017] S1. Use the camera to collect drone flight video;

[0018] S2, determining whether the current frame of the image captured by the camera is the first frame of the tracking video;

[0019] S3. If yes, manually mark the drone target and obtain the template area image;

[0020] S4. If not, determine the search area image for drone target tracking based on the tracking result of the previous frame;

[0021] S5, inputting the obtained template area and search area images into the UAV target tracking network;

[0022] S6, displaying the tracking result output by the drone target tracking network on the display module;

[0023] S7, obtaining the next frame of image data captured by the camera, and repeating steps S2-S6 until the video processing is completed.

[0024] In the above technical solution, the construction process of the template area image in step S3 is:

[0025] S301, obtaining the marked target coordinate position (x, y, w, h), where (x, y) represents the coordinates of the upper left corner of the tracking box covering the target, and (w, h) represents the width and height of the tracking box;

[0026] S302: Calculate template clipping interval Where W Z and H Z The value is calculated as follows:

[0027]

[0028] Among them, W Z Represents the template area width, H Z Represents the template area is high;

[0029] S303, convert the cropped area image of the template into Get a template image;

[0030] Among them, Z represents the template image, Represents a tensor, that is, the cropped template area is converted into a tensor of size 128*128 and the number of channels is 3.

[0031] In the above technical solution, the construction process of the search area image in step S4 is:

[0032] S401, obtaining the target coordinate position (x, y, w, h) of the previous frame, where (x, y) represents the coordinates of the upper left corner of the tracking frame covering the target, and (w, h) represents the width and height of the tracking frame;

[0033] S402: Calculate search and trim interval Where W X and H X The value is calculated as follows:

[0034]

[0035] Among them, W X Represents the width of the search area, H X It represents a high search area;

[0036] S403, the cropped search area image is transformed into Obtaining a search image;

[0037] Where X represents the search area image.

[0038] In the above technical solution, in step S5, training weights are loaded into the drone target tracking network, and target tracking is performed.

[0039] In the above technical solution, further, in step S5, loading training weights in the drone target tracking network and performing target tracking includes the following steps:

[0040] S501: When processing the first frame image, the template image obtained in step S303 is input into the feature extraction network and the spatial position perception module to obtain a feature map of the template area image.

[0041] S502: When processing subsequent frames except the first frame, the search image obtained in step S403 is input into the feature extraction network and the spatial position perception module to obtain a feature map of the search area image.

[0042] S503: The feature map generated in step S501 and the feature map generated in step S502 Generate feature maps through two-dimensional convolution, flattening and channel transformation operations respectively and feature map

[0043] S504: The feature graph F generated in step S503 is Z1 and feature map F X1 Input into the multi-frequency domain feature fusion module respectively;

[0044] First, high-frequency features are obtained through global multi-head self-attention, and then the feature map is downsampled and input into the multi-head self-attention module to obtain low-frequency features; then the high-frequency features and low-frequency features are concatenated to generate features. and Features Then, through the residual connection method, the features are generated and Features

[0045] The whole process can be summarized as follows:

[0046] F Z3 =F Z1 +F Z2 (5)

[0047] F X3 =F X1 +F X2 (6)

[0048] in,

[0049] F Z2 =MultiHead(F Z1 )+MultiHead(AvgPool(F Z1 )) (7)

[0050] F X2 =MultiHead(F X1 )+MultiHead(AvgPool(F X1 )) (8)

[0051] Among them, MultiHead(*) is a multi-head self-attention operation;

[0052] S505: The feature F obtained in step S504 is Z3 and F X3 Use cross attention for fusion. In this layer, two multi-head attentions are used. For the first multi-head attention, its matrix Q is F Z3 and the spatial position encoding generated by the sine function, K is the feature F X3 and the spatial position encoding generated by the sine function, V is the feature F X3 , generate the fusion matrix through the attention mechanism For the second multi-head attention, its matrix Q is the feature F X3 and the spatial position encoding generated by the sine function, K is the feature F Z3 and the spatial position encoding generated by the sine function, V is the feature F Z3 , generate the fusion matrix through attention

[0053] S506: The output result of step S505 is passed through a multi-head attention mechanism, where the matrix Q is a combination of the feature F2 generated in step S505 and the spatial position code generated by the sine function, K is a combination of the feature F1 and the spatial position code generated by the sine function, and V is the feature F1. The fusion feature is generated through the attention mechanism. Input it into the classification branch and regression branch network to obtain the target tracking box output by the drone target tracking network.

[0054] The beneficial effects of the present invention are:

[0055] The drone target tracking method of the present invention includes a spatial position perception module, which can retain accurate position information while capturing long-distance dependencies, enhance object features, and better extract the drone target from a complex background, thereby ultimately improving the accuracy and success rate of drone target tracking.

[0056] The UAV target tracking method of the present invention is a multi-frequency domain target positioning method, which enables the network to pay attention to high-frequency and low-frequency information at the same time, is adaptable to targets of different scales, and can effectively solve the problem of decreased tracking accuracy caused by scale changes during UAV tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0058] Figure 1 The figure is a flow chart of the UAV target tracking method of the present invention.

[0059] Figure 2 This is a network structure diagram of the UAV target tracking device of the present invention.

[0060] Figure 3 This is a comparison chart of the accuracy and success rate of the UAV target tracking method of the present invention and other tracking algorithms on the self-made data set UAV-Tracking-390. DETAILED DESCRIPTION

[0061] The inventive concept of the present invention is: With the increasing threat of drone intrusion, the perception capability of drones has become crucial in anti-drone missions. In the field of target tracking, although there are many excellent tracking algorithms, there are still many challenges in achieving accurate drone target tracking due to scale changes, motion mutations, and complex flight environments during drone flight. In order to solve this problem, we designed a drone target tracking method and device. The method designs a spatial position perception module that can retain accurate position information while capturing long-distance dependencies to enhance object features in complex background environments. In addition, we designed a multi-frequency domain target positioning method so that the network is adaptable to targets of different sizes to cope with scale changes during drone target flight, and a twin network structure is used to reduce the number of model parameters.

[0062] The present invention is described in detail below with reference to the accompanying drawings.

[0063] Combination Figure 2 Specifically describe the unmanned aerial vehicle target tracking device of the present invention, including: an unmanned aerial vehicle target tracking network structure; the unmanned aerial vehicle target tracking network structure includes: a multi-frequency domain feature fusion module, and two spatial position perception modules connected to the multi-frequency domain feature fusion module, each spatial position perception module is also respectively connected to a feature extraction network; the multi-frequency domain feature fusion module is also respectively connected to a classification branch and a regression branch;

[0064] The feature extraction network is used for feature extraction of template images and search images;

[0065] The spatial position perception module is used to retain the precise position information of the features while capturing long-range dependencies, enhancing the object features, and better extracting the drone target from the complex background;

[0066] The multi-frequency domain feature fusion module is used to fuse the features extracted by the feature extraction network and the spatial position perception module in sequence using cross attention;

[0067] The classification branch is mainly used to distinguish between target areas and non-target areas in tracking, that is, to determine where the target exists in the search area;

[0068] The regression branch is used in tracking to fine-tune the coordinates, size, and aspect ratio of the bounding box at the location where the target may exist, thereby generating an accurate target box;

[0069] The tracking result output by the UAV target tracking network structure is displayed on the display module.

[0070] The UAV target tracking method applicable to the UAV target tracking device of the present invention is shown in the flowchart Figure 1, including the following steps:

[0071] S1, use color camera to collect UAV flight video;

[0072] S2, determining whether the current frame of the image captured by the camera is the first frame of the tracking video;

[0073] S3. If yes, manually mark the drone target and obtain the template area image;

[0074] S4. If not, determine the search area image for drone target tracking based on the tracking result of the previous frame;

[0075] S5, inputting the obtained template area and search area images into the UAV target tracking network;

[0076] S6, displaying the tracking result output by the drone target tracking network on the display module;

[0077] S7, obtaining the next frame of image data captured by the camera, and repeating steps S2-S6 until the video processing is completed.

[0078] The construction process of the template region image in step S3 is:

[0079] S301, obtaining the marked target coordinate position (x, y, w, h), where (x, y) represents the coordinates of the upper left corner of the tracking box covering the target, and (w, h) represents the width and height of the tracking box;

[0080] S302: Calculate template clipping interval Where W Z and H Z The value is calculated as follows:

[0081]

[0082] Among them, W Z Represents the template area width, H Z Represents the template area is high;

[0083] S303, convert the cropped area image of the template into Get the template image;

[0084] Among them, Z represents the template image, Represents a tensor. That is, the cropped template area is converted into a tensor of size 128*128 and the number of channels is 3.

[0085] The construction process of the search area image in step S4 is:

[0086] S401, obtaining the target coordinate position (x, y, w, h) of the previous frame, where (x, y) represents the coordinates of the upper left corner of the tracking frame covering the target, and (w, h) represents the width and height of the tracking frame;

[0087] S402: Calculate search and trim interval Where W X and H X The value is calculated as follows:

[0088]

[0089] Among them, W X Represents the width of the search area, H X It represents a high search area;

[0090] S403: Change the cropped area image of the template into Obtaining a search image;

[0091] Where X represents the search area image.

[0092] Step S5 loads the training weights in the drone target tracking network and performs target tracking. The drone target tracking network structure is as follows: Figure 2 As shown, the following steps are included:

[0093] S501: When processing the first frame image, the template image obtained in step S303 is input into the feature extraction network and the spatial position perception module to obtain a feature map of the template area image.

[0094] S502: When processing subsequent frames except the first frame, the search image obtained in step S403 is input into the feature extraction network and the spatial position perception module to obtain a feature map of the search area image.

[0095] S503: The feature map generated in step S501 and feature map Generate feature maps through two-dimensional convolution, flattening and channel transformation operations respectively and feature map

[0096] S504: The feature graph F generated in step S503 is Z1 and feature map F X1 They are input into the multi-frequency domain feature fusion module respectively. First, high-frequency features are obtained through global multi-head self-attention, and then the feature map is downsampled and input into the multi-head self-attention module to obtain low-frequency features. Then, high-frequency features and low-frequency features are concatenated to generate features. and Features Then, through the residual connection method, the features are generated and Features

[0097] The whole process can be summarized by the formula:

[0098] F Z3 =F Z1 +F Z2 (5)

[0099] F X3 =F X1 +F X2 (6)

[0100] in,

[0101] F Z2 =MultiHead(F Z1 )+MultiHead(AvgPool(F Z1 )) (7)

[0102] F X2 =MultiHead(F X1 )+MultiHead(AvgPool(F X1 )) (8)

[0103] MultiHead(*) is a multi-head self-attention operation.

[0104] S505: The feature F obtained in step S504 is Z3 and feature F X3 Use cross attention for fusion, that is, in this layer, two multi-head attentions are used. For the first multi-head attention, its matrix Q is the feature F Z3 and the spatial position encoding generated by the sine function, K is the feature F X3 and the spatial position encoding generated by the sine function, V is the feature F X3 , generate the fusion matrix through the attention mechanism For the second multi-head attention, its matrix Q is the feature F X3 and the spatial position encoding generated by the sine function, K is the feature F Z3 and the spatial position encoding generated by the sine function, V is the feature F Z3 , generate the fusion matrix through attention

[0105] S506: The output result of step S505 is passed through a multi-head attention mechanism, where the matrix Q is a combination of the feature F2 generated in step S505 and the spatial position code generated by the sine function, K is a combination of the feature F1 and the spatial position code generated by the sine function, and V is the feature F1. The fusion feature is generated through the attention mechanism. Input it into the classification branch and regression branch network to obtain the target tracking box output by the drone target tracking network.

[0106] In summary, the drone target tracking method of the present invention can retain accurate position information while capturing long-distance dependencies through the spatial position perception module to enhance the object features in complex background conditions. This allows the drone target to be better extracted from the complex background. This ultimately improves the accuracy and success rate of drone target tracking (see Figure 3 ).

[0107] The unmanned aerial vehicle target tracking method of the present invention is a multi-frequency domain target positioning method, which makes the network adaptable to targets of different scales and adopts a twin network structure to reduce the number of model parameters.

[0108] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A drone target tracking device, characterized in that: include: UAV target tracking network structure; The UAV target tracking network structure includes: a multi-frequency domain feature fusion module, and two spatial position perception modules connected to the multi-frequency domain feature fusion module, each spatial position perception module is also connected to a feature extraction network; the multi-frequency domain feature fusion module is also connected to a classification branch and a regression branch; The feature extraction network is used for feature extraction of template images and search images; The spatial position perception module is used to retain the precise position information of the features while capturing long-range dependencies, enhancing the object features, and enabling the drone target to be better extracted from the complex background; The multi-frequency domain feature fusion module is used to fuse the features extracted by the feature extraction network and the spatial position perception module in sequence using cross attention; The classification branch is used to distinguish between the target area and the non-target area in tracking, that is, to determine where the target exists in the search area; The regression branch is used in tracking to fine-tune the coordinates, size, and aspect ratio of the bounding box at the location where the target may exist, thereby generating an accurate target box; The tracking result output by the UAV target tracking network structure is displayed on the display module.

2. A drone target tracking method applicable to the drone target tracking device according to claim 1, characterized in that: The steps include: S1. Use the camera to collect drone flight video; S2, determining whether the current frame of the image captured by the camera is the first frame of the tracking video; S3. If yes, manually mark the drone target and obtain the template area image; S4. If not, determine the search area image for drone target tracking based on the tracking result of the previous frame; S5, inputting the obtained template area and search area images into the UAV target tracking network; S6, displaying the tracking result output by the drone target tracking network on the display module; S7, obtaining the next frame of image data captured by the camera, and repeating steps S2-S6 until the video processing is completed.

3. The method for tracking a target by using an unmanned aerial vehicle according to claim 2, characterized in that: The construction process of the template region image in step S3 is: S301, obtaining the marked target coordinate position (x, y, w, h), where (x, y) represents the coordinates of the upper left corner of the tracking box covering the target, and (w, h) represents the width and height of the tracking box; S302: Calculate template clipping interval Where W Z and H Z The value is calculated as follows: Among them, W Z Represents the template area width, H Z Represents the template area is high; S303, convert the cropped area image of the template into Get a template image; Among them, Z represents the template image, Represents a tensor, that is, the cropped template area is converted into a tensor of size 128*128 and the number of channels is 3.

4. The method for tracking a target by using an unmanned aerial vehicle according to claim 3, characterized in that: The construction process of the search area image in step S4 is: S401, obtaining the target coordinate position (x, y, w, h) of the previous frame, where (x, y) represents the coordinates of the upper left corner of the tracking frame covering the target, and (w, h) represents the width and height of the tracking frame; S402: Calculate search and trim interval Where W X and H X The value is calculated as follows: Among them, W X Represents the width of the search area, H X It represents a high search area; S403, the cropped search area image is transformed into Obtaining a search image; Where X represents the search area image.

5. The method for tracking a target by using an unmanned aerial vehicle according to claim 2, wherein: In step S5, the training weights are loaded into the drone target tracking network, and target tracking is performed.

6. The method for tracking a target by using an unmanned aerial vehicle according to claim 4, characterized in that: In step S5, training weights are loaded into the drone target tracking network and target tracking is performed, including the following steps: S501: When processing the first frame image, the template image obtained in step S303 is input into the feature extraction network and the spatial position perception module to obtain a feature map of the template area image. S502: When processing subsequent frames except the first frame, the search image obtained in step S403 is input into the feature extraction network and the spatial position perception module to obtain a feature map of the search area image. S503: The feature map generated in step S501 and the feature map generated in step S502 Generate feature maps through two-dimensional convolution, flattening and channel transformation operations respectively and feature map S504: The feature graph F generated in step S503 is Z1 and feature map F X1 Input into the multi-frequency domain feature fusion module respectively; First, high-frequency features are obtained through global multi-head self-attention, and then the feature map is downsampled and input into the multi-head self-attention module to obtain low-frequency features; then the high-frequency features and low-frequency features are concatenated to generate features. and Features Then, through the residual connection method, the features are generated and Features The whole process can be summarized as follows: F Z3 =F Z1 +F Z2 (5) F X3 =F X1 +F X2 (6) in, F Z2 =MultiHead(F Z1 )+MultiHead(AvgPool(F Z1 )) (7) F X2 =MultiHead(F X1 )+MultiHead(AvgPool(F X1 )) (8) Among them, MultiHead(*) is a multi-head self-attention operation; S505: The feature F obtained in step S504 is Z3 and F X3 Use cross attention for fusion. In this layer, two multi-head attentions are used. For the first multi-head attention, its matrix Q is the feature F Z3 and the spatial position encoding generated by the sine function, K is the feature F X3 and the spatial position encoding generated by the sine function, V is the feature F X3 , generate the fusion matrix through the attention mechanism For the second multi-head attention, its matrix Q is the feature F X3 and the spatial position encoding generated by the sine function, K is the feature F Z3 and the spatial position encoding generated by the sine function, V is the feature F Z3 , generate the fusion matrix through attention S506: The output result of step S505 is passed through a multi-head attention mechanism, where the matrix Q is a combination of the feature F2 generated in step S505 and the spatial position code generated by the sine function, K is a combination of the feature F1 and the spatial position code generated by the sine function, and V is the feature F1. The fusion feature is generated through the attention mechanism. Input it into the classification branch and regression branch network to obtain the target tracking box output by the drone target tracking network.

Images