A Deep Learning-Based Method for Detecting and Tracking Small Unmanned Aerial Vehicles

Abstract

This invention discloses a deep learning-based method for detecting and tracking small unmanned aerial vehicles (UAVs). The method includes: 1) acquiring historical motion video data of the small UAV; 2) inputting the video into a salient target detection network to learn the UAV's features; 3) modeling the UAV's temporal and spatial maps using the features and position information of the same UAV at different times; 4) inputting the spatiotemporal maps into a trajectory tracking network based on long short-term memory (LSM) recurrent neural networks to track the UAV's trajectory; and 5) performing distance error analysis based on the tracked UAV trajectory. This invention provides a novel and practical method for tracking the motion trajectory of small UAVs. By using a salient target detection network to obtain the UAV's spatiotemporal information from historical motion videos, and by using a LSM recurrent neural network to model the UAV's trajectory features in both time and space dimensions, it achieves accurate detection and tracking of the small UAV's motion.

Description

Technical Field

[0001] This invention relates to the field of single-frame small target detection and video target tracking, specifically to a method for detecting and tracking small unmanned aerial vehicles (UAVs) based on deep learning. Background Technology

[0002] Unmanned Aerial Vehicle (UAV) is a general term for unmanned aerial vehicles. Nowadays, UAVs can be seen in various fields. With the continuous improvement of UAV technology, the application fields and scope of UAVs are constantly expanding. UAVs can be mainly used in disaster relief, environmental monitoring, urban security, maritime inspection and other unmanned operations.

[0003] The ease of operation and low cost of drones have led to an increasingly serious problem of their illegal use worldwide, ranging from misleading toys to espionage. In particular, sensitive areas such as airports, company offices, or military facilities require the detection, tracking, and surveillance of unidentified small drones.

[0004] The small size of small drones, coupled with the interference of the complex and ever-changing surrounding environment, makes the detection of small drones a challenging task. The high acceleration, high top speed, and excellent maneuverability of drones make it difficult to track them or reliably predict their flight paths.

[0005] Therefore, based on the above technical problems, a new small target detection and video tracking algorithm needs to be designed. Summary of the Invention

[0006] Purpose of the invention: To address the aforementioned existing technologies, a deep learning-based method for detecting and tracking small unmanned aerial vehicles (UAVs) is proposed, enabling accurate detection and tracking of these UAVs.

[0007] Technical solution: A method for detecting and tracking small unmanned aerial vehicles based on deep learning, including:

[0008] Step S1: Obtain historical motion video data of the small drone as training sample data;

[0009] Step S2: Input the training sample data into the deep learning network for salient target detection of small UAVs to learn the target bounding box of the small UAV in the image at each time step;

[0010] Step S3: Model a spatial map of the small drone's motion trajectory using the spatial coordinates of the small drone over a period of time;

[0011] Step S4: Input the spatial map data of the small UAV's motion trajectory into the multi-step small UAV motion trajectory tracking network based on long short-term memory recurrent neural network in time sequence to achieve accurate detection and tracking of the small UAV's motion trajectory;

[0012] Step S5: Perform distance error analysis based on the tracked small drone's motion trajectory.

[0013] Furthermore, step S1 includes:

[0014] S101: Use image acquisition equipment to acquire historical motion video image data of small drones;

[0015] S102: Normalize the data of each frame of the video image to [0,1] to obtain a continuous image set data I = {i1,i2,…,i...} t ,…,i n There are n images in total.

[0016] Furthermore, the deep learning network for salient target detection of small UAVs includes a dual-channel feature extraction module, a multi-scale nested interaction module, a dual-channel aggregation module, and a small UAV position detection head; step S2 includes the following sub-steps:

[0017] S201: The dual-channel feature extraction module uses a convolutional module and a residual module to extract the internal features and boundary detail features of the small UAV, respectively;

[0018] S202: Decouple the small target labels of the acquired image into internal labels and boundary detail labels. The multi-scale nested interaction module uses multiple nodes to nest and exchange the features finally extracted by the dual-channel feature extraction module.

[0019] S203: Uses a dual-channel aggregation module to integrate internal features and boundary detail features;

[0020] S204: The small drone prediction head uses CenterNet to predict the bounding box of the target.

[0021] Furthermore, step S3 includes the following sub-steps:

[0022] S301: Extract the spatial coordinates of all small drones in each frame of an image over a period of time;

[0023] S302: Based on spatial coordinate modeling, a spatial graph representing the interaction relationship between small drones at time t is created. Each node in the spatial graph represents a different small drone, and each edge represents the interaction relationship between different small drones.

[0024] Furthermore, the multi-step small UAV trajectory prediction network includes a trajectory tracking module; step S4 includes the following sub-steps:

[0025] S401: The motion trajectory tracking module is an encoder-decoder neural network based on a long short-term memory module. It combines the interactive features generated by the salient target detection network with the motion features of the small UAV and inputs them into the encoder of the long short-term memory module to extract features.

[0026] S402: The decoder outputs the subsequent motion trajectory tracking and positioning results sequentially; the entire network is trained using mean squared error loss.

[0027] Furthermore, step S5 includes the following sub-steps:

[0028] S50: Perform average displacement error analysis based on the average value of the Euclidean distance between the spatial coordinates of a series of points in the tracked motion trajectory and the spatial coordinates of the points in the actual trajectory.

[0029] Furthermore, the multi-scale nested interaction module covers multiple U-shaped sub-networks and applies multiple intermediate nodes on the path between the encoder and decoder sub-networks. All intermediate nodes are connected to each other, and each node can receive features from its own layer and adjacent layers, thereby forming a nested network.

[0030] Beneficial effects: Based on the historical motion trajectory modeling of small UAVs, this invention applies a multi-step small UAV motion trajectory tracking network based on long short-term memory recurrent neural network to achieve accurate detection and tracking of small UAVs. Attached Figure Description

[0031] Figure 1 This is a flowchart of the deep learning-based small drone detection and tracking method of the present invention;

[0032] Figure 2 This is a schematic diagram of the structure of the deep learning network for target detection in small unmanned aerial vehicles in this invention;

[0033] Figure 3 This is a schematic diagram of the structure of the multi-scale nested interaction module in this invention;

[0034] Figure 4 This is a screenshot of the detection results for a small drone.

[0035] Figure 5 This is a schematic diagram of the multi-step small UAV motion trajectory tracking network in this invention;

[0036] Figure 6 This is a schematic diagram showing the trajectory tracking results and error analysis of a small drone. Detailed Implementation

[0037] The present invention will be further explained below with reference to the accompanying drawings.

[0038] like Figure 1 As shown, the deep learning-based method for detecting and tracking small drones includes the following steps:

[0039] S1: Obtain historical motion video data of the small drone as training sample data, including the following specific steps:

[0040] S101: Use image acquisition equipment to acquire historical motion video image data of small drones.

[0041] S102: Normalize the data of each frame of the video image to [0,1], to obtain a continuous image set data I = {i1,i2,...,i...} t ,...,i n There are n images in total.

[0042] S2: Input the training sample data into the deep learning network for salient target detection of small drones, and learn the target bounding box of the small drone in the image at each time step. The network structure is as follows: Figure 2 As shown, the network comprises four parts: a dual-channel feature extraction module, a multi-scale nested interaction module, a dual-channel aggregation module, and a small UAV position detection head.

[0043] S201: The dual-channel feature extraction module uses 18 convolutional modules, which are then combined into 9 residual modules to extract internal features and boundary detail features, respectively. Each convolutional module contains one convolutional layer, one batch normalization layer, and one ReLU activation function. The corresponding residual module consists of two convolutional modules and a residual connection: the input features processed by the convolutional modules are added to the original input features to obtain the output features of the residual unit.

[0044] The dual-channel feature extraction module receives continuous images from the image set data I, extracts features from the images in I, and obtains an abstract feature map set F(I). I = {i1, i2, ..., i...} t ,…,i n There are n images in total, and the images are sequential in time. t For the t-th image in the image set I, which is also i t-1 The next frame image. n abstract feature maps constitute the abstract feature map set, F(I)={F(i1),F(i2),…,F(i... t ),…,F(i n )},F(i t ) for i t Abstract feature map.

[0045] S202: Due to background clutter in the acquired image, pixels near the boundary of small targets are more likely to be mispredicted, while pixels closer to the center, highlighting the internal consistency of the target, have higher prediction accuracy. Rather than treating these pixels equally, it would be more reasonable to treat them according to their individual characteristics. Based on this, the present invention decouples the small target labels in the acquired image into internal labels and boundary detail labels, such as... Figure 2 The main image and boundary detail image in the image are both auxiliary supervision to improve the overall small target detection quality and boundary definition.

[0046] A traditional U-shaped network consists of an encoder, a decoder, and ordinary skip connections, such as... Figure 3 As shown in (a), the encoder expands the receptive field and extracts high-level information, while the decoder helps restore the size of the feature map, ultimately achieving the same size as the input image and enabling progressive multi-scale feature fusion. Ordinary skip connections act as a bridge, transferring these low-level and high-level features from the encoder subnet to the decoder subnet. The size of distant UAV targets varies significantly, ranging from one pixel (i.e., a point target) to tens of pixels (i.e., an extended target). However, as the number of network layers increases, point targets are easily lost after multiple pooling operations. Therefore, this invention further designs a multi-scale nested interaction module, such as... Figure 3 As shown in (b), the above-mentioned problems in small target detection in complex images can be effectively solved.

[0047] The multi-scale nested interaction module covers multiple U-shaped sub-networks to construct a dense nested structure. Since the optimal receptive field varies greatly for targets of different sizes, these U-shaped sub-networks of varying depths are naturally applicable to targets of different sizes. Based on this, the invention applies multiple intermediate nodes along the path between the encoder and decoder sub-networks, all of which are connected to each other to form a nested network. Each node can receive features from its own layer and adjacent layers, resulting in repeated multi-layer feature fusion. Therefore, small targets can be preserved even in deeper layers, leading to better results.

[0048] like Figure 3 As shown in (b), suppose X i,j Represents a node The output of X is given by where i is the i-th downsampling layer along the encoder, and j is the j-th convolutional layer along the dense block on the lateral jump path. When j = 0, each node only receives features from the dense lateral jump connections, X. i,j The calculation formula is:

[0049] X i,j =Fconv(P(X) i,j ))

[0050] In the formula, Fconv(·) represents the corresponding residual module, which consists of two convolutional modules and a residual connection. P(·) represents the max pooling layer with a stride of 2.

[0051] When j > 0, each node receives features from dense lateral skip connections and four other directions, including two upper-layer downsampling connections, one same-layer connection, and one lower-layer upsampling connection, X. i,j The calculation formula is:

[0052]

[0053] Where U(·) represents the upsampling layer, and [·,·] represents the connection layer.

[0054] S203: This invention develops a dual-channel aggregation module to integrate multi-granularity information. After obtaining high-quality feature maps with internal and boundary details, the dual-channel aggregation module is used to integrate these features. Specifically, firstly, multi-layer feature maps are expanded to the same size using bilinear interpolation. Internal and boundary detail features of the same layer are concatenated. Then, shallow features with rich spatial and contour information and deep features with rich semantic information are connected to generate a global robust feature map. Finally, the feature map enhanced by internal and boundary detail information is added to the global robust feature map to generate the final output feature map.

[0055] S204: The small drone prediction head primarily uses CenterNet to predict the target's bounding box. First, a 3x3 convolution and a 1x1 convolution are applied to the final output feature map to obtain the target's center. Then, regression is used to predict the target's bounding box, such as... Figure 4 As shown in (a) and (b), the entire network is trained using the cross-entropy loss function.

[0056] S3: Model a spatial map of the movement trajectory of a small drone by using the spatial coordinates of the small drone over a period of time.

[0057] S301: Extract the spatial coordinates of all small drones in each frame of an image over a period of time;

[0058] S302: Based on spatial coordinate modeling, a spatial graph representing the interaction relationships between small drones at time t is constructed. Each node in the graph represents a different small drone, and each edge represents the interaction relationship between them.

[0059] S4: Input the spatial map data of the small UAV trajectory in time sequence into a multi-step small UAV motion trajectory tracking network based on a long short-term memory recurrent neural network to accurately detect and track the motion trajectory of the small UAV. The network structure is as follows: Figure 5 As shown, the network includes a trajectory tracking module:

[0060] S401: The motion trajectory tracking module is an encoder-decoder neural network (LSTM) based on a long short-term memory module. It combines the generated interaction features with the motion features of the small UAV. The interaction features are generated by a pre-built salient target detection network, and the motion features are the position coordinates of N small UAVs at a single time point. These features are fed into the encoder of a Long Short-Term Memory (LSTM) module. Each LSTM processes the input features of a moving target i and generates intermediate features r. i Then, the output features m are generated sequentially through the Long Short-Term Memory (LSTM) decoder. i And obtain the subsequent motion trajectory tracking results, such as Figure 6 As shown, the black dots represent the spatial coordinates of the same target at different times during tracking. White represents the spatial coordinates of the drone tag results at different times. The motion trajectory is represented by connecting points at different times using short lines; the entire network is trained using mean squared error loss.

[0061] S5: Perform "average displacement error" analysis based on the tracked motion trajectory of a small UAV.

[0062] S501: Perform average displacement error analysis based on the average value of the Euclidean distance between the spatial coordinates of a series of points in the tracked motion trajectory and the spatial coordinates of the points in the actual trajectory.

[0063] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting and tracking small unmanned aerial vehicles (UAVs) based on deep learning, characterized in that, include: Step S1: Obtain historical motion video data of the small drone as training sample data; Step S2: Input the training sample data into the deep learning network for salient target detection of small UAVs to learn the target bounding box of the small UAV in the image at each time step; Step S3: Model a spatial map of the small drone's motion trajectory using the spatial coordinates of the small drone over a period of time; Step S4: Input the spatial map data of the small UAV's motion trajectory into the multi-step small UAV motion trajectory tracking network based on long short-term memory recurrent neural network in time sequence to achieve accurate detection and tracking of the small UAV's motion trajectory; Step S5: Perform distance error analysis based on the tracked small UAV trajectory; The deep learning network for salient target detection of small UAVs includes a dual-channel feature extraction module, a multi-scale nested interaction module, a dual-channel aggregation module, and a small UAV position detection head; step S2 includes the following sub-steps: S201: The dual-channel feature extraction module uses a convolutional module and a residual module to extract the internal features and boundary detail features of the small UAV, respectively. S202: Decouple the small target labels of the acquired image into internal labels and boundary detail labels. The multi-scale nested interaction module uses multiple nodes to nest and exchange the features finally extracted by the dual-channel feature extraction module. S203: Uses a dual-channel aggregation module to integrate internal features and boundary detail features; S204: The small drone prediction head uses CenterNet to predict the bounding box of the target; The multi-scale nested interaction module covers multiple U-shaped sub-networks and applies multiple intermediate nodes on the path between the encoder and decoder sub-networks. All intermediate nodes are connected to each other, and each node can receive features from its own layer and adjacent layers, thus forming a nested network.

2. The method for detecting and tracking small unmanned aerial vehicles based on deep learning according to claim 1, characterized in that, Step S1 includes: S101: Use image acquisition equipment to acquire historical motion video image data of small drones; S102: Normalize the data of each frame of the video image to [0,1], to obtain the continuous image set data I={ i 1, i 2,…, i t ,…, i n }, total n Zhang image.

3. The method for detecting and tracking small unmanned aerial vehicles based on deep learning according to claim 1, characterized in that, Step S3 includes the following sub-steps: S301: Extract the spatial coordinates of all small drones in each frame of an image over a period of time; S302: Based on spatial coordinate modeling, a spatial graph representing the interaction relationship between small drones at time t is created. Each node in the spatial graph represents a different small drone, and each edge represents the interaction relationship between different small drones.

4. The method for detecting and tracking small unmanned aerial vehicles based on deep learning according to claim 1, characterized in that, The multi-step small UAV motion trajectory tracking network includes a motion trajectory tracking module; step S4 includes the following sub-steps: S401: The motion trajectory tracking module is an encoder-decoder neural network based on a long short-term memory module. It combines the interactive features generated by the salient target detection network with the motion features of the small UAV and inputs them into the encoder of the long short-term memory module to extract features. S402: The decoder outputs the subsequent motion trajectory tracking and positioning results sequentially; the entire network is trained using mean squared error loss.

5. The method for detecting and tracking small unmanned aerial vehicles based on deep learning according to claim 1, characterized in that, Step S5 includes the following sub-steps: S50: Perform average displacement error analysis based on the average value of the Euclidean distance between the spatial coordinates of a series of points in the tracked motion trajectory and the spatial coordinates of the points in the actual trajectory.

Images