A method and device for detecting and tracking unmanned aerial vehicles in complex backgrounds

Abstract

The present invention discloses a method and device for detecting and tracking drones in complex backgrounds, which can be used in the field of drone countermeasure technology. The present invention analyzes the performance of drones in images by using image processing technology, effectively improving the detection accuracy; at the same time, the drone target is detected by an image sensor to further confirm the target category and location information of the drone; and the millimeter wave radar data is integrated into the state estimation, and the tracker's estimate of the target position is corrected based on the target position and speed information obtained by the radar, so as to further enhance the prediction accuracy of the drone trajectory, and the introduction of radar data can reduce the uncertainty of single image detection data when it is blocked or the perspective changes. The present invention uses image recognition technology and radar detection technology to more comprehensively and accurately perceive the existence and dynamic changes of drones. The present invention performs well in complex environments, effectively reduces the false alarm rate, and improves the detection and tracking accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of drone detection and tracking, and in particular to a method and device for drone detection and tracking in complex backgrounds. Background Art

[0002] In recent years, drones have developed rapidly, especially civil drones, which have significantly improved in production and performance. Since 2014, the scale of the global market for civil drones has shown explosive growth. At the same time, the performance of micro drones has also been greatly improved. In addition to enhanced endurance, they also have new functions such as preset path flight, automatic return after loss of control, hovering, intelligent obstacle avoidance, intelligent following and active tracking. These drones can perform tasks such as intelligence collection, surveillance and reconnaissance, and can carry various equipment for specific tasks.

[0003] The popularity of low-cost micro drones has brought new challenges to airport security. Traditional security systems work well against large drones, but are powerless against micro drones. Micro drone systems are easy to spread, fly slowly and at low altitudes, resulting in short detection and response times, which makes existing security systems ineffective in response. In addition, the gradual rise of group combat based on micro drones, especially the rapid development of "swarm" collaborative combat, traditional security systems do not yet have the ability to effectively respond.

[0004] In summary, the increasing popularity of micro-UAVs has made them outstanding in intelligence collection, surveillance and reconnaissance missions, especially in complex environments such as airports, where their role far exceeds that of traditional means. This development trend has made micro-UAVs a new challenge that airport security and management must pay attention to. At present, traditional security systems face difficulties in detecting micro-UAVs, such as difficulty in detection, slow response, low cost-effectiveness, and possible accidental injuries. Therefore, it is necessary to develop means to quickly and accurately detect and effectively track UAVs. Summary of the invention

[0005] The present invention provides a method and device for detecting and tracking drones in complex backgrounds, which can more accurately focus on drone features, improve the accuracy of drone detection, and improve the accuracy of drone flight trajectory prediction.

[0006] On the one hand, the present invention provides a method for detecting and tracking a drone in a complex background, which specifically comprises the following steps:

[0007] Step 1: calibrate the sensor, define the global coordinate system for detection and tracking and the local coordinate system of each sensor, so as to ensure that the data of all sensors can be mapped to the same coordinate system, where the sensors include image sensors and radar sensors, and the sampling frequency of the two sensors is the same;

[0008] Step 2, extracting multiple frames of image data and corresponding radar data of the same scene from the data transmitted by the sensor, and performing coordinate transformation on the extracted data based on the calibrated local coordinate system and global coordinate system, transforming it into the global coordinate system, and obtaining the drone image and radar data corresponding to each frame after the coordinate transformation;

[0009] Step 3: Perform drone detection processing on the same frame of drone image and radar data after coordinate conversion:

[0010] Preprocessing the coordinate-converted drone image to match the input of the configured drone image detection model; then inputting the preprocessed drone image into the drone image detection model to obtain a first detection result of the drone image, including: a detection frame of the candidate target, a target category, and a confidence score;

[0011] Performing candidate target detection on the radar data after coordinate transformation to obtain a second detection result of the radar data, including: a target position and a target speed in the radar data of the candidate target;

[0012] Step 4: Fusing the first detection result and the second detection result of the same frame to obtain the third detection result of each frame of the same scene currently extracted:

[0013] Based on a preset screening number M, the candidate targets corresponding to the first M maximum confidences are screened out from the first detection result;

[0014] Based on the mapped target positions of the detection frames of the screened M candidate targets, candidate target matching is performed in the second detection results, and the third detection results of the current frames are obtained based on the detection frames, target categories, confidence scores, target positions and target speeds in the radar data of the matched candidate targets;

[0015] Step 5, performing UAV tracking fusion on the third detection results of each frame currently extracted to obtain the tracking fusion results of each tracked target UAV;

[0016] Step 6: Output the predicted flight trajectory information of the UAV based on the tracking fusion result of the target UAV.

[0017] Furthermore, in step 1, the radar sensor is a millimeter wave radar.

[0018] Furthermore, in step 1, the global coordinate system and the local coordinate system are set as:

[0019] Based on the origin of the selected global coordinate system, set the front of the origin as the x-axis, the left side of the origin as the y-axis, and the top of the origin as the z-axis;

[0020] The installation position of the sensor is set as the origin of the local coordinate system, the front of the sensor is set as the x-axis, the right of the sensor is set as the y-axis, and the top of the sensor is set as the z-axis.

[0021] Furthermore, the conversion formula between the local coordinate system and the global coordinate system is: ,in, , Represents the coordinate points in the global coordinate system and the local coordinate system, respectively, and the homogeneous transformation matrix , R represents the rotation matrix of the sensor, t represents the translation vector of the sensor .

[0022] Furthermore, in step 3, preprocessing the image data after coordinate transformation specifically includes: converting the image format into a specified image format (such as converting it into a JPEG format), and then normalizing the image size to be consistent with the size of the input data of the drone image detection model.

[0023] Furthermore, in step 3, the drone image detection model includes: a feature extraction network, a feature enhancement network, and a detection head;

[0024] The feature extraction network is used to extract features from the input drone image. It uses multiple convolutional layers to extract feature maps of different scales of the input image, and sends the feature maps of the middle layer to the feature enhancement network for feature fusion to improve the ability and efficiency of feature extraction. This structure can better learn deeper features during the training process and alleviate the problem of gradient disappearance.

[0025] The feature enhancement network uses an orthogonal channel attention mechanism module to fuse feature maps at different stages of the feature extraction network and output three fused features at different scales; the feature maps at different stages of the feature extraction network include: feature maps at three intermediate stages and feature maps outputted by the last layer of the feature extraction network; that is, the feature enhancement network uses a processing form of a path aggregation network-feature pyramid network to fuse feature maps from different stages of the feature extraction network to enhance feature representation capabilities;

[0026] The detection heads include three, which are used to respectively detect and output fusion features of three different scales to obtain a first detection result, that is, the input of each detection head is a fusion feature of one scale, which is used to output the detection frame, target category or confidence score of the candidate target.

[0027] Furthermore, the feature extraction network of the drone image detection model includes: five convolution layers with a convolution kernel of 3×3 and a maximum pooling layer, and starting from the second convolution layer, the feature map output by the convolution layer is connected to the feature enhancement network.

[0028] Furthermore, the feature enhancement network sequentially includes three stacked first enhancement blocks and two stacked second enhancement blocks; wherein the first enhancement block sequentially includes an orthogonal channel attention mechanism module, an upsampling and a connection layer, and the second enhancement block sequentially includes an orthogonal channel attention mechanism module, a 3×3 partial convolution layer and a connection layer;

[0029] The feature maps of the three intermediate stages of the feature extraction network are input into each first enhancement block in reverse order. Each first enhancement block inputs a feature map of the intermediate stage, and the input of the orthogonal channel attention mechanism module of the first first enhancement block is the feature map output by the last layer of the feature extraction network. Each enhancement block upsampling is used to upsample the output map of the corresponding orthogonal channel attention mechanism module to the feature map dimension of the feature map of the intermediate stage of the corresponding feature extraction network, and then spliced ​​through its connection layer and input into the orthogonal channel attention mechanism module of the next first enhancement block; and the feature map output by the connection layer of the third first enhancement block is sent to the first detection head;

[0030] The connection layer of the first second enhanced block is used to concatenate the output map of its 3×3 partial convolution layer and the output feature map of the connection layer of the second first enhanced block to obtain the input feature map of the second second enhanced block, and send the output feature map of the connection layer of the first second enhanced block to the second detection head;

[0031] The connection layer of the second second enhancement block is used to concatenate the output map of its 3×3 partial convolutional layer and the output feature map of the connection layer of the first first enhancement block, and then send it to the third detection head.

[0032] Furthermore, the target categories output by the detection head of the drone detection network model include two categories: fixed-wing and rotary-wing.

[0033] Furthermore, in step 5, the Kalman filter tracking method is used to perform drone tracking processing on the currently extracted multi-frame third detection results, specifically including:

[0034] Define N to represent the number of image frames of the same drone of the same scene;

[0035] For the current scene, a candidate target is randomly selected from one of the N frames of drone images to initialize a Kalman filter tracker to obtain the tracker , and initialize the tracker based on the third detection result of the currently selected candidate target The optimal estimation result includes the detection box, target category, confidence score, and target position and target speed in the radar data. The subscript j is the tracker number.

[0036] Based on the current tracker Perform candidate target tracking on each drone image and radar data: traverse each candidate target and corresponding radar data in N drone images of the same scene in turn, and for the current drone image and radar data, based on the tracker The optimal estimate result of the tracker is obtained The prediction results of candidate targets in the current drone image and radar data; and calculate the detection box of each candidate target in the current drone image and the current tracker The center distance of the detection box of the optimal estimation result, as well as the target position and target speed calculated based on the radar data and the current tracker If the center distance is less than the preset detection box distance threshold and the matching degree with the target position and target speed in the radar data are less than the preset target position threshold and target speed threshold, it means that the current candidate target is the tracker in Matching candidate targets in the current UAV image and radar data, and record the matching candidate targets; update the tracker based on the matching candidate targets in the current UAV image and the matching data of the radar : Tracker based The prediction results, the detection results of the matching candidate targets, and the weighted update tracker of the target speed and target position The optimal estimate result of Continue to track the candidate targets for the next frame of drone image and the corresponding radar data; if the detection frame of each candidate target in the current drone image matches the current tracker If the center distance of the detection frame of the optimal estimation result is greater than or equal to the preset detection frame distance threshold, or the target position is greater than or equal to the preset target position threshold, or the speed is greater than or equal to the preset speed threshold, then a new tracker is randomly selected from the non-matching candidate targets to reinitialize , and then based on the new tracker Perform candidate target tracking processing on each UAV image and radar data until there are no unmatched candidate targets;

[0037] Based on the final optimal estimation result of each tracker, the drone tracking fusion result of each tracker is obtained.

[0038] On the other hand, the present invention also provides a computer device, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement a drone detection and tracking method for complex backgrounds of the present invention.

[0039] The technical solution provided by the present invention brings at least the following beneficial effects:

[0040] The present invention can be used in the field of drone countermeasure technology. According to the characteristics of drone detection, the proportion of pixels occupied by drones in images is small. In drone detection processing, drone images are fully utilized, and more accurate and reliable drone category detection is achieved through an image detection method based on deep learning. The performance of drones in images is analyzed by using image processing technology, which effectively improves the detection accuracy; at the same time, drone target detection is performed by image sensors to further confirm the target category and position information of drones (determined based on the position mapped by the detection frame), so that the detection and tracking method of the present invention has good robustness, can rely on visual information to intuitively identify the target, and enhance the perception of drones. In addition to using image detection results, the present invention also integrates millimeter-wave radar data into state estimation, and corrects the target position estimation of the tracker based on the target position and speed information obtained by the radar to further enhance the prediction accuracy of the drone trajectory, and the introduction of radar data can reduce the uncertainty of single image detection data when it is blocked or the perspective changes. The present invention uses image recognition technology and radar detection technology to more comprehensively and accurately perceive the existence and dynamic changes of drones. Compared with the existing processing methods, the present invention performs well in complex environments, effectively reduces the false alarm rate, and improves the overall detection and tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of a processing flow of a method for detecting and tracking drones in complex backgrounds provided for implementation of the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the drone image detection model used in an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the structure of the orthogonal channel attention mechanism module implemented in the present invention. DETAILED DESCRIPTION

[0045] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0046] An embodiment of the present invention provides a method for detecting and tracking drones in complex backgrounds. The method combines image detection based on deep neural networks and radar detection technology, and can accurately identify drones and perform effective flight trajectory prediction.

[0047] like Figure 1 As shown, as a possible implementation method, the specific implementation steps of a method for detecting and tracking a drone in a complex background provided by an embodiment of the present invention include:

[0048] Step S1, calibrating each integrated sensor.

[0049] Due to the differences in spatial positions and sampling frequencies of different sensors, the perception information of the same target may not match between different sensors. To solve this problem, the perception information of different sensors needs to be calibrated. Define the global coordinate system of the system and the local coordinate system of each sensor. Ensure that the data of all sensors can be mapped to the same coordinate system. The sensors include image sensors and radar sensors, and the sampling frequencies of the two are the same. Millimeter wave radar sensors are preferred.

[0050] Step S2: performing coordinate transformation on the data collected by the sensor.

[0051] Multi-frame image data and corresponding radar data of the same scene are extracted from the data transmitted by the sensor, and the extracted data are transformed into the global coordinate system based on the calibrated local coordinate system and the global coordinate system to obtain the drone images and radar data corresponding to each frame after the coordinate transformation.

[0052] Step S3: Perform drone detection on the same frame of drone image and radar data.

[0053] The coordinate-converted drone image is preprocessed to match the input of the configured drone image detection model; the preprocessed drone image is then input into the drone image detection model to obtain a first detection result, including: a detection box, a target category, and a confidence score of the candidate target.

[0054] The radar data after coordinate transformation is subjected to candidate target detection to obtain a second detection result, including: a target position and a target speed in the radar data of the candidate target.

[0055] Step S4: Fusing the two detection results to obtain a third detection result for each frame.

[0056] Based on the preset screening number M, for the same frame of sensor data, the candidate targets corresponding to the first M maximum confidences are screened out from the first detection result; based on the target positions corresponding to the detection frames of the screened M candidate targets, candidate target matching is performed in the second detection result, and the third detection result of each frame is obtained based on the detection frames, target categories, confidence scores, target positions and target speeds of the matched candidate targets.

[0057] Step S5: Performing UAV tracking fusion based on the third detection result, that is, fusing multiple frames of the third detection results of the same UAV through the configured tracking method to obtain a tracking fusion result for obtaining the predicted flight trajectory information of the UAV.

[0058] Step S6: Output the predicted flight trajectory information of the UAV. For example, based on the tracking fusion result, the least square method is used to obtain the predicted flight trajectory information of the UAV and output it in real time. In a short period of time, the flight of the UAV can be regarded as an approximate linear motion, so polynomial fitting can be used to accurately estimate the future trajectory of the UAV.

[0059] As a possible implementation, the sensor calibration process in step S1 of the embodiment of the present invention is to ensure that the data of all sensors can be mapped to the same global coordinate system, thereby ensuring the accuracy of multi-sensor fusion, which specifically includes:

[0060] S101, define a global coordinate system and a local coordinate system.

[0061] First, you need to define a reference global coordinate system, which is usually a fixed coordinate system in the installation scene. The center of the installation scene is the origin of the global coordinate system, the front is the x-axis, the left is the y-axis, and the top is the z-axis.

[0062] Each sensor has its own local coordinate system. Generally speaking, the local coordinate system of the sensor takes the installation position of the sensor as the origin, the front of the sensor as the x-axis, the right as the y-axis, and the upward as the z-axis.

[0063] S102, measurement of sensor installation position and direction.

[0064] Measure the installation position of each sensor and determine its relative position in the global coordinate system (usually expressed in three-dimensional coordinates). ). To determine the orientation of the sensor, measure its rotation angle relative to the global coordinate system. Usually a rotation matrix or quaternion is used to represent the rotation of the sensor relative to the global coordinate system.

[0065] S103, determining the transformation matrix of the sensor.

[0066] To transform the data in the sensor's local coordinate system into the global coordinate system, we first need to know the translation of the sensor relative to the global coordinate system, which can be represented by a translation matrix T.

[0067]

[0068] in is the position of the sensor in the global coordinate system.

[0069] Next, we need to determine the orientation of the sensor, which can be represented by the rotation matrix R. The rotation matrix is ​​defined according to the rotation angle of the sensor relative to the global coordinate system. Assuming the rotation angles are α, β, and γ (the rotation angles around the x, y, and z axes, respectively), we can combine the rotation matrices:

[0070]

[0071] The final rotation matrix R is the combination of these single-axis rotations:

[0072]

[0073] The rotation and translation are combined to form a homogeneous transformation matrix H, which is used to transform points in the sensor's local coordinate system to the global coordinate system.

[0074]

[0075] Where R is the rotation matrix and t is the translation vector .

[0076] Then, based on the homogeneous transformation matrix H obtained above, the data collected by the sensor is transformed to apply the transformation matrix to the sensor data: the data obtained by each sensor is transformed from the local coordinate system to the global coordinate system using the homogeneous transformation matrix H. The point of the sensor in the local coordinate system Can be converted to a point in the global coordinate system through the coordinate conversion formula : .

[0077] The mapping accuracy of coordinate transformation can also be further improved through calibration verification. For example, by using a known calibration object, verify that all sensor data is correctly mapped to the same global coordinate system. Use a calibration plate or a known structure for comparison to verify that the data of all sensors is consistent in the global coordinate system.

[0078] As a possible implementation method, in the embodiment of the present invention, the drone image detection model is as follows: Figure 2 As shown, it consists of three parts: feature extraction network, feature enhancement network and detection head.

[0079] The structure of the feature extraction network is designed as follows: it contains five 3×3 convolutional layers and one maximum pooling layer in sequence. Through these convolutional layers, the model can effectively capture the detailed features in the input image, and the maximum pooling layer is used to reduce the dimension of the feature map and reduce the subsequent calculation amount. It is worth noting that starting from the second convolutional layer, the convolutional feature map will be connected to the feature enhancement network part to prepare for the subsequent feature fusion of the model.

[0080] In the feature enhancement network, the feature enhancement network includes three stacked first enhancement blocks and two stacked second enhancement blocks in sequence; wherein the first enhancement block includes an orthogonal channel attention mechanism module, an upsampling layer, and a connection layer in sequence, and the second enhancement block includes an orthogonal channel attention mechanism module, a 3×3 partial convolution layer, and a connection layer in sequence. Among them, the orthogonal channel attention module can dynamically assign weights to different feature channels, thereby emphasizing feature information that is critical to target detection.

[0081] The connection layer of the first enhancement block is used to fuse the upsampled feature map with the feature map output by the feature extraction network to provide richer feature information for the detection head. Specifically, the output feature map of the second convolutional layer of the feature extraction network is sent to the connection layer of the third first enhancement block, the output feature map of the third convolutional layer of the feature extraction network is sent to the connection layer of the second first enhancement block, and the output feature map of the fifth convolutional layer of the feature extraction network is sent to the connection layer of the first first enhancement block. Each upsampling layer of the first enhancement block is used to upsample the output feature map of its corresponding orthogonal channel attention mechanism module to the feature map dimension of the output feature map of the convolutional layer corresponding to the upsampled feature extraction network. In addition, the two second enhancement blocks of the feature enhancement network are used to further improve the expressiveness of the features and the accuracy of detection. The input feature map of the orthogonal channel attention mechanism module of the first enhancement block is the feature map output by the connection layer of the third first enhancement block, and the feature map output by the connection layer of the third first enhancement block is sent to the first detection head to obtain the detection result of the micro-scale target; the connection layer of the first second enhancement block is used to splice the output map of its 3×3 partial convolution layer and the output feature map of the connection layer of the second first enhancement block to obtain the input feature map of the second second enhancement block, and the output feature map of the connection layer of the first second enhancement block is sent to the second detection head to obtain the detection result of the small-scale target; the connection layer of the second second enhancement block is used to splice the output map of its 3×3 partial convolution layer and the output feature map of the connection layer of the first first enhancement block, and then sent to the third detection head to obtain the detection result of the medium-scale target. Among them, the specific scale values ​​of the three target scales (micro-scale, small scale and medium scale) can be set based on the actual scene and task requirements, that is, the three detection heads are used to realize target detection for targets of three scales, and the scale of the feature map input by the detection head gradually increases from the first to the third.

[0082] The three detection heads are the last part of the network and are responsible for generating the final detection results. Its main task is to predict the location and category of the target from the output of the feature enhancement network. The detection head further processes the enhanced feature map, integrates the location information (i.e., bounding box coordinates) and category information (e.g., the type of drone) of the target, and finally outputs the detection results. These detection results will be used for subsequent drone target recognition and tracking to ensure that the system can accurately identify drones in complex backgrounds.

[0083] As a possible implementation, in the embodiment of the present invention, the orthogonal channel attention mechanism module is as follows Figure 3 As shown, it consists of two stages. The first stage is to initialize a random filter whose size matches the size of the input feature map to be processed, and then use Gram- (Gram-Schmidt) orthogonalization orthogonalizes these filters to obtain orthogonalized filters; the second stage is to use these filters to extract compressed vectors. First, the orthogonalized filter is used to squeeze the input feature map vector X (that is, the input feature map of the orthogonal channel attention mechanism module), and then the squeezed feature map is subjected to a one-dimensional convolutional layer to report information interaction, that is, a convolution operation is performed on the squeezed feature map. The convolution operation result is then mapped by the Sigmoid activation function and used as a weight to weight the input feature map vector X to obtain a weighted feature map, so as to achieve the purpose of reproducing the weighted original feature map. Finally, the weighted feature map is added to the input feature map (that is, residual connection) to obtain the output feature map of the orthogonal channel attention mechanism module.

[0084] The orthogonal channel attention mechanism module used in the embodiment of the present invention is used as a computing unit to extract and emphasize important features. Define the feature map vector is the feature tensor, where C represents the number of channels, H and W represent the height and width respectively. is a real number domain. The channel attention mechanism module is used to calculate a vector , which is used to highlight the most relevant channels in X. The output feature map of the channel attention mechanism module is expressed as In this invention, orthogonality is used to enhance the channel attention mechanism. Specifically, a filter of appropriate dimension is first randomly selected. , and then apply the Gram-Schmidt orthogonalization filter. Define represents the orthogonal filter, and the compression process can be specifically expressed as:

[0085]

[0086] in, It represents the feature map compressed by the orthogonal filter. h and w are the pixel indexes in the height and width directions of the feature map respectively. , Represents the orthogonal filter and input feature map respectively In pixels The value at .

[0087] Then, local information interaction is performed in the channel dimension through a one-dimensional convolution operation. The size k of the convolution kernel is usually dynamically determined according to the number of channels C, and k is an odd number.

[0088]

[0089] in, is the activation function, is the output of a one-dimensional convolution operation.

[0090] The vector E after one-dimensional convolution is used as the scaling factor to re-weight the original feature map, that is, the input feature map of the weighted orthogonal channel attention mechanism module, and the output feature map of the orthogonal channel attention mechanism module can be obtained:

[0091]

[0092] During processing, the Sigmoid activation function is used to limit the scaling factor E between 0 and 1 as the spatial attention weight. Applying the spatial attention weight to each channel of the convolutional network branch can obtain a weighted feature map.

[0093] As a possible implementation method, in an embodiment of the present invention, the training of the constructed drone image detection model includes:

[0094] (1) Obtain original image data information.

[0095] UAV image data and manual annotation information of target categories.

[0096] Preferably, in an embodiment of the present invention, the original sensor data is obtained. The drone images can be obtained by collecting drone data in different scenes through sensors, and the drone images are manually labeled with drone categories through labeling tools.

[0097] (2) Preprocess the original image data, that is, preprocess the drone image data and the manually labeled information of the target category. It is a crucial step to enhance the drone dataset image. Random rotation, scaling, flipping, cropping, and introducing occlusion can effectively increase the diversity of the data. Obtain the drone image and its corresponding target category label.

[0098] Convert the drone image data from the original format (such as PNG, BMP, etc.) to JPEG format for easy storage and processing. At the same time, convert the category information marked as the drone target area to obtain the corresponding label file and store it.

[0099] Based on steps (1) and (2), a training set for training the drone image detection model can be obtained.

[0100] (3) Drone image detection model training.

[0101] The drone image detection model constructed by the embodiment of the present invention pays more attention to accurately detecting drone targets, especially when the field of view is large, the drone target will be relatively small, so the small target detection ability of the model is more important. Therefore, the embodiment of the present invention skips the high-resolution feature output of the first convolutional layer, because these high-resolution features may not be necessary for general target detection, while small target detection requires higher spatial resolution. Based on the number of additional sampling layers, the loss of spatial information is reduced. Combined with the feature enhancement network and detection head set in this embodiment, accurate detection of drones is achieved. During training, the model parameters of the constructed drone image detection model are optimized based on the configured classification recognition loss function and training set. When the preset training convergence conditions are met (the number of training times reaches the upper limit or the loss function value converges), it stops to obtain a trained drone image detection model.

[0102] In step S5, in addition to the candidate target detection results in the image, the detection data of the millimeter-wave radar is also combined for fusion processing. First, the image detection method based on deep learning performs target detection processing on each drone image to obtain several drone target detection results, which include detection boxes, drone categories, and confidence scores. At the same time, the millimeter-wave radar detects the same scene to obtain the location information, speed, and other physical characteristics of the target. By adjusting the confidence threshold of each image, the recommended number of candidate drones for each image in the same scene is controlled to be a preset value M, and these candidate drone targets are further verified based on the detection data of the millimeter-wave radar.

[0103] In step S5, the drone image is input into the preset target detection model, and the detection results of the millimeter-wave radar are used to assist in screening out more accurate candidate targets. The drone detection results output by the model are combined with radar information, such as the target's motion trajectory, speed, and corresponding confidence threshold. Subsequently, the Kalman filter tracking method is used to perform multi-source fusion of the image and radar detection results, aiming to determine whether the candidate drones in multiple images and the targets detected by the radar are the same target, further improving the accuracy of detection. The final fusion result includes the location, category, confidence, motion information of the candidate drone, and the image and radar detection number in which it is located.

[0104] As a possible implementation, in step S5 of the embodiment of the present invention, a Kalman filter tracking method is used to fuse multiple images and target detection results of the same candidate drone on the millimeter wave radar. The specific process is as follows:

[0105] S501, multiple images of the same scene and the corresponding millimeter-wave radar data are input into the drone target detection and fusion model to obtain several drone detection results for each image and the detection results of the millimeter-wave radar. Specifically, N (N≥1) images (I0, I1, ..., IN-1) of the same scene and the corresponding radar data are input into the fusion model, where Ii represents the i-th (i=0,…,N-1) image and Ri represents the millimeter-wave radar data of the i-th frame. Based on the output of the drone detection model and the radar fusion model, the drone detection results in each image and the corresponding radar data are obtained. Then, based on the set confidence threshold, the valid results are screened out from the image detection results and the radar detection results, and matched.

[0106] The detection result of each image is recorded as ,in:

[0107]

[0108] The millimeter wave radar detection result is recorded as ,in:

[0109]

[0110] in, and They represent the image detection result and millimeter-wave radar detection result of the mth (m=0,…,M-1) candidate drone, respectively, and M represents the number of candidate drones in the image.

[0111] Image detection results The detection box (also called target bounding box) is expressed as:

[0112]

[0113] in, and Respectively represent the left and right sides of the detection box, and Represents the coordinates of the upper and lower sides of the bounding box respectively.

[0114] Millimeter wave radar detection results include the target location of the drone and target speed .

[0115] In order to facilitate the calculation of Kalman filter tracking, the target bounding box Convert to center point coordinates And width and height , the conversion formula is as follows:

[0116]

[0117] Millimeter wave radar detection results Target location in and target speed , directly used as input variables in tracking.

[0118] After conversion, the comprehensive detection results of candidate drones can be described as:

[0119]

[0120] in, Indicates the category of the drone. Represents the confidence score.

[0121] S502: Initialize the Kalman filter tracker.

[0122] Initialize the Kalman filter tracker. The principle of Kalman filter tracking UAV targets is mainly composed of two key formulas: one is the two prediction formulas used to generate the corresponding candidate UAV prediction results; the other is the update formula used to update the candidate UAV tracking results on the current image. In the prediction formula, in addition to using the image detection results, the millimeter wave radar data is also integrated into the state estimation, that is:

[0123]

[0124]

[0125] ,

[0126] in, , Respectively represent the current frame and previous frame -1 The state estimate of the candidate target of the image, Represents the compensation value of radar data for the estimated position of the drone, based on the millimeter wave radar The information corrects the current tracker's estimate of the target position, further enhancing the prediction accuracy. The introduction of radar data can reduce the uncertainty of single image detection data when it is blocked or the perspective changes. It represents the control amount of the prediction result, and the matrices A and B represent the state transfer matrix and the control input matrix respectively. The dimensions of the control amount, matrix A and matrix B depend on the number of candidate drones set (i.e. the recommended number of candidate drones), so the control amount is an M-dimensional column vector, and matrices A and B are both square matrices of dimension M×M.

[0127] The covariance between the detection result of the current image and the prediction result of the tracker can be expressed as:

[0128]

[0129] ,

[0130] in, , They represent the covariance of the candidate drone detection results of the current frame and the previous frame image and the optimal estimation result of the tracker (with a dimension of M×M), and Q represents the noise matrix of the Kalman filter tracker, with a dimension of M-dimensional column vector. Represents the millimeter-wave radar detection noise matrix, which can effectively reduce the estimation bias caused by radar detection error, and its dimension is M×M.

[0131] The update formula for updating the candidate drone tracking results on the current image is as follows: based on the weighted prediction results of the current tracker and the candidate drone detection results of the current image, the optimal estimation result of the tracker is updated, specifically:

[0132] Calculate the Kalman gain of the current image:

[0133]

[0134] ,

[0135] Among them, Kalman gain It is used to control the influence ratio of detection results and tracker prediction results when generating the optimal estimation result. The value range is [0,1]. H represents the state observation matrix, which is an M×M square matrix. R represents the noise matrix of the detection system, which is an M-dimensional column vector.

[0136] Update the optimal estimate of the current tracker based on the detection result on the current match:

[0137]

[0138] in, Indicates the detection box attribute information (position and size) of the currently matched candidate drone. is the frame index. The optimal estimation result is composed of the weighted prediction result of the tracker and the detection result of the drone detection model. The weight is the Kalman gain , The larger the value, the higher the tracker's confidence in the detection result.

[0139] Kalman Gain , state observation matrix H and covariance Satisfy between:

[0140]

[0141]

[0142] in, It represents the covariance between the detection result of the current image and the optimal estimation result of the tracker, and I is the M×M unit matrix.

[0143] S503: Calculate the center distance CD (Center Distance) between the detection frame of each candidate drone in the current image and the optimal estimation result of the tracker:

[0144]

[0145] in, , are the mapping target positions of the detection frames of the previous target and the current target respectively (e.g., taking the center point position coordinates of the detection frame, , are the target positions of the previous target and the current target in the radar data, respectively, and λ is the weight coefficient, which is used to balance the influence of the image and radar detection results. If the center distance CD is less than the preset threshold, it can be considered that the current image and radar detection results match successfully.

[0146] S504: If the distance CD is less than the empirical threshold, it means that the tracker has matched the current candidate UAV, and the UAV is a matching candidate for the tracker; the tracker is updated based on the matching candidate UAV.

[0147] S505: If the distance CD is greater than or equal to the empirical threshold, it means that the tracker has failed to match the candidate drone, and a new tracker needs to be initialized. : Randomly select one from the non-matching candidate drones and reinitialize the new tracker .

[0148] S506: When all candidate drones are processed, the final fusion result is output.

[0149] In an exemplary embodiment, the process of obtaining the predicted flight trajectory information of the UAV using the least squares method in step S6 of the embodiment of the present invention can find an optimal polynomial function through historical data fitting to approximate the historical trajectory of the UAV and capture the trend and pattern of the UAV movement.

[0150] The least square method is a classic parameter estimation method, which is fast and simple to calculate. Therefore, this embodiment uses a polynomial fitting model based on least squares to approximate the target trajectory, and optimizes the algorithm to improve the accuracy and speed of prediction.

[0151] Given that there are k different points among the positions where the target has passed most recently, find a polynomial of degree m (m < k) such that its value at each point is as close as possible to the corresponding , where the subscript i is the index of the position point.

[0152] Let the polynomial to be found be:

[0153]

[0154] where, ~ are the polynomial coefficients of the coordinate x, ~ are the polynomial coefficients of the coordinate y. By the least squares method, the coefficient matrix A (composed of all polynomial coefficients of coordinates x and y) can be obtained. From this coefficient matrix A, the flight trajectory of the target in the next short period of time can be predicted, and the position at a future moment n can be expressed by the formula:

[0155]

[0156] where, represents the predicted position of the i-th position point at the n-th moment.

[0157] In an exemplary embodiment, the embodiment of the present invention further provides a computer device, which includes a processor and a memory, and at least one computer program is stored in the memory. The at least one computer program is loaded and executed by one or more processors to implement any one of the above methods for detecting and tracking an unmanned aerial vehicle in a complex background.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0159] The above are only some embodiments of the present invention. For those of ordinary skill in the art, without departing from the inventive concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. A method for detecting and tracking drones in complex backgrounds, characterized in that: The following steps are involved: Step 1: calibrate the sensor to define the global coordinate system for detection and tracking and the local coordinate system of each sensor, where the sensor includes an image sensor and a radar sensor, and the sampling frequency of the two sensors is the same; Step 2, extracting multiple frames of image data and corresponding radar data of the same scene from the data transmitted by the sensor, and performing coordinate transformation on the extracted data based on the calibrated local coordinate system and global coordinate system, transforming it into the global coordinate system, and obtaining the drone image and radar data corresponding to each frame after the coordinate transformation; Step 3: Perform drone detection processing on the same frame of drone image and radar data after coordinate conversion: Preprocess the coordinate-converted drone images to match the input of the configured drone image detection model; Then, the preprocessed drone image is input into the drone image detection model to obtain the first detection result of the drone image, including: the detection frame of the candidate target, the target category and the confidence score; Performing candidate target detection on the radar data after coordinate transformation to obtain a second detection result of the radar data, including: a target position and a target speed in the radar data of the candidate target; Step 4: Fusing the first detection result and the second detection result of the same frame to obtain the third detection result of each frame of the same scene currently extracted: Based on a preset screening number M, the candidate targets corresponding to the first M maximum confidences are screened out from the first detection result; Based on the mapped target positions of the detection frames of the screened M candidate targets, candidate target matching is performed in the second detection results, and the third detection results of the current frames are obtained based on the detection frames, target categories, confidence scores, target positions and target speeds in the radar data of the matched candidate targets; Step 5, performing UAV tracking fusion on the currently extracted multi-frame third detection results to obtain the tracking fusion results of each tracked target UAV; Step 6: output the predicted flight trajectory information of the UAV based on the tracking fusion result of the target UAV; Among them, in step 5, the Kalman filter tracking method is used to perform drone tracking fusion on the currently extracted multi-frame third detection results, specifically including: Define N to represent the number of image frames of the same drone of the same scene; For the current scene, a candidate target is randomly selected from one of the N frames of drone images to initialize a Kalman filter tracker to obtain the tracker , and initialize the tracker based on the third detection result of the currently selected candidate target The optimal estimation result includes the detection box, target category, confidence score, target position and target speed. The subscript j is the tracker number. Based on the current tracker Perform candidate target tracking and fusion processing on each drone image and radar data: traverse each candidate target and corresponding radar data in N drone images of the same scene in turn, and for the current drone image and radar data, based on the tracker The optimal estimate result of the tracker is obtained The prediction results of candidate targets in the current drone image and radar data; and calculate the detection box of each candidate target in the current drone image and the current tracker The center distance of the detection box of the optimal estimation result, as well as the target position and target speed calculated based on the radar data and the current tracker If the center distance is less than the preset detection box distance threshold and the matching degree with the target position and target speed in the radar data is less than the preset target position and target speed threshold, it means that the current candidate target is the tracker in Matching candidate targets in the current UAV image and radar data, and record the matching candidate targets; update the tracker based on the matching candidate targets in the current UAV image and the matching data of the radar : Tracker based The prediction results, the detection results of the matching candidate targets, and the weighted update tracker of the target speed and distance The optimal estimate result of Continue to track the candidate targets for the next frame of drone image and the corresponding radar data; if the detection frame of each candidate target in the current drone image matches the current tracker If the center distance of the detection frame of the optimal estimation result is greater than or equal to the preset detection frame distance threshold, or the target position in the radar data is greater than or equal to the preset target position threshold, or the target speed in the radar data is greater than or equal to the preset speed threshold, then a new tracker is randomly selected from the non-matching candidate targets to reinitialize , and then based on this new tracker Perform candidate target tracking processing on each UAV image and radar data until there are no unmatched candidate targets; Based on the final optimal estimation result of each tracker, the tracking fusion result of the target UAV of each tracker is obtained.

2. A method for detecting and tracking drones in complex backgrounds as claimed in claim 1, characterized in that: In step 1, the global coordinate system and the local coordinate system are set as: Based on the origin of the selected global coordinate system, set the front of the origin as the x-axis, the left side of the origin as the y-axis, and the top of the origin as the z-axis; The installation position of the sensor is set as the origin of the local coordinate system, the front of the sensor is set as the x-axis, the right of the sensor is set as the y-axis, and the top of the sensor is set as the z-axis.

3. A method for detecting and tracking drones in complex backgrounds as claimed in claim 2, characterized in that: The conversion formula between the local coordinate system and the global coordinate system is: ， in, , Represents the coordinate points in the global coordinate system and the local coordinate system, respectively, and the homogeneous transformation matrix , R represents the rotation matrix of the sensor, t represents the translation vector of the sensor .

4. The method for detecting and tracking a drone in a complex background as claimed in claim 1, characterized in that: In step 3, preprocessing the image data after coordinate transformation specifically includes: converting the image format into a specified image format, and then performing image size normalization processing.

5. The method for detecting and tracking a drone in a complex background according to claim 1, characterized in that: In step 3, the drone image detection model includes: feature extraction network, feature enhancement network and detection head; Among them, the feature extraction network is used to extract features from the input drone image; The feature enhancement network uses the orthogonal channel attention mechanism module to fuse the feature maps of different stages of the feature extraction network and output three fused features of different scales; the feature maps of different stages of the feature extraction network include: the feature maps of the three intermediate stages and the feature map output by the last layer of the feature extraction network; The detection heads include three, which are used to respectively detect and output the fusion features of three different scales to obtain a first detection result, including: a detection frame of the candidate target, a target category and a confidence score.

6. A method for detecting and tracking drones in complex backgrounds as claimed in claim 5, characterized in that: The feature extraction network of the drone image detection model includes five convolutional layers with a convolution kernel of 3×3 and a maximum pooling layer. Starting from the second convolutional layer, the feature map output by the convolutional layer is connected to the feature enhancement network.

7. The method for detecting and tracking a drone in a complex background as claimed in claim 5, characterized in that: The feature enhancement network includes three stacked first enhancement blocks and two stacked second enhancement blocks in sequence; wherein the first enhancement block includes an orthogonal channel attention mechanism module, an upsampling and a connection layer in sequence, and the second enhancement block includes an orthogonal channel attention mechanism module, a 3×3 partial convolution layer and a connection layer in sequence; The feature maps of the three intermediate stages of the feature extraction network are input into each first enhancement block in reverse order. Each first enhancement block inputs a feature map of the intermediate stage, and the input of the orthogonal channel attention mechanism module of the first first enhancement block is the feature map output by the last layer of the feature extraction network. Each enhancement block upsampling is used to upsample the output map of the corresponding orthogonal channel attention mechanism module to the feature map dimension of the feature map of the intermediate stage of the corresponding feature extraction network, and then spliced ​​through its connection layer and input into the orthogonal channel attention mechanism module of the next first enhancement block; and the feature map output by the connection layer of the third first enhancement block is sent to the first detection head; The connection layer of the first second enhanced block is used to concatenate the output map of its 3×3 partial convolution layer and the output feature map of the connection layer of the second first enhanced block to obtain the input feature map of the second second enhanced block, and send the output feature map of the connection layer of the first second enhanced block to the second detection head; The connection layer of the second second enhancement block is used to concatenate the output map of its 3×3 partial convolutional layer and the output feature map of the connection layer of the first first enhancement block, and then send it to the third detection head.

8. The method for detecting and tracking a drone in a complex background as claimed in claim 1, characterized in that: The target categories output by the detection head of the drone detection network model include two categories: fixed-wing and rotary-wing.

9. A computer device comprising a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the steps of a method for detecting and tracking a drone in a complex background as described in any one of claims 1 to 8.

Images