Unmanned aerial vehicle detection tracking method based on multi-modal image

Abstract

The invention discloses an unmanned aerial vehicle detection and tracking method based on a multi-modal image, and belongs to the field of target detection and tracking. The unmanned aerial vehicle in the air is detected through the combined action of multiple modes, so that the reliability of unmanned aerial vehicle detection in various environments is realized, and identification and classification of the unmanned aerial vehicle are realized; in the tracking stage, a prediction structure in which LSTM and a Kalman filter are fused is introduced, so that the state modeling capability under the conditions of nonlinear motion and shielding of a target is enhanced, and the stability and continuity of tracking are improved; a fusion multi-stage matching strategy is adopted to effectively improve the matching accuracy and robustness in a complex environment; meanwhile, a confidence shunt strategy and a trajectory management mechanism are combined, so that the response speed to a new target and the fault-tolerant capability to a lost target are improved; through combination of global motion estimation and height information judgment, accurate correction and target distinguishing of the track of the unmanned aerial vehicle are realized, aliasing and misjudgment are avoided, and the detection and tracking capability of the unmanned aerial vehicle is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and tracking, and in particular to a method for detecting and tracking unmanned aerial vehicles (UAVs) based on multimodal images. Background Art

[0002] Traditional drone identification is carried out through radar monitoring, optical monitoring, sound monitoring, electromagnetic monitoring and other methods, but these methods have certain defects to a greater or lesser extent. Radar monitoring cannot effectively distinguish between drones and other flying objects, optical monitoring is limited by weather and lighting conditions, sound monitoring is easily interfered by environmental noise, and electromagnetic monitoring will be affected by environmental electromagnetic interference.

[0003] To address these issues, some studies have used acoustic signal and image fusion to achieve drone identification and positioning. However, this approach is susceptible to noise interference, leading to problems with initial recognition and insufficient recognition and positioning accuracy. Other studies have used radar and image recognition platforms to collaborate and synchronize bilateral drone target recognition through a target matching handshake. When a single path is missing, collaborative path completion is performed to achieve complementary recognition. When both paths are missing, matching path completion is performed to achieve complete tracking of the target path. The flight status of the drone target is determined by infrared image data before and after the target is lost. Although this approach achieves complete target path tracking and improves recognition by fusing radar and image modalities, the characteristics of the two modalities differ significantly, making it impossible to guarantee that both paths will function effectively in complex environments. Furthermore, operating a single path significantly impacts detection performance.

[0004] Therefore, in order to meet the needs of drone detection in complex environments, it is necessary to propose a drone detection and tracking technology based on multimodal images to achieve real-time target detection in various complex environments. Summary of the Invention

[0005] The present invention proposes a drone detection and tracking method based on multimodal images to solve the problems of poor recognition and detection of "low, small, and slow" drones in complex environments, poor multimodal feature fusion, and inadequate target tracking. It can realize real-time discovery, positioning and tracking of "low, small, and slow" drones in complex environments, provide a detection and tracking method for drones near the ground in complex backgrounds, and improve the robustness of tracking drones.

[0006] The technical solution adopted by the present invention is: a drone detection and tracking method based on multimodal images, the method comprising the following steps:

[0007] Step 1: Input a multimodal UAV image sequence acquired by a millimeter-wave radar, a white-light camera, and an infrared camera; the infrared camera includes a long-wave infrared camera and a short-wave infrared camera; and the modal images of the multimodal UAV image sequence include: a radar point cloud image, a white-light image, and two-band infrared images.

[0008] Step 2: Reconstruct each frame of the multimodal drone image sequence in sequence through the image restoration module to remove the degradation of each modality image and obtain a reconstructed multimodal drone image;

[0009] Step 3: Input the reconstructed multimodal drone image sequence into the target detection network, extract image features from each input image frame in turn, and obtain multimodal features for each image frame. The multimodal features are then fused and processed for drone detection to obtain drone detection results for each image frame, including: detection box, drone category, and confidence score.

[0010] Among them, the image feature information of the radar point cloud image includes the spatial state information of the UAV; the image features of the white light image and the infrared light image include background features and detail features;

[0011] During multimodal fusion, the target detection network fuses the background features of the white light image and the infrared image to obtain the first fused feature. The image feature information of the radar point cloud image is obtained through the attention mechanism and then fused with the detail features of the white light image and the infrared image to obtain the second fused feature.

[0012] Obtaining a multimodal fusion feature of each image frame based on the first fusion feature and the second fusion feature;

[0013] Step 4: Based on the drone detection results of each image frame, multi-target tracking is performed using the cascaded Hungarian algorithm based on motion priors, and the trajectory tracking results of multiple drone targets are output;

[0014] When tracking multiple targets, a long short-term memory network (LSTM) is combined with a Kalman filter to predict nonlinear motion trends. The output of LSTM is used as the state input of the Kalman filter, and the target trajectory information of each image frame is predicted through the Kalman filter.

[0015] Furthermore, the image restoration module is a deep neural network based on the Transformer module.

[0016] Furthermore, the network structure of the image restoration module includes a downsampling branch, an upsampling branch, and a reconstruction branch;

[0017] The downsampling branch includes, in sequence: a convolutional layer with a 3×3 convolution kernel, a first Transformer module, a first downsampling layer, a second Transformer module, a second downsampling layer, a third Transformer module, a third downsampling layer, and a fourth Transformer module. The output feature map size of the convolutional layer of the downsampling branch is denoted as H×W×C, where H and W represent the height and width of the output feature map, and C represents the number of channels. The output feature map sizes of the first, second, and third Transformer modules are H / 2×W / 2×2C, H / 4×W / 4×4C, and H / 8×W / 8×8C, respectively.

[0018] The upsampling branch includes: the first upsampling layer, the first convolutional layer, the Transformer module, the second upsampling layer, the second convolutional layer, the multi-head conversion attention layer and the gated feedforward network (GFN); the convolution kernels of the two convolutional layers are both 1×1; the input feature map of the first upsampling layer is the output feature map of the fourth Transformer module of the downsampling branch, which is upsampled to H / 4×W / 4×4C; the input feature map of the first convolutional layer is: the pixel-by-pixel addition result of the output feature map of the Transformer module of the first upsampling layer and the downsampling branch; the second upsampling layer upsamples the output feature map of the Transformer module of the upsampling branch to H / 2×W / 2×2C; the input feature map of the second convolutional layer is: the pixel-by-pixel addition result of the output feature map of the second upsampling layer and the second Transformer module of the downsampling branch; and the size of the output feature map of the gated feedforward network is H×W×C;

[0019] The reconstruction branch includes: two Transformer modules and a convolution layer with a convolution kernel of 3×3; the input feature map of the first Transformer module is: the pixel-by-pixel addition result of the output feature map of the gated feedforward network and the first Transformer module of the downsampling branch, and the output feature map size of the Transformer module is H×W×2C; the output feature map size of the second Transformer module of the reconstruction branch is H×W×2C; the output feature map of the same size as the input image of the image restoration module is obtained through the convolution layer with a convolution kernel of 3×3 in the reconstruction branch, and then the output feature map is added to the input image pixel by pixel to obtain the output image of the image restoration module, that is, the reconstructed drone image.

[0020] Furthermore, when training the target detection network, it also includes: obtaining manual annotation information of drones in white light images and infrared light images to construct drone category labels; and the drones in the white light images and infrared light images are drones of different sizes and shapes.

[0021] Furthermore, before the image restoration module sequentially reconstructs the white light image and the infrared light image in the multimodal UAV image sequence, the image preprocessing of the white light image and the infrared light image is also included, including grayscale conversion, binarization and filtering.

[0022] Furthermore, in the target detection network, the Transformer model is used to extract background features of white light images / infrared images.

[0023] Furthermore, in the target detection network, reversible neural networks are used to extract detailed features of white light images / infrared images.

[0024] Furthermore, in step 4, the multi-target tracking using the cascaded Hungarian algorithm based on motion prior specifically includes:

[0025] The detection frames of the drone detection results are divided into two categories: high confidence and low confidence according to the set confidence threshold;

[0026] A matching cost matrix is ​​constructed based on the high-confidence detection frame and the current target trajectory, and the first-stage matching (matching between the detection frame and each target trajectory) is performed using the Hungarian algorithm. A supplementary matching cost matrix is ​​constructed for the unmatched target trajectory and the low-confidence detection frame, and the second-stage matching is performed using the Hungarian algorithm. That is, the present invention adopts a two-stage Hungarian algorithm in the matching process. In the first stage, fast matching is performed for high-confidence targets, and in the second stage, supplementary matching is performed for the remaining trajectories and the low-confidence detection frame, effectively improving the matching coverage and overall tracking accuracy, and enhancing the stability and anti-interference ability of the tracking system in complex scenes.

[0027] Motion prior information is introduced into the matching process. The position of the target trajectory in the next frame is predicted through the Kalman filter. The predicted bounding box of the current frame (i.e., the prediction result) is obtained. The predicted bounding box is compared with the detection box to obtain the matching cost including the geometric cost and the motion prior cost, so as to construct the matching cost matrix and the supplementary matching cost matrix.

[0028] The state of successfully matched target trajectories is updated; the lost frame count of unmatched target trajectories is increased, and when the lost frame count of a target trajectory exceeds a set threshold, the target trajectory is removed; and the unmatched detection box is used as a new target and the target trajectory corresponding to the new target is initialized to obtain a new target trajectory. The target trajectory state update can update the state and matching confidence of successfully matched target trajectories, determine whether to terminate the target trajectory based on the activity of unmatched target trajectories, and initialize the trajectory of newly appeared high-confidence detection boxes, realizing continuous trajectory update and management.

[0029] Furthermore, the calculation formula of the geometric cost is:

[0030]

[0031] in, Indicates the current frame The detection box and The geometric cost between target trajectories, Indicates the current frame The detection box and The intersection-over-union ratio between the target trajectories, which represents the The intersection-union ratio between the jth detection box and the predicted bounding box of the jth target track in the current frame;

[0032] The motion prior cost is calculated based on the predicted state of the Kalman filter, and its calculation formula is:

[0033]

[0034] in, Indicates the current frame The detection box and The motion prior cost between target trajectories, Indicates the The predicted state corresponding to the detection box, Indicates the The detection measurement corresponding to the target trajectory , is the state covariance matrix, the superscript Represents a transpose operation.

[0035] In the present invention, the matching cost not only considers the intersection over union (IoU), but also integrates metrics such as Euclidean distance, thereby improving the matching accuracy under occlusion, track intersection and detection noise conditions.

[0036] Furthermore, during the two-stage matching process, target trajectories are sorted based on their survival time and historical matching confidence, and then the detection boxes are matched against the target trajectories in order. Detections with longer survival times and higher historical matching confidences are ranked higher, prioritizing the matching of long-term stable target trajectories with high-confidence detection boxes. Short-term target trajectories are then processed sequentially with new target trajectories. Furthermore, by setting a decreasing confidence threshold, the interference of low-quality detections on high-confidence trajectories can be effectively suppressed. Specifically, as the confidence threshold decreases during sorting, detection boxes with a confidence threshold greater than or equal to the threshold are matched with the target trajectories at the current sort position.

[0037] The technical solution provided by the present invention brings at least the following beneficial effects:

[0038] The present invention uses multiple modes to work together, taking advantage of their strengths and overcoming their weaknesses, and can efficiently complete drone target detection and tracking tasks in complex environments such as cloud cover and tree cover. The present invention uses millimeter wave radar, white light camera, long wave + short wave dual-band infrared camera to detect drones in the air;

[0039] Based on real-time image acquisition from a multimodal camera and combined with a deep learning algorithm, the point cloud image generated by the millimeter-wave radar is processed to infer the azimuth coordinates and speed of the moving object. Combined with an attention mechanism, the white-light image and infrared image focus on the object position displayed by the millimeter-wave radar, extracting detailed features from the white-light and infrared images and performing feature-layer fusion. This enables the network to complete robust detection tasks in complex environments such as cloud and fog interference, greatly improving the drone identification task. Furthermore, when a single modality is limited, the coordination of other modalities can also effectively complete the drone identification task, ensuring the reliability of drone detection in various environments, enabling drone identification and classification, and helping to accurately determine the type of drone.

[0040] During the tracking stage, the present invention enhances the state modeling capability of the target under nonlinear motion and occlusion by introducing a prediction structure that integrates LSTM and Kalman filter, thereby improving the stability and continuity of tracking. The cascaded Hungarian algorithm that integrates motion prior information is used for multi-stage matching, which effectively improves the matching accuracy and robustness in complex environments such as multiple targets, occlusion, and trajectory intersection. At the same time, the confidence diversion strategy and trajectory management mechanism are combined to improve the response speed to new targets and the fault tolerance for lost targets. The global motion estimation and altitude information judgment are further combined to achieve accurate correction of the UAV trajectory and target differentiation, avoiding aliasing and misjudgment, thereby significantly improving the UAV detection and tracking capability of the present invention in all-weather, multi-target airspace. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A flowchart of a multimodal image-based drone detection and tracking method provided by an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the architecture of a target detection network according to an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of the structure of an image restoration module according to an embodiment of the present invention;

[0045] Figure 4 Schematic diagram of a multi-target tracking process according to an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the Kalman filter structure integrated with the long short-term memory network LSTM according to an embodiment of the present invention;

[0047] Figure 6 Schematic diagram of a multi-stage matching process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0049] In one embodiment, the present invention provides a method for detecting and tracking drones based on multimodal images. Figure 1 As shown, the following steps are included:

[0050] Step 1: Input a multimodal UAV image sequence acquired by a millimeter-wave radar, a white-light camera, and an infrared camera. The infrared camera includes a long-wave infrared camera and a short-wave infrared camera. The modal images of the multimodal UAV image sequence include: radar point cloud images, white-light images (also known as visible light images), and two-band infrared images.

[0051] Step 2: The image restoration module reconstructs each frame of the multimodal drone image sequence in sequence to remove the degradation of each modality image and obtain a high-quality reconstructed multimodal drone image, namely, a reconstructed radar point cloud image, a white light image, and a two-band infrared image.

[0052] Degradation includes noise during image capture, image blur caused by drone motion, and interference from complex environments like raindrops.

[0053] Step 3: Input the reconstructed multimodal drone image sequence into the target detection network, extract image features from each input image frame in turn, and obtain multimodal features for each image frame. The multimodal features are then fused and processed for drone detection to obtain drone detection results for each image frame, including: detection box, drone category, and confidence score.

[0054] The image feature information of the radar point cloud image includes the spatial state information of the drone (such as position, distance, direction, speed, angle, etc.); the point cloud data of the millimeter radar wave is converted into effective information, so that the attention mechanism can be combined with the attention mechanism to enhance the robustness of feature extraction when extracting detail features of white light images and infrared images; the image features of white light images and infrared images include background features and detail features (i.e., detail texture features);

[0055] In the embodiment of the present invention, background features are extracted from images captured by different cameras at the same time and location. The extraction structure can adopt a Transformer architecture with a long-range attention mechanism to complete the alignment and feature extraction of different modal backgrounds by fusing background feature points. Detail feature extraction adopts a reversible neural network, which can effectively capture the detailed information of the high-frequency texture features of the image.

[0056] In multimodal fusion, the background features of the white light image and the infrared image are fused to obtain the first fusion feature. The image feature information of the radar point cloud image is obtained through the attention mechanism and then fused with the detail features of the white light image and the infrared image to obtain the second fusion feature.

[0057] Obtaining a multimodal fusion feature of each image frame based on the first fusion feature and the second fusion feature;

[0058] Step 4: Based on the drone detection results of each image frame, multi-target tracking is performed using the cascaded Hungarian algorithm based on motion priors, and the trajectory tracking results of multiple drone targets are output to complete robust multi-drone tracking;

[0059] When tracking multiple targets, a long short-term memory (LSTM) network is combined with a Kalman filter to predict nonlinear motion trends. The LSTM output serves as the state input to the Kalman filter, which then predicts the target trajectory for each image frame. This improves prediction accuracy and state estimation robustness, enhancing stable tracking capabilities.

[0060] Because multimodal drone images are captured from the same scene at the same moment, the low-frequency information across the modalities contains commonalities, such as background and large-scale environmental features. However, the high-frequency information across the modalities is independent, such as texture and detail in visible light images and thermal radiation in infrared images. Therefore, the present invention facilitates the extraction of both modality-specific and modality-shared features by increasing and decreasing the correlation between low-frequency and high-frequency features, respectively.

[0061] In an embodiment of the present invention, to obtain a training dataset for training neural networks (such as image restoration modules and object detection networks), the system also includes: obtaining manually annotated drone information for white-light and infrared images to obtain drone category labels. Specifically, the system performs data preprocessing on the manually annotated drone information to construct drone category labels. The drones in the white-light and infrared images are of varying sizes and shapes. Furthermore, before the image restoration module sequentially reconstructs the white-light and infrared images in a multimodal drone image sequence, the system also performs image preprocessing on the white-light and infrared images. This includes grayscaling, binarization, and filtering, and may also include image enhancement to enrich the data sample. Image preprocessing is intended to improve image quality and feature extraction capabilities. For point cloud data collected by millimeter-wave radar, preprocessing (including filtering, noise reduction, alignment, etc.) is performed, and then the point cloud data is visualized to obtain the corresponding radar point cloud image.

[0062] In the embodiment of the present invention, the architecture of the target detection network is as follows: Figure 2As shown, images from different modalities are first processed using an image restoration module. This module restores blur caused by drone flight or camera motion, or image degradation caused by fog and fog, making the images easier to process. The processed image data is then further processed. A feature encoder (i.e., a feature extractor, such as one based on a deep convolutional network) is used to extract feature information from the point cloud image generated by the millimeter-wave radar. The point cloud image contains valuable information such as the position, distance, and orientation of the target (i.e., the drone) in the image. After obtaining this extracted feature information, an attention mechanism is used to extract features from key areas of the other modal images (visible light and infrared). The low-frequency background information is similar for both white light and infrared images, so a Transformer model can be used to process these low-frequency features. Considering real-time performance, a lightweight Transformer architecture is used to extract background features shared by multiple modalities. The detailed high-frequency features contained in infrared and visible light images are very different, but complementary. The detailed features of the two images are extracted through a detail feature encoder (which can use a reversible neural network). At the same time, the feature information of the millimeter-wave radar is received. An attention mechanism is used to focus on the millimeter-wave radar feedback for feature extraction. This can efficiently extract effective information, and then perform feature fusion to complete the drone recognition task. Figure 2 In

[15] , F represents the feature-level fusion operation, such as feature-level fusion based on attention mechanism, feature-level fusion based on feature pyramid network, etc.

[0063] The image restoration module of the present invention facilitates the task of reconstructing high-quality images by removing degradation from degraded input. This degradation includes noise from image capture, image blur caused by drone motion, and interference from complex environments, such as raindrops. This image restoration module can effectively assist in subsequent drone identification. Furthermore, the features of the three modalities complement and compensate for each other. The feature-level fusion of the feature layers enables the network to achieve superior detection performance compared to simple stacking. Even if one or two modules are disrupted by the complex environment, resulting in modal failure, the entire network can still effectively complete drone identification, ensuring effective drone recognition in complex environments.

[0064] In one embodiment, the network structure of the image restoration module of the present invention is as follows: Figure 3As shown, it includes a downsampling branch, an upsampling branch and a reconstruction branch; wherein the downsampling branch includes: a convolution layer with a convolution kernel of 3×3, a first Transformer module, a first downsampling layer, a second Transformer module, a second downsampling layer, a third Transformer module, a third downsampling layer and a fourth Transformer module; wherein the output feature map size of the convolution layer of the downsampling branch is recorded as H×W×C, where H and W represent the height and width of the output feature map, and C represents the number of channels; the first, second and third Transformer modules have The output feature map sizes are: H / 2×W / 2×2C, H / 4×W / 4×4C, H / 8×W / 8×8C; the upsampling branch includes: the first upsampling layer, the first convolutional layer, the Transformer module, the second upsampling layer, the second convolutional layer, the multi-head conversion attention layer and the gated feedforward network; the convolution kernels of the two convolutional layers are both 1×1; the input feature map of the first upsampling layer is the output feature map of the fourth Transformer module of the downsampling branch, which is upsampled to H / 4×W / 4×4C; the input feature map of the first convolutional layer is: the first upsampling layer and the downsampling branch The second upsampling layer upsamples the output feature map of the Transformer module of the upsampling branch to H / 2×W / 2×2C; the input feature map of the second convolutional layer is: the pixel-by-pixel addition result of the output feature map of the second upsampling layer and the second Transformer module of the downsampling branch; and the size of the output feature map of the gated feedforward network is H×W×C; the reconstruction branch includes: two Transformer modules and a convolution layer with a convolution kernel of 3×3; among them, the first Transfor The input feature map of the mer module is: the pixel-by-pixel addition result of the output feature map of the gated feedforward network and the first Transformer module of the downsampling branch. The output feature map size of the Transformer module is H×W×2C; the output feature map size of the second Transformer module of the reconstruction branch is H×W×2C; the output feature map of the same size as the input image of the image restoration module is obtained through the convolution layer with a convolution kernel of 3×3 in the reconstruction branch, and then the output feature map of the same size as the input image of the image restoration module is obtained by adding the output feature map to the input image pixel by pixel. That is, the output image of the image restoration module, that is, the reconstructed drone image.

[0065] Multiple Object Tracking (MOT), commonly referred to as Multiple Object Tracking, detects and assigns IDs to multiple targets (in this case, multiple drones) in a video, even without knowing the number of targets in advance. Different targets have unique IDs, enabling subsequent trajectory prediction and accurate search.

[0066] In the embodiment of the present invention, multi-target tracking mainly includes target detection, trajectory prediction, first matching, second association and trajectory management. Figure 4 When processing multiple targets, this implementation first receives the detection boxes and their confidence scores output by the target detection network in each frame. Based on a preset confidence threshold (track_thresh), the detection results are divided into high-confidence detection boxes (referred to as high-scoring detection boxes) and low-confidence detection boxes (referred to as low-scoring detection boxes). Detection boxes with confidence scores above track_thresh are considered high-confidence detection boxes, while the remaining ones are low-confidence detection boxes. Target association is performed on the drone objects corresponding to the high-confidence detection boxes. Simultaneously, the target trajectory information obtained by fusing global motion estimation and Kalman filter prediction in the previous frame is used to form a predicted bounding box, which participates in the initial matching process along with the detection results. If the detection box matches the predicted target trajectory (referred to as the trajectory) in terms of intersection over union (IoU), an updated trajectory segment is generated. If the match fails, the secondary association phase begins.

[0067] In the secondary association process, low-confidence detection boxes are matched with unmatched tracks, and a cost matrix is ​​constructed using Kalman prediction and IoU metrics to further improve the integrity of track matching. After matching, all tracks are managed, and track retention, update, or termination are determined based on the target state (State), matching status, and maximum number of lost frames (max_time_lost). This prevents long periods of unmatched tracks from disrupting system stability. Figure 4 In the , Lost indicates that the target is lost, Tracked indicates that the tracking is successful, and Is_activated indicates the activation status.

[0068] In this embodiment, for high-confidence detection frames that are not matched initially, they are allowed to be initialized as new targets when their confidence exceeds the set threshold high_thresh. They are set to a temporarily inactive state (Is_activated=False) and activated after stable matching in subsequent frames. To further enhance the ability to distinguish in dense target scenes, a height determination mechanism based on the vertical position of the image can also be introduced to assist in determining the relative height of the target, effectively avoiding track aliasing and ID switching problems when multiple targets overlap, significantly improving the stability and robustness of the tracking system in complex scenes.

[0069] In the embodiment of the present invention, an infrared-visible light multi-target tracking mode that integrates LSTM and Kalman filter is adopted. It mainly addresses the nonlinear and time-varying problems of target motion under multimodal perception. By introducing LSTM to model the motion sequence and combining the state estimation and error correction mechanism of Kalman filter, the system's adaptability to complex dynamics is effectively improved. Figure 5 , perform Kalman filter prediction on the input data to obtain the state prediction result of the Kalman filter, and predict the historical trajectory of the target through the LSTM network to obtain the prior state estimation result, which serves as a reference for the subsequent filtering process; the LSTM-KF fusion is obtained by weighted fusion between the state prediction result of the Kalman filter and the prior state estimation result output by the LSTM network, and then the Kalman filter correction is performed on it to obtain the final target trajectory information of each image frame.

[0070] Specifically, in the offline stage, the LSTM network is trained based on the historical motion trajectory to learn the temporal characteristics and change patterns of the target; in the online stage, the historical trajectory of the target is predicted by LSTM to obtain the prior state estimation result. , as a reference for the subsequent filtering process.

[0071] The state prediction of the traditional Kalman filter is in the following form:

[0072]

[0073] Based on the prediction results, the state is updated according to the measurement residuals. The update formula is:

[0074]

[0075] To enhance the accuracy of prediction, the embodiment of the present invention introduces LSTM output in the state prediction stage and integrates it with the traditional prediction model through linear weighting to construct an improved prediction expression:

[0076]

[0077] Among them, k represents the current moment, Represents the prior estimate, which is derived based on the optimal estimate at the previous moment; Represents the state transfer matrix, which characterizes the impact of the state at the previous moment on the current state; represents the optimal estimate of k-1 at the previous moment; represents the control input value, that is, the external control signal received at the previous moment k-1; Represents the control input transfer matrix, which converts the control input Mapping the control to the state, Represents the optimal estimate at the current moment, which is a fusion of the prior estimate and the observed value; represents the Kalman gain; Represents the observation value at the current moment (i.e., detection measurement); Represents measurement system parameters; A preset fusion weight factor is used to adjust the balance between the LSTM prediction value and the traditional linear model. This mechanism not only fully leverages the advantages of deep learning in modeling nonlinear dynamics, but also combines the error minimization estimation strategy of the Kalman filter to achieve a more stable and accurate estimation of the target state.

[0078] This method demonstrates excellent versatility and robustness in infrared and visible light fusion scenarios, making it particularly suitable for complex tracking environments with frequent occlusions, small target differences, and irregular motion. By dynamically fusion-learning priors and filtering optimization, it significantly improves the accuracy of target state prediction, providing greater continuity and anti-interference capabilities for multi-target tracking systems.

[0079] In the embodiment of the present invention, the target association process mainly adopts a cascade matching strategy that integrates geometric information and motion priors, and uses the Hungarian algorithm to achieve global optimal matching. Figure 6 During the multi-stage matching process, a Kalman filter is first used to predict the state of all surviving trajectories in the previous frame. This generates a predicted bounding box (i.e., prediction result) and covariance matrix for the current frame, providing motion prior information for subsequent matching. Based on this, a matching cost matrix is ​​constructed between the detection box and the predicted trajectory, comprehensively considering spatial geometric overlap and dynamic motion information to quantify the associated cost between the two.

[0080] The matching process uses a cascaded strategy, prioritizing stable trajectories that have survived for more than a specified number of frames (e.g., 5). This phase relies primarily on the IoU between the detection box and the predicted bounding box, as well as the consistency of the motion prediction. For short-term trajectories that survive for less than 5 frames, motion information is incorporated for supplementary matching when the Intersection over Union (IoU) is low. This strategy ensures long-term stable tracking while improving responsiveness to new targets.

[0081] After constructing the cost matrix, the Hungarian algorithm is used to globally solve the overall matching relationship, obtaining the optimal one-to-one correspondence between the detection box and the trajectory. After the matching is completed, the status of the successfully matched trajectory is updated, including position, confidence, motion status, etc. The tracklet that failed to match has its lost frame count increased. When the lost frame count exceeds a set threshold, it is removed. The unmatched detection box is then used as a new target to initialize the corresponding new trajectory.

[0082] Among them, the matching cost matrix It is the core of the entire association process, and each element represents the The detection box and The cost value generated when the trajectories are associated. To more realistically reflect the spatiotemporal relationship between targets, the cost matrix consists of two parts: geometric cost and motion prior cost, and is constructed through weighted fusion. Specifically, it includes:

[0083] First, the geometric cost part mainly measures the degree of spatial overlap between the detection box and the predicted trajectory. The IoU indicator is usually used and is defined as follows:

[0084]

[0085] in, Indicates the The detection box and The geometric cost between trajectories, Indicates the The detection box and The intersection-and-union ratio between the trajectories, i.e. The detection box and the target trajectory information in the call The intersection-over-union ratio between the predicted bounding boxes. This definition ensures that the greater the overlap of the matching boxes, the lower the cost, thus giving priority to matching objects with closer spatial locations.

[0086] Secondly, the motion prior cost is calculated based on the predicted state of the Kalman filter and its uncertainty, and the Mahalanobis distance is used to measure the The predicted state corresponding to the detection box With the The detection measurement corresponding to each trajectory The difference is calculated as:

[0087]

[0088] in, is the state covariance matrix, which reflects the uncertainty of the predicted state. This indicator is still robust when the detection accuracy is unstable.

[0089] Finally, the two parts of the cost are linearly weighted to construct the total matching cost:

[0090]

[0091] Among them, the fusion weight It can be flexibly adjusted according to the actual scenario to adapt to the importance ratio of geometric and motion information in different modal fusion environments.

[0092] In summary, the embodiments of the present invention, through the deep integration of the cascade matching strategy and motion prior modeling, ensure the matching accuracy while taking into account the spatiotemporal continuity and real-time requirements among multiple targets.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0094] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A drone detection and tracking method based on multimodal images, characterized in that: The following steps are involved: Step 1: Input a multimodal UAV image sequence acquired by a millimeter-wave radar, a white-light camera, and an infrared camera; the infrared camera includes a long-wave infrared camera and a short-wave infrared camera; and the modal images of the multimodal UAV image sequence include: a radar point cloud image, a white-light image, and two-band infrared images. Step 2: Reconstruct each frame of the multimodal drone image sequence in sequence through the image restoration module to remove the degradation of each modality image and obtain a reconstructed multimodal drone image; Step 3: Input the reconstructed multimodal drone image sequence into the target detection network, extract image features from each input image frame in turn, and obtain multimodal features for each image frame. The multimodal features are then fused and processed for drone detection to obtain drone detection results for each image frame, including: detection box, drone category, and confidence score. Among them, the image feature information of the radar point cloud image includes the spatial state information of the UAV; the image features of the white light image and the infrared light image include background features and detail features; During multimodal fusion, the target detection network fuses the background features of the white light image and the infrared image to obtain the first fused feature. The image feature information of the radar point cloud image is obtained through the attention mechanism and then fused with the detail features of the white light image and the infrared image to obtain the second fused feature. Obtaining a multimodal fusion feature of each image frame based on the first fusion feature and the second fusion feature; Step 4: Based on the drone detection results of each image frame, multi-target tracking is performed using the cascaded Hungarian algorithm based on motion priors, and the trajectory tracking results of multiple drone targets are output; When tracking multiple targets, a long short-term memory network (LSTM) is combined with a Kalman filter to predict nonlinear motion trends. The output of LSTM is used as the state input of the Kalman filter, and the target trajectory information of each image frame is predicted through the Kalman filter.

2. The method for detecting and tracking drones based on multimodal images according to claim 1, wherein: The image restoration module is a deep neural network based on the Transformer module.

3. The method for detecting and tracking drones based on multimodal images according to claim 2, wherein: The network structure of the image restoration module includes a downsampling branch, an upsampling branch, and a reconstruction branch; The downsampling branch includes, in sequence: a convolutional layer with a 3×3 convolution kernel, a first Transformer module, a first downsampling layer, a second Transformer module, a second downsampling layer, a third Transformer module, a third downsampling layer, and a fourth Transformer module. The output feature map size of the convolutional layer of the downsampling branch is denoted as H×W×C, where H and W represent the height and width of the output feature map, and C represents the number of channels. The output feature map sizes of the first, second, and third Transformer modules are H / 2×W / 2×2C, H / 4×W / 4×4C, and H / 8×W / 8×8C, respectively. The upsampling branch includes, in sequence: the first upsampling layer, the first convolutional layer, the Transformer module, the second upsampling layer, the second convolutional layer, the multi-head conversion attention layer and the gated feedforward network; the convolution kernels of the two convolutional layers are both 1×1; the input feature map of the first upsampling layer is the output feature map of the fourth Transformer module of the downsampling branch, which is upsampled to H / 4×W / 4×4C; the input feature map of the first convolutional layer is the pixel-by-pixel addition result of the output feature map of the Transformer module of the first upsampling layer and the downsampling branch; the second upsampling layer upsamples the output feature map of the Transformer module of the upsampling branch to H / 2×W / 2×2C; the input feature map of the second convolutional layer is the pixel-by-pixel addition result of the output feature map of the second upsampling layer and the second Transformer module of the downsampling branch; and the size of the output feature map of the gated feedforward network is H×W×C; The reconstruction branch includes: two Transformer modules and a convolution layer with a convolution kernel of 3×3; the input feature map of the first Transformer module is: the pixel-by-pixel addition result of the output feature map of the gated feedforward network and the first Transformer module of the downsampling branch, and the output feature map size of the Transformer module is H×W×2C; the output feature map size of the second Transformer module of the reconstruction branch is H×W×2C; the output feature map of the same size as the input image of the image restoration module is obtained through the convolution layer with a convolution kernel of 3×3 of the reconstruction branch, and then the output feature map is added to the input image pixel by pixel to obtain the output image of the image restoration module.

4. The method for detecting and tracking drones based on multimodal images according to claim 1, wherein: When training the target detection network, it also includes: obtaining manual annotation information of drones in white light images and infrared light images to construct drone category labels; and the drones in the white light images and infrared light images are drones of different sizes and shapes.

5. The method for detecting and tracking drones based on multimodal images according to claim 1, wherein: Before the white light images and infrared images in the multimodal UAV image sequence are reconstructed in sequence through the image restoration module, the white light images and infrared images are also preprocessed, including grayscale, binarization and filtering.

6. The method for detecting and tracking drones based on multimodal images according to claim 1, wherein: In the target detection network, the Transformer model is used to extract background features of white light images / infrared images.

7. The method for detecting and tracking drones based on multimodal images according to claim 1, wherein: In the target detection network, a reversible neural network is used to extract the detailed features of white light images / infrared images.

8. The method for detecting and tracking drones based on multimodal images according to claim 1, wherein: The cascaded Hungarian algorithm based on motion prior for multi-target tracking specifically includes: The detection frames of the drone detection results are divided into two categories: high confidence and low confidence according to the set confidence threshold; A matching cost matrix is ​​constructed based on the high-confidence detection box and the current target trajectory, and the first-stage matching is performed using the Hungarian algorithm. A supplementary matching cost matrix is ​​constructed for the unmatched target trajectory and the low-confidence detection box, and the second-stage matching is performed using the Hungarian algorithm. Motion prior information is introduced into the matching process. The position of the target trajectory in the next frame is predicted through the Kalman filter to obtain the predicted bounding box of the current frame. The predicted bounding box is compared with the detection box to obtain the matching cost including geometric cost and motion prior cost, so as to construct the matching cost matrix and the supplementary matching cost matrix. The state of the successfully matched target track is updated; the lost frame count of the unmatched target track is increased, and the track is removed when the lost frame count of the target track exceeds the set threshold; and the unmatched detection box is used as the new target and the target track corresponding to the new target is initialized.

9. The method for detecting and tracking drones based on multimodal images according to claim 8, wherein: The calculation formula of geometric cost is: ； in, Indicates the current frame The detection box and The geometric cost between target trajectories, Indicates the current frame The detection box and The intersection-over-union ratio between target trajectories; The motion prior cost is calculated based on the predicted state of the Kalman filter, and its calculation formula is: ； in, Indicates the current frame The detection box and The motion prior cost between target trajectories, Indicates the The predicted state corresponding to the detection box, Indicates the The detection measurement corresponding to the target trajectory , is the state covariance matrix, the superscript Represents a transpose operation.

10. The method for detecting and tracking drones based on multimodal images according to claim 8, wherein: In the two-stage matching process, the target trajectories are sorted according to their survival time and historical matching confidence, and then the detection boxes are matched with the target trajectories in order; among them, the longer the survival time and the higher the historical matching confidence, the higher the ranking.

Images