A multi-modal image-based unmanned aerial vehicle detection and tracking method

Abstract

The application discloses a kind of unmanned aerial vehicle detection tracking methods based on multi-modal image, belong to target detection and tracking field.The present application detects aerial unmanned aerial vehicle by the joint action of multiple modes to realize the reliability of unmanned aerial vehicle detection under various environments, realize the identification and classification of unmanned aerial vehicle;In the tracking stage, the prediction structure of the fusion of LSTM and Kalman filter is introduced, the state modeling ability under the nonlinear motion of target and the shielding condition is enhanced, and the stability and continuity of tracking are improved;Adopting the multi-stage matching strategy effectively improves the matching accuracy and robustness in complex environment;At the same time, the confidence shunt strategy and trajectory management mechanism are combined, the response speed to new target and the fault tolerance to lost target are improved;Combined with global motion estimation and height information determination, the precise correction of unmanned aerial vehicle trajectory and target differentiation are realized, aliasing and misjudgment are avoided, and the detection and tracking ability of unmanned aerial vehicle is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection and tracking, and particularly relates to a UAV detection and tracking method based on multi-modal images. BACKGROUND

[0002] Traditional UAV identification is detected by radar monitoring, optical monitoring, sound monitoring, electromagnetic monitoring and the like, but these methods have more or less certain defects. Radar monitoring cannot effectively distinguish UAVs from other flying objects, optical monitoring is limited by weather and light conditions, sound monitoring is easily disturbed by environmental noise, and electromagnetic monitoring is affected by environmental electromagnetic interference.

[0003] In order to solve the above problems, some researches realize UAV identification and positioning based on sound signal and image fusion, but this method is easily disturbed by noise, resulting in problems in preliminary identification, so that the identification and positioning accuracy cannot meet the demand. Some researches realize the identification of UAV targets on both sides through the mutual coordination of radar identification platforms and image identification platforms, complete the identification synchronization of UAV targets on both sides through target matching handshake; when the single-path is missing, the missing path is supplemented, so as to realize the identification complement; when the double-path is missing, the missing path is supplemented, so as to realize the complete tracking of the target path; the flight state of the UAV target is judged through the infrared image data before and after the UAV target is lost. Although this method realizes the complete tracking of the target path, fuses radar and image two modalities, and improves the identification effect, the feature difference of the two modalities is large, which cannot guarantee that both paths can work effectively in complex environments, and the single-path working will greatly affect the detection effect.

[0004] Therefore, in view of the UAV detection demand in complex environments, it is necessary to propose a UAV detection and tracking technology based on multi-modal images to realize real-time target detection in various complex environments. SUMMARY

[0005] The present application proposes a UAV detection and tracking method based on multi-modal images to solve the problems of poor identification and detection effect of "low, small and slow" UAVs, poor multi-modal feature fusion and inaccurate target tracking in complex environments, realize real-time discovery and positioning tracking of "low, small and slow" UAVs in complex environments, provide a detection and tracking method for UAVs in near-ground complex backgrounds, and improve the robustness of tracking UAVs.

[0006] The technical scheme adopted by the present application is: a UAV detection and tracking method based on multi-modal images, which comprises the following steps:

[0007] Step 1, input a multi-modal UAV image sequence acquired by a millimeter wave radar, a white light camera and an infrared camera; wherein the infrared camera comprises a long-wave infrared camera and a short-wave infrared camera; the multi-modal UAV image sequence comprises a radar point cloud image, a white light image and two-band infrared light images;

[0008] Step 2, sequentially reconstruct each frame of the multi-modal UAV image sequence by an image restoration module to remove the degradation of each modal image, and obtain a reconstructed multi-modal UAV image sequence;

[0009] Step 3, input the reconstructed multi-modal UAV image sequence into a target detection network, sequentially extract image features of each image frame, obtain multi-modal features of each image frame, and perform multi-modal fusion on the multi-modal features for UAV detection processing to obtain a UAV detection result of each image frame, including a detection box, a UAV category and a confidence score;

[0010] Wherein, the image feature information of the radar point cloud image comprises spatial state information of the UAV; the image features of the white light image and the infrared light image comprise background features and detail features;

[0011] When the target detection network performs multi-modal fusion, the background feature points of the white light image and the infrared light image are fused to obtain first fusion features; the image feature information of the radar point cloud image is obtained after attention mechanism, and then the detail features of the white light image and the infrared light image are fused to obtain second fusion features;

[0012] Based on the first fusion features and the second fusion features, multi-modal fusion features of each image frame are obtained;

[0013] Step 4, based on the UAV detection result of each image frame, a multi-target tracking is performed by a motion prior-based cascaded Hungarian algorithm, and a trajectory tracking result of multi-UAV targets is output;

[0014] In the multi-target tracking, a long short-term memory network (LSTM) combined with a Kalman filter is used for nonlinear motion trend prediction, wherein the output of the LSTM is used as the state input of the Kalman filter, and the Kalman filter is used to predict the target trajectory information of each image frame.

[0015] Further, the image restoration module is a deep neural network based on a Transformer module.

[0016] Further, the network structure of the image restoration module comprises a down-sampling branch, an up-sampling branch and a reconstruction branch;

[0017] The down-sampling branch sequentially comprises a convolution layer with a convolution kernel of 3*3, a first Transformer module, a first down-sampling layer, a second Transformer module, a second down-sampling layer, a third Transformer module, a third down-sampling layer and a fourth Transformer module; wherein the output feature map size of the convolution layer of the down-sampling branch is denoted as H*W*C, wherein H and W represent the height and width of the output feature map, and C represents the number of channels; the output feature map sizes of the first, second and third Transformer modules are H / 2*W / 2*2C, H / 4*W / 4*4C and H / 8*W / 8*8C, respectively;

[0018] The up-sampling branch sequentially comprises a first up-sampling layer, a first convolution layer, a Transformer module, a second up-sampling layer, a second convolution layer, a multi-head conversion attention layer and a Gated Feedforward Network (GFN); the convolution kernels of the two convolution layers are both 1*1; the input feature map of the first up-sampling layer is the output feature map of the fourth Transformer module of the down-sampling branch, which is up-sampled to H / 4*W / 4*4C; the input feature map of the first convolution layer is the pixel-by-pixel addition result of the output feature maps of the first up-sampling layer and the Transformer module of the down-sampling branch; the second up-sampling layer up-samples the output feature map of the Transformer module of the up-sampling branch to H / 2*W / 2*2C; the input feature map of the second convolution layer is the pixel-by-pixel addition result of the output feature maps of the second up-sampling layer and the second Transformer module of the down-sampling branch; and the size of the output feature map of the Gated Feedforward Network is H*W*C;

[0019] The reconstruction branch sequentially comprises two Transformer modules and a convolution layer with a convolution kernel of 3*3; wherein the input feature map of the first Transformer module is the pixel-by-pixel addition result of the output feature map of the first Transformer module of the down-sampling branch and the output feature map of the Gated Feedforward Network, and the size of the output feature map of this Transformer module is H*W*2C; the size of the output feature map of the second Transformer module of the reconstruction branch is H*W*2C; the output feature map with the same size as the input image of the image restoration module is obtained through the convolution layer with a convolution kernel of 3*3 of the reconstruction branch, and then the output feature map is added pixel by pixel with the input image to obtain the output image of the image restoration module, i.e., the reconstructed unmanned aerial vehicle image.

[0020] Further, when training the target detection network, the unmanned aerial vehicle artificial labeling information of the white light image and the infrared light image is obtained to construct the unmanned aerial vehicle category label; and the unmanned aerial vehicles in the white light image and the infrared light image are unmanned aerial vehicles with different sizes and shapes.

[0021] Further, before the white light image and the infrared light image in the multi-modal unmanned aerial vehicle image sequence are sequentially reconstructed by the image recovery module, the white light image and the infrared light image are further pre-processed, including grayscale, binarization and filtering.

[0022] Further, in the target detection network, a Transformer model is used to extract the background features of the white light image / infrared light image.

[0023] Further, in the target detection network, a reversible neural network (Reversible Neural Networks) is used to extract the detailed features of the white light image / infrared light image.

[0024] Further, in step 4, the multi-target tracking based on the cascaded Hungarian algorithm with motion prior includes:

[0025] According to the set confidence threshold, the detection boxes of the unmanned aerial vehicle detection results are divided into high confidence and low confidence;

[0026] Based on the high confidence detection box and the current target track, a matching cost matrix is constructed, and the first stage matching (matching between the detection box and each target track) is performed through the Hungarian algorithm. A supplementary matching cost matrix is constructed for the unmatched target track and the low confidence detection box, and the second stage matching is performed through the Hungarian algorithm. That is, the two-stage Hungarian algorithm is used in the matching process, the first stage is to perform fast matching for high confidence targets, and the second stage is to perform supplementary matching for the remaining tracks and low confidence detection boxes, which effectively improves the matching coverage and overall tracking accuracy, and enhances the stability and anti-interference ability of the tracking system in complex scenes.

[0027] In the matching process, motion prior information is introduced, the position of the target track in the next frame is predicted through a Kalman filter to obtain a predicted bounding box (i.e. a prediction result) of the current frame, and the predicted bounding box is compared with the detection box to obtain a matching cost including a geometric cost and a motion prior cost, so as to construct a matching cost matrix and a supplementary matching cost matrix.

[0028] The state of the matched target track is updated; the unmatched target track increases its lost frame count, and the track is removed when the lost frame count exceeds a set threshold; and the unmatched detection box is initialized as a new target and a target track corresponding to the new target is initialized to obtain a new target track. The state update of the matched target track can update the state and confidence of the matched target track, and can also determine whether the target track is terminated based on the activity of the unmatched target track. Meanwhile, the high confidence detection box is initialized as a new target track, which realizes the continuous update and management of the target track.

[0029] Furthermore, the formula for calculating the geometric cost is as follows:

[0030]

[0031] in, Indicates the current frame's... The detection box and the first The geometric cost between target trajectories Indicates the current frame's... The detection box and the first The intersection-union ratio (IUU) between the target trajectories, i.e., the ratio that represents the intersection The intersection-union (IoU) ratio of the detection box and the predicted bounding box of the j-th target trajectory in the current frame;

[0032] The motion prior cost is calculated based on the predicted state of the Kalman filter, and its calculation formula is as follows:

[0033]

[0034] in, Indicates the current frame's... The detection box and the first The prior cost of motion between target trajectories Indicates the first The predicted state corresponding to each detection box Indicates the first Detection and measurement corresponding to each target trajectory , The state covariance matrix is ​​represented by the superscript. This indicates the transpose operation.

[0035] In this invention, the matching cost not only considers the intersection-over-union ratio (IoU), but also incorporates metrics such as Euclidean distance, thereby improving the matching accuracy under conditions of occlusion, trajectory intersection, and detection noise.

[0036] Furthermore, in the two-stage matching process, target trajectories are sorted based on their survival time and historical matching confidence, and then the detection boxes are matched with the target trajectories sequentially. Longer survival times and higher historical matching confidence result in higher rankings, thus prioritizing the matching of long-term stable target trajectories and high-confidence detection boxes, followed by processing short-term and newly emerging target trajectories. In addition, by setting a decreasing confidence threshold, interference from low-quality detections on high-confidence trajectories can be effectively suppressed. That is, as the ranking decreases, detection boxes with a confidence threshold greater than or equal to that threshold are matched with the target trajectory at the current ranking position.

[0037] The technical solution provided by this invention brings at least the following beneficial effects:

[0038] The present application can efficiently complete the unmanned aerial vehicle target detection and tracking task in complex environments such as cloud and fog obstruction and tree obstruction through the joint action of multiple modalities. The present application detects the unmanned aerial vehicle in the air through millimeter wave radar, white light camera, long wave + short wave two-band infrared camera and other means;

[0039] Based on the real-time image acquisition of the multi-modal camera, the depth learning algorithm is combined to process the point cloud image generated by the millimeter wave radar, and the position coordinates and speed of the moving object are inferred. In combination with the attention mechanism, the white light image and the infrared image focus on the object position displayed by the millimeter wave radar, and the detailed features in the white light image and the infrared image are extracted. The feature layer fusion is performed, so that the network can complete the robust detection task in the complex environment such as cloud and fog interference, and greatly improve the unmanned aerial vehicle recognition task. When a single mode is limited, the mutual coordination of other modes can also complete the unmanned aerial vehicle recognition task well, ensuring the reliability of unmanned aerial vehicle detection in various environments, and realizing the recognition and classification of unmanned aerial vehicles, which is helpful for accurately judging the type of unmanned aerial vehicles.

[0040] In the tracking stage, the prediction structure of the fusion of LSTM and Kalman filter is introduced, the state modeling ability under the condition of nonlinear motion and occlusion of the target is enhanced, and the stability and continuity of tracking are improved. The cascaded Hungarian algorithm combining motion prior information is used for multi-stage matching, which effectively improves the matching accuracy and robustness in complex environments such as multiple targets, occlusion and trajectory intersection. At the same time, combined with the confidence shunt strategy and the trajectory management mechanism, the response speed to new targets and the fault tolerance to lost targets are improved. Further combined with global motion estimation and height information determination, the precision correction and target differentiation of the unmanned aerial vehicle trajectory are realized, the aliasing and misjudgment are avoided, and then the unmanned aerial vehicle detection and tracking ability of the present application in all-weather and multi-target airspace is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 A flow chart of an unmanned aerial vehicle detection and tracking method based on multi-modal images provided by the embodiment of the present application;

[0043] Figure 2 An architecture diagram of the target detection network of the embodiment of the present application;

[0044] Figure 3 A structural schematic diagram of an image restoration module of an embodiment of the present application;

[0045] Figure 4 A schematic diagram of a multi-target tracking processing procedure of an embodiment of the present application;

[0046] Figure 5 A structural schematic diagram of a Kalman filter of a fusion long short-term memory network (LSTM) of an embodiment of the present application;

[0047] Figure 6 A schematic diagram of a multi-stage matching processing procedure of an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will describe the technical solutions in the embodiments of the present application in detail and completely with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Generally, the components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application.

[0049] In one embodiment, the method for detecting and tracking a UAV based on multi-modal images provided by the embodiments of the present application includes the following steps as shown in Figure 1

[0050] Step 1: input multi-modal UAV image sequences acquired by a millimeter wave radar, a white light camera and an infrared camera; wherein the infrared camera includes a long-wave infrared camera and a short-wave infrared camera; the modal images of the multi-modal UAV image sequences include a radar point cloud image, a white light image (also referred to as a visible light image) and two-band infrared light images;

[0051] Step 2: sequentially perform image reconstruction on each frame of the multi-modal UAV image sequences by an image restoration module to remove the degradation of each modal image, to obtain high-quality multi-modal UAV images after reconstruction, i.e., to obtain the reconstructed radar point cloud image, the white light image and the two-band infrared light images;

[0052] Wherein, the degradation includes noise contained when the image is taken, image blur caused by UAV movement, and interference similar to raindrops in a complex environment, etc.

[0053] ​Step 3, input the reconstructed multi-modal unmanned aerial vehicle image sequence into a target detection network, and sequentially perform image feature extraction on each input image frame to obtain multi-modal features of each image frame; and after multi-modal fusion, unmanned aerial vehicle detection processing is performed to obtain the unmanned aerial vehicle detection result of each image frame, including: detection frame, unmanned aerial vehicle category and confidence score;

[0054] Wherein, the image feature information of the radar point cloud image includes the spatial state information of the unmanned aerial vehicle (such as position, distance, direction speed, angle, etc.); The point cloud data of the millimeter radar wave is converted into effective information, so that the robustness of feature extraction can be enhanced when extracting the detailed features of the white light image and the infrared light image by combining the attention mechanism; The image features of the white light image and the infrared light image include background features and detailed features (i.e. detailed texture features);

[0055] In the embodiment of the application, the background features are extracted from images taken by different cameras at the same time and at the same place, and the extraction structure can adopt a Transformer architecture with a long-range attention mechanism to complete the alignment and feature extraction of different modal backgrounds through background feature point fusion; The detailed feature extraction adopts a reversible neural network, which can effectively capture the detailed information of high-frequency texture features of the image;

[0056] In the multi-modal fusion, the background feature points of the white light image and the infrared light image are fused to obtain the first fusion feature; The image feature information of the radar point cloud image is obtained after the attention mechanism is used to obtain the attention information, and then the detailed features of the white light image and the infrared light image are fused to obtain the second fusion feature;

[0057] Based on the first fusion feature and the second fusion feature, the multi-modal fusion features of each image frame are obtained;

[0058] Step 4, based on the unmanned aerial vehicle detection result of each image frame, the multi-target tracking is performed through the cascaded Hungarian algorithm based on motion prior, and the trajectory tracking result of the multi-unmanned aerial vehicle target is output to complete the robust multi-unmanned aerial vehicle tracking;

[0059] In the multi-target tracking, the long short-term memory network LSTM and the Kalman filter are combined to perform nonlinear motion trend prediction, the output of the LSTM is used as the state input of the Kalman filter, and the target trajectory information of each image frame is predicted through the Kalman filter. To improve the prediction accuracy and the robustness of state estimation, and enhance the stability of tracking ability.

[0060] Since the multi-modal UAV images come from the same scene at the same time, the low-frequency information of several modalities contains commonalities, such as the background and large-scale environmental features. While the high-frequency information of several modalities is independent, such as the texture and detail information in the visible light image and the thermal radiation information in the infrared image. Therefore, the present application promotes the extraction of modal-specific features and modal-shared features by respectively increasing and reducing the correlation between low-frequency and high-frequency features.

[0061] In the embodiment of the present application, in order to obtain the training data set for training the neural network (such as the image restoration module, the target detection network, etc.), the UAV artificial annotation information of the white light image and the infrared light image is also obtained to obtain the UAV category label, that is, the UAV artificial annotation information is preprocessed to construct the UAV category label, and the UAV in the white light image and the infrared light image is a UAV with different sizes and shapes. In addition, before the white light image and the infrared light image in the multi-modal UAV image sequence are sequentially restored by the image restoration module, the white light image and the infrared light image are also preprocessed, including grayscale, binarization and filtering, and can also include image enhancement to enrich the data samples. Image preprocessing is to improve the image quality and feature extraction capability. For the point cloud data collected by the millimeter wave radar, the point cloud data is preprocessed (including filtering, noise reduction, alignment, etc.), and then visualized to obtain the corresponding radar point cloud image.

[0062] In the embodiment of the present application, the architecture of the target detection network is as follows Figure 2As shown, first, the images of different modalities are processed by the image restoration module, that is, the image degradation problems caused by the unmanned aerial vehicle flight or camera motion or cloud and fog are restored, so that the image is more convenient for processing, and then the processed image data is further processed. The feature encoder (that is, a feature extractor such as a deep convolutional network-based feature extractor) is used to extract feature information from the point cloud image generated by the millimeter wave radar. The point cloud image contains effective information such as the position, distance, and direction of the target (that is, the unmanned aerial vehicle) in the image. After obtaining the extracted feature information, the key parts of the other several modal images (visible light image and infrared light image) are extracted in combination with the attention mechanism. For the white light image and the infrared image, the low-frequency information of the background is similar, and the Transformer model can be used to process the low-frequency features of the background. Considering the real-time performance, a lightweight Transformer architecture is selected, and the background feature information common to multiple modalities is extracted on this basis. The detailed high-frequency features contained in the infrared light image and the visible light image are very different but complementary. The detailed feature encoder (which can use a reversible neural network) is used to extract the detailed features of the two kinds of images, and the feature information of the millimeter wave radar is received. The attention mechanism is used to focus on the place where the millimeter wave radar feedbacks to extract features, so that effective information can be efficiently extracted, and then the feature fusion is completed to complete the unmanned aerial vehicle recognition task. Figure 2 In the formula, F represents a feature level fusion operation, such as an attention mechanism-based feature level fusion, a feature pyramid network-based feature level fusion, etc.

[0063] The image restoration module of the present application is helpful for the task of removing degradation from degraded input to reconstruct high-quality images, wherein the degradation includes noise contained when the image is taken, image blur caused by unmanned aerial vehicle motion, and interference similar to raindrops in a complex environment, etc. The image restoration module can well help to complete further unmanned aerial vehicle recognition work. At the same time, the features of the three modalities complement each other and compensate for each other. The feature level fusion of the feature layer makes the network better than the simple stacked detection effect. Even if one or two modules are disturbed by a complex environment and cause modal failure, the entire network can still effectively complete the unmanned aerial vehicle recognition work, and the recognition effect of the unmanned aerial vehicle in a complex environment is ensured.

[0064] In one embodiment, the network structure of the image restoration module of the present application is as shown in Figure 3As shown, it comprises a downsampling branch, an upsampling branch and a reconstruction branch; wherein the downsampling branch comprises, in sequence, a convolution layer with a convolution kernel of 3x3, a first Transformer module, a first downsampling layer, a second Transformer module, a second downsampling layer, a third Transformer module, a third downsampling layer and a fourth Transformer module; wherein the output feature map size of the convolution layer of the downsampling branch is denoted as HxWxC, wherein H and W represent the height and width of the output feature map, and C represents the number of channels; the output feature map sizes of the first, second and third Transformer modules are H / 2xW / 2x2C, H / 4xW / 4x4C, H / 8xW / 8x8C, respectively; the upsampling branch comprises, in sequence, a first upsampling layer, a first convolution layer, a Transformer module, a second upsampling layer, a second convolution layer, a multi-head conversion attention layer and a gated feedforward network; the convolution kernels of the two convolution layers are both 1x1; the input feature map of the first upsampling layer is the output feature map of the fourth Transformer module of the downsampling branch, which is upsampled to H / 4xW / 4x4C; the input feature map of the first convolution layer is the pixel-by-pixel addition result of the output feature maps of the first upsampling layer and the Transformer module of the downsampling branch; the output feature map of the Transformer module of the upsampling branch is upsampled to H / 2xW / 2x2C by the second upsampling layer; the input feature map of the second convolution layer is the pixel-by-pixel addition result of the output feature maps of the second upsampling layer and the second Transformer module of the downsampling branch; and the size of the output feature map of the gated feedforward network is HxWxC; the reconstruction branch comprises, in sequence, two Transformer modules and a convolution layer with a convolution kernel of 3x3; wherein the input feature map of the first Transformer module of the reconstruction branch is the pixel-by-pixel addition result of the output feature map of the first Transformer module of the downsampling branch and the gated feedforward network, and the size of the output feature map of the Transformer module is HxWx2C; the size of the output feature map of the second Transformer module of the reconstruction branch is HxWx2C; the output feature map with the same size as the input image of the image restoration module is obtained through the convolution layer with a convolution kernel of 3x3 of the reconstruction branch, and then the output feature map is added pixel by pixel with the input image to obtain the output image of the image restoration module, i.e., the reconstructed unmanned aerial vehicle image.

[0065] Multiple object tracking, generally referred to as MOT (Multiple Object Tracking), is to detect and assign IDs to track the trajectories of multiple targets (in this application, multiple unmanned aerial vehicles) in a video without knowing the number of targets in advance. Different targets have different IDs in order to realize subsequent trajectory prediction, accurate search and other work.

[0066] In the embodiment of the present application, multi-target tracking mainly includes target detection, trajectory prediction, first matching, secondary association and trajectory management processes. Referring to Figure 4 In the embodiment, when processing multi-target tracking, the first receives the detection frame and confidence score output by the target detection network in each frame of image, and divides the detection results into high-confidence detection frames (referred to as high-score detection frames) and low-confidence detection frames (referred to as low-score detection frames) according to the pre-set confidence threshold track_thresh, that is, the detection frame with a confidence score higher than track_thresh is a high-confidence detection frame, and the remaining is a low-confidence detection frame. The target association is performed on the unmanned aerial vehicle object corresponding to the high-confidence detection frame, and the target trajectory information predicted by fusing global motion estimation and Kalman filter in the previous frame is called to form a predicted bounding box, which participates in the first matching together with the detection result. If the detection frame and the predicted target trajectory (referred to as trajectory) match on the intersection over union IoU, an updated trajectory segment is generated; if the matching fails, the secondary association stage is entered.

[0067] In the secondary association processing, the low-confidence detection frame is matched with the unmatched trajectory, and Kalman prediction and IoU measurement are introduced to construct a cost matrix to further improve the integrity of trajectory matching. After the matching is completed, the trajectory management processing is performed on all trajectories, and whether the trajectory should be retained, updated or terminated is judged according to the target state (State), the matching condition and the maximum lost frame number (max_time_lost) to avoid the interference of long-time unmatched trajectory on the system stability. Figure 4 In the embodiment, Lost represents target loss, Tracked represents tracking success, and Is_activated represents the activated state.

[0068] In the embodiment, for the first unmatched high-confidence detection frame, it is allowed to be initialized as a new target when its confidence is higher than the set threshold high_thresh, and set to a temporarily inactive state (Is_activated=False), and activated after stable matching in the subsequent frame. To further enhance the discrimination ability in the dense target scene, a height determination mechanism based on the vertical position of the image can also be introduced to assist in judging the relative height of the target, thereby effectively avoiding the trajectory aliasing and ID switching problem when multiple targets overlap, and significantly improving the stability and robustness of the tracking system in complex scenes.

[0069] In the embodiment of the present application, an infrared-visible light multi-target tracking mode fusing LSTM and Kalman filter is adopted, which mainly aims at the nonlinearity and time-varying problem of target motion under multi-modal perception. By introducing LSTM to model the motion sequence and combining the state estimation and error correction mechanism of Kalman filter, the adaptability of the system to complex dynamics is effectively improved. Referring toFigure 5 The input data is subjected to Kalman filtering prediction to obtain a Kalman filtered state prediction result, a historical trajectory of the target is predicted by an LSTM network to obtain a prior state estimation result as a reference basis for a subsequent filtering process, and the LSTM-KF fusion is obtained based on a weighted fusion between the Kalman filtered state prediction result and the prior state estimation result output by the LSTM network, and the target trajectory information of each image frame is obtained after Kalman filtering correction.

[0070] Specifically, in the offline stage, the LSTM network is trained based on the historical motion trajectory to learn the time sequence characteristics and change mode of the target, and in the online stage, the historical trajectory of the target is predicted by the LSTM to obtain a prior state estimation result as a reference basis for a subsequent filtering process.

[0071] The state prediction of the traditional Kalman filter adopts the following form:

[0072]

[0073] On the basis of the prediction result, the state is updated according to the measurement residual, and the update formula is:

[0074]

[0075] To enhance the accuracy of prediction, the LSTM output is introduced in the state prediction stage in the embodiment of the application, the linear weighting is fused with the traditional prediction model, and an improved prediction expression is constructed:

[0076]

[0077] Wherein, k represents the current time, represents a prior estimation value, which is derived based on the optimal estimation value of the previous time; represents a state transition matrix, representing the influence of the state of the previous time on the current state; represents the optimal estimation value of the previous time k-1; represents a control input value, that is, an external control signal received at the previous time k-1; represents a control input transition matrix, which maps and converts the control input into the control of the state, represents the optimal estimation value of the current time, which is the fusion of the prior estimation value and the observation value; represents a Kalman gain; represents the observation value (i.e., the detection measurement) of the current time; represents a measurement system parameter; ​The preset fusion weight factor is used to adjust the balance between the LSTM prediction value and the traditional linear model. The mechanism not only can give full play to the advantage of deep learning in modeling nonlinear dynamics, but also can combine the error minimum estimation strategy of Kalman filter, so that more stable and accurate estimation of the target state is realized.

[0078] The application has good universality and robustness in the infrared and visible light fusion scene, and is especially suitable for complex tracking environments with frequent occlusion, small difference between targets, irregular motion, etc. Through dynamic fusion of prior learning and filtering optimization process, the accuracy of target state prediction is significantly improved, and the continuity and anti-interference ability of the multi-target tracking system are improved.

[0079] In the embodiment of the application, the target association processing process mainly adopts a cascade matching strategy that fuses geometric information and motion prior, and realizes global optimal matching with the help of the Hungarian algorithm. Referring to Figure 6 In the multi-stage matching process, first, all surviving trajectories in the last frame are subjected to state prediction using a Kalman filter to generate a predicted bounding box (i.e. prediction result) and a covariance matrix in the current frame, thereby providing motion prior information for subsequent matching. On this basis, a matching cost matrix between the detection box and the predicted trajectory is constructed, and the spatial geometric overlap and dynamic motion information are considered to quantify the association cost between them.

[0080] The matching process adopts a cascade strategy, first, stable trajectories with a survival time exceeding a specified number of frames (e.g. 5 frames) are preferentially matched, this stage mainly relies on the IoU between the detection box and the predicted bounding box and the consistency of motion prediction; for short-term trajectories with a survival time not exceeding 5 frames, motion information is introduced for supplementary matching under the condition of low intersection and union. Through this strategy, the response ability to new targets is improved while ensuring long-term stable tracking.

[0081] After the cost matrix is constructed, the Hungarian algorithm is used to globally solve the overall matching relationship to obtain the optimal one-to-one correspondence between the detection box and the trajectory. After matching is completed, the state of the successfully matched trajectory is updated, including position, confidence, motion state, etc.; the trajectory that is not successfully matched will increase its lost frame count, and will be removed when the lost frame count exceeds a specified threshold; and the unmatched detection box is initialized as a new target to initialize the corresponding new trajectory.

[0082] wherein the matching cost matrix is the core of the entire association process, each element of which represents the matching cost between the i-th detection box and the j-th trajectory. ​The cost generated when associating trajectories. To more realistically reflect the spatiotemporal relationship between targets, this cost matrix consists of two parts: geometric cost and motion prior cost, constructed through a weighted fusion method, specifically including:

[0083] First, the geometric cost component primarily measures the spatial overlap between the detection bounding box and the predicted trajectory. The Interchange of Units (IoU) metric is typically used, defined as follows:

[0084]

[0085] in, Indicates the first The detection box and the first The geometric cost between trajectories Indicates the first The detection box and the first The intersection-union ratio between the trajectories, i.e. the th trajectories The first detection box and the first in the retrieved target trajectory information The intersection-over-union ratio (IoU) between predicted bounding boxes. This definition ensures that the greater the overlap of the matching boxes, the lower the cost, thus prioritizing the matching of targets that are spatially closer.

[0086] Secondly, the motion prior cost is calculated based on the predicted state of the Kalman filter and its uncertainty, using Mahalanobis distance to measure the prior cost. The predicted state corresponding to each detection box With the Detection and measurement corresponding to each trajectory The difference is calculated using the following formula:

[0087]

[0088] in, This is the state covariance matrix, reflecting the uncertainty of the predicted state. This index still exhibits strong robustness even when the detection accuracy is unstable.

[0089] Finally, the two costs are linearly weighted to construct the total matching cost:

[0090]

[0091] Among them, the fusion weight It can be flexibly adjusted according to the actual scenario to adapt to the importance ratio of geometric and motion information in different modal fusion environments.

[0092] In summary, the embodiments of the present invention, through the deep integration of cascaded matching strategy and motion prior modeling, ensure matching accuracy while taking into account the spatiotemporal continuity and real-time requirements among multiple targets.

[0093] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0094] The above only describes some embodiments of the present application. For those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1.A method for detecting and tracking unmanned aerial vehicles (UAVs) based on multi-modal images, the method comprising: The method comprises the following steps: Step 1, inputting a multi-modal unmanned aerial vehicle image sequence acquired by a millimeter wave radar, a white light camera and an infrared camera; wherein the infrared camera comprises a long-wave infrared camera and a short-wave infrared camera; the multi-modal unmanned aerial vehicle image sequence comprises a radar point cloud image, a white light image and two-band infrared light images; Step 2, sequentially performing image reconstruction on each frame of the multi-modal unmanned aerial vehicle image sequence by an image restoration module to remove the degradation of each modal image, and obtaining a reconstructed multi-modal unmanned aerial vehicle image; Step 3, inputting the reconstructed multi-modal unmanned aerial vehicle image sequence into a target detection network, sequentially performing image feature extraction on each image frame inputted, obtaining multi-modal features of each image frame, and performing unmanned aerial vehicle detection processing on the multi-modal features after multi-modal fusion to obtain unmanned aerial vehicle detection results of each image frame, including a detection frame, a unmanned aerial vehicle category and a confidence score; wherein the image feature information of the radar point cloud image comprises spatial state information of the unmanned aerial vehicle; the image features of the white light image and the infrared light image comprise background features and detail features; when the target detection network performs multi-modal fusion, the background features of the white light image and the infrared light image are fused to obtain first fusion features; the image feature information of the radar point cloud image is obtained after attention mechanism to obtain attention information, and then the attention information is fused with the detail features of the white light image and the infrared light image to obtain second fusion features; the multi-modal fusion features of each image frame are obtained based on the first fusion features and the second fusion features; Step 4, performing multi-target tracking based on the unmanned aerial vehicle detection results of each image frame by using a cascaded Hungarian algorithm based on motion prior, and outputting trajectory tracking results of multi-unmanned aerial vehicle targets; when performing multi-target tracking, a long short-term memory network (LSTM) is combined with a Kalman filter to perform nonlinear motion trend prediction, wherein the output of the LSTM is used as the state input of the Kalman filter, and the target trajectory information of each image frame is predicted by the Kalman filter. 2.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 1, wherein, The image restoration module is a deep neural network based on a Transformer module. 3.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 2, wherein, The network structure of the image restoration module comprises a down-sampling branch, an up-sampling branch and a reconstruction branch; wherein the down-sampling branch comprises, in sequence, a convolution layer with a convolution kernel of 3x3, a first Transformer module, a first down-sampling layer, a second Transformer module, a second down-sampling layer, a third Transformer module, a third down-sampling layer and a fourth Transformer module; wherein the output feature map size of the convolution layer of the down-sampling branch is denoted as HxWxC, wherein H and W represent the height and width of the output feature map, and C represents the number of channels; the output feature map sizes of the first, second and third Transformer modules are H / 2xW / 2x2C, H / 4xW / 4x4C and H / 8xW / 8x8C, respectively; The upsampling branch sequentially comprises: a first upsampling layer, a first convolutional layer, a Transformer module, a second upsampling layer, a second convolutional layer, a multi-head conversion attention layer, and a gated feedforward network; the convolutional kernels of the two convolutional layers are both 1×1; the input feature map of the first upsampling layer is the output feature map of the fourth Transformer module of the downsampling branch, which is upsampled to H / 4×W / 4×4C; the input feature map of the first convolutional layer is the pixel-by-pixel addition result of the output feature maps of the first upsampling layer and the Transformer module of the downsampling branch; the second upsampling layer upsampled the output feature map of the Transformer module of the upsampling branch to H / 2×W / 2×2C; the input feature map of the second convolutional layer is the pixel-by-pixel addition result of the output feature maps of the second upsampling layer and the second Transformer module of the downsampling branch; and the size of the output feature map of the gated feedforward network is H×W×C; The reconstruction branch sequentially comprises: two Transformer modules and a convolutional layer with a 3×3 convolutional kernel; wherein the input feature map of the first Transformer module is the pixel-by-pixel addition result of the output feature map of the first Transformer module of the downsampling branch and the gated feedforward network, and the size of the output feature map of the Transformer module is H×W×2C; the size of the output feature map of the second Transformer module of the reconstruction branch is H×W×2C; the output feature map with the same size as the input image of the image restoration module is obtained through the convolutional layer with a 3×3 convolutional kernel of the reconstruction branch, and the output image of the image restoration module is obtained by pixel-by-pixel addition of the output feature map and the input image. 4.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 1, wherein, When training the target detection network, the UAV artificial labeling information of the white light image and the infrared light image is obtained to construct a UAV class label; and the UAVs in the white light image and the infrared light image are UAVs with different sizes and shapes. 5.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 1, wherein, Before the white light image and the infrared light image in the multi-modal UAV image sequence are sequentially reconstructed by the image restoration module, the white light image and the infrared light image are preprocessed, including grayscale, binarization, and filtering. 6.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 1, wherein, In the target detection network, a Transformer model is used to extract the background features of the white light image / infrared light image. 7.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 1, wherein, In the target detection network, a reversible neural network is used to extract the detailed features of the white light image / infrared light image. 8.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 1, wherein, The multi-target tracking based on the motion-prior cascaded Hungarian algorithm specifically includes: According to the set confidence threshold, the detection boxes of the UAV detection results are divided into high confidence and low confidence; Based on the high confidence detection boxes and the current target track, a matching cost matrix is constructed, and the first stage matching is performed through the Hungarian algorithm; a supplementary matching cost matrix is constructed for the unmatched target track and the low confidence detection box, and the second stage matching is performed through the Hungarian algorithm; In the matching process, motion prior information is introduced, the position of the target trajectory in the next frame is predicted by a Kalman filter, a predicted bounding box of the current frame is obtained, and the predicted bounding box is compared with the detection box to obtain a matching cost including a geometric cost and a motion prior cost, so as to construct a matching cost matrix and a supplementary matching cost matrix; The state of the successfully matched target trajectory is updated, the lost frame count of the unsuccessfully matched target trajectory is increased, the target trajectory is removed when the lost frame count exceeds a set threshold, and the unsuccessfully matched detection box is taken as a new target and the target trajectory corresponding to the new target is initialized. 9.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 8, wherein, The calculation formula of the geometric cost is: ； in, Indicates the current frame's... The detection box and the first The geometric cost between target trajectories Indicates the current frame's... The detection box and the first Intersection over union ratio among target trajectories; The motion prior cost is calculated based on the predicted state of the Kalman filter, and the calculation formula is: ； wherein, represents a motion prior cost between the i-th detection box of the current frame and the j-th target trajectory, represents a motion prior cost between the i-th detection box of the current frame and the j-th target trajectory, represents a motion prior cost between the i-th detection box of the current frame and the j-th target trajectory, represents a predicted state corresponding to the i-th detection box, represents a predicted state corresponding to the i-th detection box, represents a detection measurement corresponding to the i-th target trajectory, represents a detection measurement corresponding to the i-th target trajectory, , is a state covariance matrix, and the superscript represents a transpose operation. 10.The unmanned aerial vehicle detection and tracking method based on multi-modal images of claim 8, wherein, In the two-stage matching process, the target trajectories are sorted according to the survival time and the historical matching confidence of each target trajectory, and then the matching between the detection box and the target trajectory is performed in sequence; wherein the longer the survival time and the higher the historical matching confidence, the higher the sorting.

Images