Unmanned aerial vehicle detection method based on radar infrared visible light multi-mode fusion

Abstract

The invention relates to the technical field of low-altitude unmanned aerial vehicle detection, in particular to an unmanned aerial vehicle detection method based on radar infrared visible light multi-mode fusion, which comprises the following steps: S1, respectively inputting an obtained unmanned aerial vehicle infrared image and an obtained visible light image into two visual branches of a double-flow feature extraction network, and extracting scale features by adopting a lightweight convolutional backbone network; s2, acquiring three-dimensional coordinate data of the unmanned aerial vehicle after radar detection and clutter removal processing in real time, and projecting the three-dimensional coordinate data into two-dimensional image coordinates through a camera external parameter matrix; s3, generating a two-dimensional Gaussian distribution probability graph based on the real-time two-dimensional image coordinates, and inputting the two-dimensional Gaussian distribution probability graph as a mask into the radar feature extraction network to extract a radar feature graph; and S4, under the same scale, performing dynamic weighted fusion on the scale features of the infrared branches and the visible light branches to generate a visual fusion feature map. The method greatly improves the capability of detecting the low-altitude unmanned aerial vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of low-altitude unmanned aerial vehicle detection, and particularly relates to an unmanned aerial vehicle detection method based on radar, infrared and visible light multi-modal fusion. BACKGROUND

[0002] In recent years, unmanned aerial vehicle technology has developed rapidly and has been widely applied in fields such as city patrol, power inspection, agricultural monitoring and logistics transportation, providing significant convenience and improving efficiency for social development. However, the number of unmanned aerial vehicles in low-altitude airspace has increased rapidly, and problems such as illegal flight, hidden surveillance and malicious intrusion have become increasingly prominent. Therefore, reliable perception and accurate identification of unmanned aerial vehicles have become an important research direction in the field of airspace regulation.

[0003] Existing unmanned aerial vehicle detection technologies are mainly based on single-modal sensors, including visible light sensors, infrared sensors, radars and radio signal sensors. Among them, the detection method based on visible light has better performance in good lighting conditions, but it is easy to cause image overexposure and loss of texture information in strong light, glare or reflective environment, resulting in a significant decrease in recognition accuracy; the detection method based on infrared imaging has strong robustness to strong light interference, but lacks fine texture and edge information; the detection method based on radar has all-weather working ability, but its resolution is limited, making it difficult to achieve accurate detection at the image level.

[0004] To overcome the limitations of single-modal methods, multi-modal fusion detection methods have gradually developed in recent years, such as using the complementary characteristics of visible light and infrared, or combining radar data with visual information. However, multi-modal fusion still faces many challenges: on the one hand, radar detection has angle error and resolution limitations, which can easily cause target positioning deviation, affecting the image alignment accuracy and feature guidance effectiveness of multi-modal data; on the other hand, existing fusion strategies are mostly shallow feature splicing or simple weighted fusion, lacking deep modeling ability for cross-modal semantic information, and failing to cooperatively associate the spatial positioning stability of radar with the detailed recognition ability of visual modalities, resulting in coexistence of positioning deviation and feature information loss, and inability to effectively address the image modal degradation problem in strong light, making it difficult to adapt to strong light and other high-difficulty detection scenarios.

[0005] The image modal degradation problem in strong light environment is particularly prominent. In high-reflective backgrounds such as glass curtain walls, water reflection and metal structures, the unmanned aerial vehicle target often exhibits characteristics such as local overexposure, low contrast and blurred edges, making it easy to miss detection or misjudge. The existing dual-modal fusion method has a miss detection rate of more than 15% in the above strong light and high reflection scenarios, which cannot meet the low-altitude security needs of key areas such as airports, nuclear power plants and military management areas; at the same time, the single-modal method has a misjudgment rate of more than 20% in strong light, high reflection and other complex environments, and the detection accuracy and robustness still cannot meet the requirements of practical applications.

[0006] Therefore, in order to improve the robustness and real-time performance of the unmanned aerial vehicle detection, especially in strong light environment, it has important practical significance and application value to study the multi-modal fusion unmanned aerial vehicle detection technology based on the existing multi-modal unmanned aerial vehicle detection sensor, and give full play to the complementary advantages of radar-infrared-visible light. SUMMARY

[0007] The technical problem to be solved by the present application is to overcome the defects existing in the prior art. The present application proposes a radar-infrared-visible light multi-modal fusion unmanned aerial vehicle detection method, aiming to solve the problem of performance degradation of unmanned aerial vehicle detection in strong light environment. The present application introduces a two-dimensional Gaussian probability graph generated by radar track generation as a spatial prior, guides the visual modal to focus on the potential target area, and combines the weighted fusion of infrared and visible light features to realize the cross-modal deep semantic level complementarity, and improve the accuracy of detection. By fusing the feature information of phased array radar, visible light image and infrared (supplementary light) image, the unmanned aerial vehicle detection precision is significantly improved and the fast reasoning speed is ensured, which is suitable for low-altitude unmanned aerial vehicle safety monitoring tasks in strong light environment. The method belongs to the cross technical field of multi-modal sensor information fusion and computer vision, and is especially suitable for low-altitude unmanned aerial vehicle monitoring scenes in strong reflection background such as urban high-rise glass curtain wall area, lake surface and large metal venue.

[0008] To solve the above technical problems, the technical scheme adopted by the present application is: a radar-infrared-visible light multi-modal fusion unmanned aerial vehicle detection method, comprising:

[0009] S1: input the obtained unmanned aerial vehicle infrared image and visible light image into two visual branches of a double-flow feature extraction network respectively, and extract scale features by using a lightweight convolutional backbone network;

[0010] S2: real-time acquire three-dimensional coordinate data of the unmanned aerial vehicle after radar detection and clutter removal processing, and project the three-dimensional coordinate data into two-dimensional image coordinates through a camera external parameter matrix;

[0011] S3: generate a two-dimensional Gaussian distribution probability graph based on the real-time two-dimensional image coordinates, and input the two-dimensional Gaussian distribution probability graph as a mask into a radar feature extraction network to extract a radar feature map;

[0012] S4: under the same scale, dynamically weight and fuse the scale features of the infrared branch and the visible light branch to generate a visual fusion feature map;

[0013] S5: channel dimensionally splice the visual fusion feature map and the radar feature map under the same scale to form a three-modal fusion feature map;

[0014] S6: input the three-modal fusion feature map into a detection head network to generate unmanned aerial vehicle target information.

[0015] As a preferred scheme of the present application, wherein: in S1, the scale features include shallow texture features and deep semantic features;

[0016]

[0017] wherein, is the first layer feature of the visible light branch, is the first layer feature of the visible light branch, is the first layer feature of the infrared branch, is the first layer feature of the infrared branch, and are the visible light original image and the infrared original image respectively, represents the first layer feature extraction operation.

[0018] As a preferred scheme of the present application, wherein: in S3, the method for extracting the radar feature map by inputting the two-dimensional Gaussian distribution probability map as a mask into the radar feature extraction network includes: extracting shallow position features and deep semantic features whose spatial prior is contained, and the expression of the first layer feature extraction is as follows:

[0019]

[0020] wherein, is the first layer feature of the radar branch, is the first layer feature of the radar branch, is the radar original probability map, represents the first layer radar feature extraction operation.

[0021] As a preferred scheme of the present application, wherein: in S4, the method for generating the visual fusion feature map includes: the fusion formula is as follows:

[0022] , ,

[0023] wherein, is the first layer visual fusion feature map, is an adaptive weight coefficient, and its value range is: , which is dynamically adjusted according to the light intensity of the detection scene, wherein the stronger the light intensity is, the smaller the value is, and the greater the infrared feature weight is.

[0024] As a preferred scheme of the present application, wherein: in S5, the visual fusion feature map is channel-dimensionally spliced with the radar feature map under the same scale to form a three-modal fusion feature map, and the formula is as follows:

[0025]

[0026] wherein, denotes a channel-dimension splicing operation, is the first layer three-modal fusion feature map.

[0027] As a preferred scheme of the present application, wherein: in S6, the three-modal fusion feature map is input into a detection head network to generate the unmanned aerial vehicle target information, and the expression is as follows:

[0028]

[0029] wherein, denotes a detection head prediction network, contains target boundary box coordinates , class label and confidence.

[0030] A computer readable storage medium has a computer program stored thereon, and the computer program performs the above-mentioned unmanned aerial vehicle detection method based on radar, infrared and visible light multi-modal fusion when running.

[0031] A computer device includes a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor performs the above-mentioned unmanned aerial vehicle detection method based on radar, infrared and visible light multi-modal fusion when executing the program.

[0032] The first aspect of the present application provides a radar-based Gaussian probability map generation method, specifically comprising the following steps:

[0033] Real-time acquisition of three-dimensional track information of an unmanned aerial vehicle detected by a phased array radar after de-cluttering processing;

[0034] A pinhole projection model is used to map the radar points to the image plane coordinate system through an external parameter matrix transformation to obtain corresponding image coordinate points;

[0035] Based on a radar point track to Gaussian probability generation algorithm, the two-dimensional image coordinate points obtained above are taken as the center, the diffusion degree of the Gaussian kernel is controlled according to the projection size of the unmanned aerial vehicle and the detection distance, and a two-dimensional Gaussian probability distribution map consistent with the input source size of the infrared image and the visible light image is generated in real time.

[0036] By the above design of adaptively adjusting the diffusion degree of the Gaussian kernel, compared with a traditional visual network, the method can improve the attention degree of the visual network to a potential target region by more than 30%, effectively reduces the interference of irrelevant background regions under strong light, and improves the fault tolerance of target positioning.

[0037] The second aspect of the application provides a radar-driven multi-modal semantic fusion method, specifically comprising the following steps:

[0038] Real-time acquisition of visible light video frames and infrared video frames captured by a camera;

[0039] A double-flow feature extraction network is built to extract shallow texture information and deep semantic information of the visible light video frames and the infrared video frames;

[0040] Real-time acquisition of a two-dimensional Gaussian probability distribution frame generated by the above Gaussian probability generation algorithm based on radar point traces;

[0041] A radar semantic feature extraction network is built to extract shallow position information and deep semantic information of the two-dimensional Gaussian probability graph;

[0042] A three-modal semantic fusion network is built to perform weighted fusion on the shallow texture information and deep semantic information of the visible light, infrared and radar after feature extraction;

[0043] A path aggregation network is built to input the feature layers after weighted fusion into the cross-path aggregation network for feature splicing;

[0044] A detection head network is built to detect the three different size feature layers output by the path aggregation network and output the results.

[0045] In the above weighted fusion process, by dynamically adjusting the weight proportion of the infrared features and the visible light features, the advantages of the infrared modal in resisting strong light interference and the advantages of the visible light modal in retaining details can be fully utilized, and the detection error when a single modal fails can be reduced.

[0046] Compared with the prior art, the application has the following beneficial effects:

[0047] Mechanism innovation: a two-dimensional Gaussian probability graph generation mechanism based on radar projection is proposed, the spatial positioning stability of the radar is converted into visual attention prior information, and through explicit projection transformation and adaptive adjustment of the Gaussian kernel, the positioning error problem caused by image overexposure and texture loss of the visual modal under strong light is directly solved;

[0048] Architecture innovation: a three-modal fusion architecture of "double-flow visual feature extraction + radar semantic extraction" is constructed, breaking through the limitations of shallow feature splicing or simple weighting in traditional multi-modal fusion, realizing deep semantic complementation of radar, infrared and visible light through deep semantic extraction of different modalities and cross-modal collaborative correlation.

[0049] Strategy innovation: design light adaptive dynamic weighting fusion strategy, adjust the weight proportion of infrared and visible light features in real time according to the actual light intensity of the detected scene, and accurately adapt to the detection needs of different sub-scenes (such as glass curtain wall reflection and water reflection) under strong light;

[0050] By projecting the radar Gaussian probability graph to guide the visual attention area, when the visual modality fails due to overexposure and glare in strong light scenes, the target positioning accuracy can still be guaranteed, and the positioning fault tolerance is improved by more than 25% compared with the traditional dual-modality method;

[0051] By utilizing the complementary characteristics of infrared and visible light, the saliency and texture details of the unmanned aerial vehicle target are enhanced, and the miss detection rate is reduced by more than 12% compared with the single-modality method in strong reflection background;

[0052] The three-modality deep semantic fusion architecture improves the detection accuracy (AP@0.5) by 3.9% compared with the single-modality method (such as YOLOv5s) and by 2.2% compared with the dual-modality fusion method (such as SLBAF-Net) in real strong light environment (light intensity > 50000lux);

[0053] The physical system has been verified, and on a high-performance server equipped with an Intel Core i9-10900HX processor and an NVIDIA RTX3090 graphics card, the inference speed reaches 25fps, meeting the real-time detection requirements and having engineering landing ability, which can be directly applied to low-altitude unmanned aerial vehicle security monitoring scenes. BRIEF DESCRIPTION OF DRAWINGS

[0054] The disclosure of the present application is explained with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present application. In the drawings, the same reference numerals are used to refer to the same parts. Among them:

[0055] Figure 1 The flowchart in the embodiment of the present disclosure is shown;

[0056] Figure 2 The performance comparison chart of the embodiment of the present application and other algorithms is shown; Figure 2 The overall structure of the multi-modality semantic fusion is shown;

[0057] Figure 3 The performance comparison chart of the embodiment of the present application and other algorithms is shown;

[0058] Figure 4 The application result chart of the present application is shown. DETAILED DESCRIPTION

[0059] It is easy to understand that the person skilled in the art can propose alternative implementations without changing the essential spirit of the application. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solutions of the application, and should not be considered as the whole or as a limitation or restriction on the technical solutions of the application.

[0060] The radar track Gaussian probability graph generation method provided by the application realizes the transfer of radar spatial positioning information to the visual mode by projecting the radar three-dimensional track point to the image plane and generating a Gaussian probability distribution graph. In implementation, the phased array radar and the Hikvision visible-infrared integrated holder camera (hereinafter referred to as “optoelectronic camera”) are erected at the same position, the same altitude and consistent orientation, so as to ensure that the coordinate systems of the radar and the optoelectronic camera can be aligned through simple transformation, and the specific steps are as follows:

[0061] 1. Space coordinate conversion from radar to optoelectronic

[0062] The conversion process includes the following three steps:

[0063] (1) Radar track point coordinate conversion

[0064] The radar output track point is based on its own radar coordinate system, including distance (R), azimuth angle (θ) and pitch angle (φ) three parameters, which need to be converted into the same geodetic coordinate system as the optoelectronic, which includes the following steps:

[0065] Radar polar coordinate to radar rectangular coordinate

[0066] The rectangular coordinate system (X, Y, Z) is established with the phase center of the radar antenna as the origin, the X-axis pointing forward, the Y-axis pointing right, and the Z-axis pointing up. The three-dimensional rectangular coordinates of the track point are calculated by trigonometric functions, and the calculation formula is as follows:

[0067]

[0068] Among them, is the radar detection distance (unit: m), is the azimuth angle (unit: rad, value range: -π / 2~π / 2), is the pitch angle (unit: rad, value range: 0~π / 2).

[0069] Radar rectangular coordinate system to geodetic coordinate system

[0070] ​​​​​​Since the radar and electro-optical camera are already mounted at the same location and altitude, the origin of the radar's Cartesian coordinate system (i.e., the phase center of the radar antenna) can be directly aligned with the geodetic coordinates of the electro-optical camera (including latitude Lat, longitude Lon, and altitude Alt). The geodetic coordinates of the track point can then be obtained through translation transformation (without rotation or scaling). The calculation formula is as follows:

[0071]

[0072] in, The geodetic coordinates of the radar antenna phase center (i.e., the geodetic coordinates of the photoelectric camera).

[0073] (2) Camera internal and external parameter calibration

[0074] Camera internal reference ( ): Describes the camera's own optical characteristics, determined by the camera's hardware parameters, obtained through the checkerboard calibration method, including focal length. Image center point The distortion coefficients (no distortion correction is needed in this embodiment, so the distortion coefficients are set to 0) are shown in the following intrinsic parameter matrix form:

[0075]

[0076] in, , The camera is in , Focal length in direction (unit: pixel). , The center point of the image is at , The coordinates of the direction (unit: pixel).

[0077] Camera external parameters ( ): Describes the attitude and position of the photoelectric camera in the geodetic coordinate system. Since the radar and photoelectric camera are "at the same location, at the same altitude, and facing the same direction," the external parameters can be simplified in setting.

[0078] Translation vector T: The geodetic coordinates of the camera optical center are the same as those of the radar origin, so T = (0,0,0) (unit: m);

[0079] Rotation matrix R: Photoelectric camera coordinate system ( The axis moves forward along the optical axis. Axial downwards, The rotation relationship between the axis to the right and the radar rectangular coordinate system is fixed. If both are oriented due north, then the rotation matrix... For identity matrix:

[0080]

[0081] No additional calibration is needed.

[0082] (3) Projection calculation: from geodetic coordinates to image coordinates

[0083] After the above two steps, the pixel coordinates of the track point on the image are obtained through the two-step conversion of "geodetic coordinates to camera coordinate system and then to image coordinate system", as follows:

[0084] Geodetic coordinates to camera coordinate system

[0085] Substitute the geodetic coordinates of the track point into the rotation matrix and the translation vector to convert it into three-dimensional coordinates in the camera coordinate system , and the calculation formula is as follows:

[0086]

[0087] In this embodiment, T = 0, so the formula is simplified as:

[0088]

[0089] Camera coordinate system to image coordinate system

[0090] Project the three-dimensional camera coordinates into two-dimensional image coordinates through the camera intrinsic matrix , which is specifically divided into:

[0091] First, calculate the normalized image coordinates :

[0092]

[0093] where is the distance from the point to the camera optical center, which ensures the correct projection scale. Then, calculate the pixel coordinates:

[0094]

[0095] In this embodiment, there is no need to use distortion coefficients to correct the pixel coordinates.

[0096] 2. Two-dimensional Gaussian probability map generation

[0097] According to the physical size of the unmanned aerial vehicle and the detection distance of the radar, the scale parameter of the two-dimensional Gaussian distribution is determined to generate the unmanned aerial vehicle positioning probability distribution map. In the image plane, take the pixel coordinates obtained in step 1.3 as the center to construct a two-dimensional Gaussian response region, and generate a size of ( Width of image, unit: pixel Probability map of image height, unit: pixel Response value at pixel point The calculation formula is as follows:

[0098]

[0099] Wherein, The coordinates of any pixel point on the plane in the image, The projection coordinates of the radar track point on the image plane, The standard deviation of the Gaussian kernel (unit: pixel), used to control the diffusion degree of the Gaussian kernel, which is determined by the projection size of the UAV and the radar detection distance, and the calculation formula is as follows:

[0100]

[0101] Wherein, The radar detection distance (unit: m), The projection size of the UAV on the image plane (unit: pixel), The scaling constant, in this embodiment =0.8.

[0102] (2) Radar-driven multi-modal semantic fusion

[0103] 1. Feature extraction

[0104] Visual model feature extraction: the infrared image and the visible light image are respectively input into the two visual branches of the double-flow feature extraction network, and a lightweight convolutional backbone network is used to extract multi-scale features, including shallow texture features and deep semantic features, and the expression of the first layer feature extraction is as follows:

[0105]

[0106] Wherein, The first layer feature of the visible light branch, The first layer feature of the visible light branch, The first layer feature of the infrared branch, The first layer feature of the infrared branch, The first layer feature of the infrared branch, The first layer feature of the infrared branch, The visible light original image and the infrared original image, The first layer feature extraction operation.

[0107] ​​​​​Radar model feature extraction: input the radar probability map as a mask into the independent radar feature extraction network to extract the shallow position features and deep semantic features contained in the spatial prior of the radar probability map. The expression of the layer feature extraction is as follows:

[0108]

[0109] wherein, is the first layer feature of the radar branch, is the first layer feature of the radar branch, is the original radar probability map, represents the first layer radar feature extraction operation.

[0110] 2. Feature fusion

[0111] Intra-modal fusion of the visual modality: the intermediate features of the infrared branch and the visible light branch are dynamically weighted and fused under the same scale, and the fusion formula is as follows:

[0112]

[0113] wherein, is the first layer visual fusion feature map, is the adaptive weight coefficient (the value range is: ), which is dynamically adjusted according to the light intensity of the detection scene (the stronger the light intensity is, the smaller the is, and the greater the infrared feature weight is).

[0114] 3. Three-modal fusion

[0115] The visual fusion feature map and the radar feature map under the same scale are spliced in the channel dimension to form a three-modal deep fusion feature map, and the formula is as follows:

[0116]

[0117] wherein, represents the channel dimension splicing operation, is the first layer three-modal fusion feature map.

[0118] 4. Detection output

[0119] The three-modal fusion feature map is input into the detection head network to generate the class, bounding box and confidence information of the unmanned aerial vehicle target, and the expression is as follows:

[0120]

[0121] wherein,​​​​​ a detection head prediction network, containing target bounding box coordinates , class labels and confidence.

[0122] System implementation

[0123] The multi-modal semantic fusion system based on radar-visible light-infrared (supplementary light) includes a hardware subsystem and an algorithm subsystem, and the specific configuration is as follows:

[0124] 1. Hardware subsystem

[0125] Radar module: a two-dimensional active phased array pulse Doppler system radar is adopted, the working frequency band is 15.6~16.9GHz, the detection distance range is 200~2000m, the azimuth coverage range is-45°~45°, and the pitch angle coverage range is 0°~90°.

[0126] Optoelectronic imaging module: a visible light-infrared integrated pan-tilt camera of Hikvision is adopted, the visible light and infrared image resolution is 1920×1080pixel, the frame rate is 25fps, it is connected with an edge server through a gigabit Ethernet (RJ45 interface), the time synchronization is realized by using NTPv4 protocol, and the synchronization error is ≤50ms.

[0127] Ground computing module: an industrial-grade edge server with Intel Core i9-10900HX processor (10 cores 20 threads, basic frequency 2.7GHz), NVIDIA RTX 3090 graphics card (24GB GDDR6X video memory) and 128GB DDR4 memory is adopted, which is used for deploying algorithm model, and the inference speed reaches 25fps, meeting the real-time detection requirement.

[0128] 2. Algorithm subsystem

[0129] The algorithm subsystem is deployed in the ground computing module, realizes real-time data processing and model inference, and mainly includes two core modules:

[0130] Radar Gaussian probability map generation module: based on the above radar track Gaussian probability map generation method, the radar three-dimensional track information is converted into a two-dimensional Gaussian probability map, and radar semantic prior information is provided;

[0131] Three-modal semantic fusion module: based on the above radar-driven multi-modal semantic fusion method, the deep semantic features of radar-infrared-visible light are fused, the defect of visible light modal failure under strong light is made up, and the unmanned aerial vehicle detection result is output.

[0132] Implementation effect

[0133] To verify the effectiveness of the method, a multi-modal image dataset is collected under strong light scenes for experimental verification. The experimental configuration and results are as follows:

[0134] (I) Experimental dataset configuration

[0135] Data volume and scene: More than 10,000 multi-modal images (visible light, infrared images each more than 5,000), of which strong light scenes (light intensity > 50,000 lux) account for more than 50%;

[0136] Target and interference: including 1 unmanned aerial vehicle target (DJI Air 3S, body size 325.47x266.11x106.00mm, flight height 50m~120m) and 1 interference target (flying bird);

[0137] Data labeling: labeled using LabelImg tool, boundary box labeling accuracy ≤5 pixels, class labeled as "UAV", labelers are professionally trained, and labeling consistency Kappa coefficient > 0.9;

[0138] (II) Experimental evaluation index

[0139] Five commonly used evaluation indexes in the field of target detection are used, and the specific definitions are as follows:

[0140] Precision (P): the proportion of true positives (TP) in all positive detections (TP+FP), the calculation formula is:

[0141]

[0142] Recall (R): the proportion of all true positives (TP) in labeled positives (TP+FN), the calculation formula is:

[0143]

[0144] Average precision (AP@0.5): the average value of precision corresponding to different recall rates under the condition that the IoU threshold is 0.5;

[0145] Average precision mean (AP50:95): the average value of AP under the condition that the IoU threshold is 0.5~0.95 (step 0.05);

[0146] F1 score: the harmonic mean of precision (Precision) and recall (Recall), the calculation formula is:

[0147]

[0148] The results of the specific embodiments are shown in the attached Figure 3 .

[0149] The technical scope of the present application is not limited to the above description, and those skilled in the art can make various modifications and changes to the above embodiments without departing from the technical idea of the present application, and these modifications and changes should all belong to the protection scope of the present application.

Claims

1. A method for detecting unmanned aerial vehicles (UAVs) based on radar infrared and visible light multimodal fusion, characterized in that, include: S1: Input the acquired UAV infrared image and visible light image into the two visual branches of the dual-stream feature extraction network respectively, and use a lightweight convolutional backbone network to extract scale features; S2: Real-time acquisition of UAV 3D coordinate data after radar detection and clutter removal, and projection of 3D coordinate data into 2D image coordinates through camera extrinsic matrix; S3: Generate a two-dimensional Gaussian probability map based on real-time two-dimensional image coordinates, and use the two-dimensional Gaussian probability map as a mask to input the radar feature extraction network to extract radar feature maps; S4: At the same scale, the scale features of the infrared branch and the visible light branch are dynamically weighted and fused to generate a visual fusion feature map; S5: The visual fusion feature map is stitched together with the radar feature map at the same scale by channel dimension to form a three-modal fusion feature map; S6: Input the three-modal fused feature map into the detection head network to generate UAV target information.

2. The UAV detection method based on radar infrared and visible light multimodal fusion according to claim 1, characterized in that, In S1, the scale features include shallow texture features and deep semantic features; ， in, For the visible light branch Layer features, For the visible light branch Layer features, For the infrared branch Layer features, For infrared Layer features, and These are the original visible light image and the original infrared image, respectively. Indicates the first Feature extraction operations for layers.

3. The UAV detection method based on radar infrared and visible light multimodal fusion according to claim 2, characterized in that, In S3, the method of using a two-dimensional Gaussian probability map as a mask input to the radar feature extraction network to extract radar feature maps includes: extracting its shallow spatial prior-implied location features and deep semantic features, the first... The expression for layer feature extraction is as follows: ， in, For radar branch number Layer features, For radar branch number Layer features, This is the original probability map of the radar. Indicates the first Layered radar feature extraction operation.

4. The UAV detection method based on radar infrared and visible light multimodal fusion according to claim 3, characterized in that, In S4, the method for generating the visual fusion feature map includes the following fusion formula: ， in, For the first Layered visual fusion feature maps The adaptive weighting coefficient has the following range of values: It dynamically adjusts according to the light intensity of the detection scene, where the stronger the light intensity, the better. The smaller the value, the greater the infrared feature weight.

5. The UAV detection method based on radar infrared and visible light multimodal fusion according to claim 4, characterized in that, In S5, the visual fusion feature map is concatenated with the radar feature map at the same scale along the channel dimension to form a three-modal fusion feature map, as shown in the following formula: ， in, This indicates a channel-level concatenation operation. For the first Three-modal fusion feature map.

6. The UAV detection method based on radar infrared and visible light multimodal fusion according to claim 5, characterized in that, In S6, the three-modal fused feature map is input into the detection head network to generate UAV target information, as expressed below: ， in, This indicates the detection head prediction network. Includes target bounding box coordinates Category labels and confidence levels.

7. A UAV detection system based on radar infrared visible light multimodal fusion, characterized in that, The UAV detection method based on radar infrared and visible light multimodal fusion according to any one of claims 1-6 includes: The radar module is used to acquire the UAV's three-dimensional coordinate data after radar detection and clutter removal, and project the three-dimensional coordinate data into two-dimensional image coordinates through the camera extrinsic matrix; The photoelectric imaging module is used to input the acquired UAV infrared image and visible light image into the two visual branches of the dual-stream feature extraction network, respectively, and to extract scale features using a lightweight convolutional backbone network. The computation module includes a radar Gaussian probability map generation module and a three-modal semantic fusion module. The module generates a two-dimensional Gaussian probability map based on real-time two-dimensional image coordinates. It then uses this two-dimensional Gaussian probability map as a mask to input into a radar feature extraction network to extract radar feature maps. At the same scale, it dynamically weights and fuses the scale features of the infrared and visible light branches to generate a visual fusion feature map. The visual fusion feature map is then concatenated with the radar feature map at the same scale along the channel dimension to form a three-modal fusion feature map. Finally, the three-modal fusion feature map is input into a detection head network to generate UAV target information.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it performs the UAV detection method based on radar infrared visible light multimodal fusion as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a UAV detection method based on radar infrared visible light multimodal fusion as described in any one of claims 1-6.

Images