A multi-modal detection information fusion method based on environment assessment

Abstract

This invention relates to the field of UAV detection technology and discloses a multimodal detection information fusion method based on environmental assessment. This method obtains the coordinate transformation matrix between radar and camera through calibration, achieving spatiotemporal alignment of point clouds and images. Visual and radar feature maps are extracted in parallel from the aligned data, and sensor reliability is simultaneously assessed based on image information entropy and point cloud clutter density, generating a visual weight factor α and a radar weight factor β. The dual-mode features are adaptively weighted and fused using these weight factors, and after channel integration and target detection network processing, the UAV category and location are finally output. This invention significantly improves the system's robustness, detection accuracy, and anti-false alarm capability under extreme weather and interference scenarios through real-time environmental perception and dynamic weight adjustment, achieving reliable all-weather detection.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) detection technology and discloses a multimodal detection information fusion method based on environmental assessment. Background Technology

[0002] With the rapid popularization of drone technology and the increasing prominence of the threat posed by "low, slow, and small" targets, the rapid and accurate detection and identification of drones in the airspace has become an urgent need in fields such as public safety and key area defense. A reliable drone detection system is a prerequisite for effective early warning and response, and its core challenge lies in how to achieve stable and robust perception of various drones in complex and ever-changing environments. Currently, mainstream automated detection technologies mainly rely on the following types of solutions: First, optical detection solutions based on a single visible light camera. This solution is low-cost and can provide intuitive and rich image information under good lighting and simple background conditions. However, its performance is highly dependent on ambient light and weather; its detection capability drops sharply under low visibility conditions such as nighttime, fog, haze, rain, and snow, and it is easily interfered with by target-like objects in the background. Second, detection solutions based on a single millimeter-wave radar. Millimeter-wave radar has all-weather, all-day operation capabilities, is unaffected by lighting conditions, and can directly acquire target distance and speed information. However, in complex electromagnetic environments such as cities and building clusters, radar is susceptible to multipath effects, ground clutter, and electromagnetic interference, leading to an increased false alarm rate. Furthermore, its sensitivity to non-metallic or small targets is limited, and it struggles to provide intuitive target type identification information. Thirdly, some solutions employ fixed, simple data-level or decision-level fusion schemes combining radar and vision. These schemes attempt to combine the advantages of both, typically displaying radar detection results in conjunction with camera images, or weighting the results after detection. However, existing fusion methods often use fixed rules or static weights, lacking the ability to perceive dynamic environmental changes. When a sensor's performance is severely degraded due to environmental interference (such as fog causing visual failure or strong clutter causing radar false alarms), low-quality data or decisions directly "contaminate" the fusion results, causing the overall system performance to be even lower than that of a single reliable sensor, failing to achieve significant robustness improvements.

[0003] While the aforementioned solutions all aim to achieve effective UAV detection, they generally present irreconcilable contradictions between detection accuracy, environmental adaptability, system reliability, and false alarm control. Existing technologies either sacrifice all-weather capability in pursuit of high-precision identification or suffer from high false alarm rates or loss of type discrimination information to ensure all-weather operation. Especially in the context of complex environments and intelligent interference, how to construct an intelligent fusion perception system capable of real-time assessment of sensor reliability and dynamic adjustment of fusion strategies, thereby maintaining or even enhancing the overall system detection performance even when the performance of any single sensor deteriorates, remains a key technological bottleneck that urgently needs to be overcome in this field. Therefore, there is an urgent need for an innovative multimodal fusion architecture that can fundamentally and synergistically improve the environmental robustness, perception accuracy, and decision-making intelligence of the detection system. Summary of the Invention

[0004] This application provides a multimodal detection information fusion method based on environmental assessment to solve the technical problem that the lack of perception capability for dynamic environmental changes in existing fusion detection schemes leads to a decrease or even failure of fusion effect when the performance of single sensors degrades. It realizes an intelligent fusion perception process that can evaluate the quality of sensor data in real time and adaptively adjust the fusion weights, thereby maintaining high robustness and high precision of UAV detection in all weather and complex environments.

[0005] This application provides a multimodal detection information fusion method based on environmental assessment, the method comprising:

[0006] Simultaneously acquire radar point clouds and visible light images, and map the radar coordinate system corresponding to the radar point cloud to the image pixel coordinate system corresponding to the visible light image to obtain spatiotemporally aligned three-dimensional point clouds and two-dimensional images;

[0007] The spatiotemporally aligned 2D image is input into the visual processing branch, and the spatiotemporally aligned 3D point cloud is simultaneously input into the radar processing branch.

[0008] The visual processing branch's processing flow is as follows: The two-dimensional image is input into a convolutional neural network for visual feature extraction to obtain a deep visual feature map; simultaneously, the information entropy of the target region in the two-dimensional image is calculated, and a visual weight factor is generated based on the information entropy through a first mapping function. ;

[0009] The radar processing branch process is as follows: After voxelizing the 3D point cloud, it is input into the point cloud network for radar geometric feature extraction to obtain a pseudo-feature map with the same size as the deep visual feature map; simultaneously, the clutter density is calculated based on the number of clutter points in the target area of ​​the 3D point cloud, and a radar weighting factor is generated based on the clutter density through a second mapping function. ;

[0010] Based on the visual weight factor and radar weighting factor The deep visual feature map and the pseudo feature map are weighted and fused to obtain a fused feature map;

[0011] The fused feature map is integrated through a 1×1 convolutional layer and then input into the target detection network. The final output includes at least the drone target category and the drone target bounding box coordinates.

[0012] Optionally, radar point clouds and visible light images are acquired simultaneously, and the radar coordinate system corresponding to the radar point cloud is mapped to the image pixel coordinate system corresponding to the visible light image to obtain spatiotemporally aligned 3D point clouds and 2D images, including:

[0013] Multiple sets of radar calibration point clouds and calibration visible light images are collected for a calibration board. The coordinate transformation matrix is ​​calculated based on the radar calibration point clouds and the calibration visible light images.

[0014] Simultaneously acquire radar point cloud and visible light image, and project the radar point cloud onto the image pixel coordinate system corresponding to the visible light image based on the coordinate transformation matrix to obtain spatiotemporally aligned 3D point cloud and 2D image.

[0015] Optionally, calculating the information entropy of the target region in the two-dimensional image includes:

[0016] Obtain the grayscale image of the target region in the two-dimensional image;

[0017] Statistically analyze the grayscale histogram of the grayscale image to obtain each grayscale level. probability of occurrence ;

[0018] Based on the probability of occurrence Through formula The information entropy is calculated. .

[0019] Optional, also includes:

[0020] Set a first threshold and a second threshold, when the information entropy When the image quality is above the first threshold or below the second threshold, the two-dimensional image quality is determined to have degraded, and a visual weight factor is generated using the first mapping function. Specifically:

[0021]

[0022] in, The baseline entropy value under ideal conditions. This is the variance parameter, which is used as an adjustable parameter to adjust the visual weight factor. Information entropy Deviation from baseline entropy value Sensitivity.

[0023] Optionally, the clutter density is calculated based on the number of clutter points within the target area of ​​the 3D point cloud, and a radar weighting factor is generated based on the clutter density using a second mapping function. ,include:

[0024] Centered on the current detection location, within a three-dimensional spatial neighborhood of a preset radius, count the number of points with reflection intensity lower than a preset intensity threshold, and use this count as the number of clutter points.

[0025] The clutter density is calculated based on the number of clutter points and the volume of the three-dimensional spatial neighborhood.

[0026] Based on the clutter density, a radar weighting factor is generated using the second mapping function. Specifically:

[0027]

[0028] in, clutter density, This represents the upper limit of clutter density tolerance. This is the adjustment coefficient.

[0029] Optionally, based on the visual weighting factor and radar weighting factor The deep visual feature map and the pseudo-feature map are weighted and fused to obtain a fused feature map, including:

[0030] Based on the visual weight factor The deep visual feature map is weighted to obtain a weighted visual feature map; simultaneously, based on the radar weighting factor... The pseudo-feature map is weighted to obtain a weighted pseudo-feature map;

[0031] The weighted visual feature map and the weighted pseudo-feature map are added element-wise to obtain the fused feature map. The specific process is represented by the following weighted summation formula:

[0032]

[0033] in, This is a deep visual feature map. It is a pseudo feature map. To fuse feature maps.

[0034] Optional, visual weighting factor and the radar weighting factor All are normalized values ​​between 0 and 1; wherein, the visual weight factor The higher the value, the higher the confidence level of the current two-dimensional image; the radar weighting factor. The higher the value, the higher the confidence level of the current 3D point cloud.

[0035] Optionally, the two-dimensional image is input into a convolutional neural network for visual feature extraction to obtain a deep visual feature map, and the three-dimensional point cloud is voxelized and then input into a point cloud network for radar geometric feature extraction to obtain a pseudo-feature map with the same size as the deep visual feature map, including:

[0036] The two-dimensional image is input into the backbone of a convolutional neural network based on the CSPDarkNet architecture;

[0037] Multi-scale visual feature maps are extracted from the third layer of the convolutional neural network backbone as the deep visual feature maps;

[0038] Simultaneously, the three-dimensional point cloud is divided into a regular voxel grid to obtain voxelized point cloud data;

[0039] The voxelized point cloud data is input into the point cloud network PointNet++ to extract radar geometric features;

[0040] The extracted radar geometric features are transformed in space dimension and mapped to a pseudo-feature map with the same size as the deep visual feature map.

[0041] Optionally, the fused feature maps are integrated through a 1×1 convolutional layer and then input into the target detection network. The final output includes detection results that at least contain the UAV target category and the coordinates of the UAV target bounding box, including:

[0042] The fused feature map is channel-integrated through a 1×1 convolutional layer;

[0043] The processed fused feature map is input into the target detection network for category classification and bounding box regression to obtain the initial detection result; non-maximum suppression is applied to the initial detection result to obtain the deduplicated final detection box; the coordinates of the final detection box are mapped back to the image pixel coordinate system of the visible light image, and the final output is the detection result containing the UAV target category and the UAV bounding box coordinates.

[0044] Optionally, the method further includes optimizing the parameters in the convolutional neural network, point cloud network, first mapping function, and second mapping function by minimizing the total loss function, wherein the total loss function includes classification loss, localization loss, and a term related to the visual weight factor. and radar weighting factor Related weight consistency loss.

[0045] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present invention include at least the following:

[0046] This invention addresses the problem of overall performance degradation caused by single sensor failure in complex environments by constructing an adaptive multimodal fusion architecture based on real-time environmental perception. It achieves spatiotemporal consistent fusion of multi-source information by synchronously acquiring and precisely aligning visual and radar data. By introducing a parallel evaluation mechanism based on image information entropy and point cloud clutter density, it generates dynamic visual weighting factors α and radar weighting factors β in real time, enabling the fusion strategy to intelligently adjust according to environmental changes. This mechanism effectively suppresses interference from low-quality sensor data on the fusion results, ensuring that even when any sensor is affected by environmental interference, the system can still select reliable information based on the weighting factors, thus significantly improving detection robustness and accuracy under all-weather conditions. The entire method is clear, can be embedded and deployed, and achieves efficient, stable, and intelligent detection of "low, slow, and small" UAV targets. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 The overall flowchart of the multimodal detection information fusion method provided by the present invention;

[0049] Figure 2 This is a schematic diagram of the adaptive fusion module structure provided by the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0052] Example 1

[0053] As the background technology described above indicates, achieving all-weather, robust, and accurate detection of "low, slow, and small" unmanned aerial vehicles (UAVs) is a crucial aspect of airspace security and key area defense. However, existing detection solutions, relying on single visible light cameras, single millimeter-wave radars, or multimodal systems employing fixed-rule fusion, while capable of detection under specific conditions, generally suffer from weak environmental adaptability, high false alarm rates, and reduced or even failed fusion performance when a single sensor is interfered with. These solutions cannot dynamically assess the reliability of data from each sensor in complex and changing environments, making it difficult to simultaneously achieve detection accuracy, robustness, and real-time performance. Consequently, they fail to meet the practical requirements for continuous, stable, and intelligent perception of highly concealed and maneuverable UAV targets.

[0054] Therefore, this application provides a multimodal detection information fusion method based on environmental assessment, such as... Figure 1 and Figure 2 As shown, the method includes:

[0055] S100, sensor calibration and coordinate transformation matrix acquisition.

[0056] Before system deployment, joint calibration of the millimeter-wave radar and visible light camera is required to obtain the mapping relationship between their coordinate systems. Specifically, a calibration board containing both high reflectivity (for radar identification) and high-contrast patterns (for camera identification) is used. The calibration board is positioned at multiple different locations and orientations within the common field of view of the sensors, simultaneously acquiring multiple sets of calibration point clouds from the millimeter-wave radar and calibration 2D images from the visible light camera. By matching the feature points of the calibration board extracted from each set of data (corner positions in the radar point cloud and corner pixel coordinates in the image), and using optimization algorithms such as least squares, a coordinate transformation matrix from the radar coordinate system to the camera coordinate system is calculated. This matrix will be used for spatial alignment of all subsequent real-time data.

[0057] S200, real-time data synchronous acquisition and spatiotemporal alignment.

[0058] During real-time detection, the system strictly synchronizes the acquisition times of the millimeter-wave radar and the visible light camera through hardware triggers or high-precision software timestamps. At the same time, the millimeter-wave radar acquires the raw 3D point cloud, and the visible light camera acquires the raw 2D visible light image.

[0059] Based on the coordinate transformation matrix pre-stored in S100, the original 3D point cloud is projected onto the pixel coordinate system of the original 2D visible light image to obtain a spatially aligned 3D point cloud and a 2D image. This projection process specifically includes:

[0060] Let the coordinates of a point in the radar coordinate system be... To perform projection calculations, we first represent it in homogeneous coordinate form. .

[0061] The coordinate transformation matrix (typically a 3×4 matrix) contains the rotation, translation, and camera intrinsic parameters from the radar coordinate system to the camera coordinate system. Using this matrix... Transform the point and calculate its homogeneous coordinates in the image pixel coordinate system:

[0062]

[0063] in:

[0064] : The calculated homogeneous coordinates in the image pixel coordinate system. Here It is a non-zero scaling factor.

[0065] Subsequently, the homogeneous coordinates are normalized to obtain the standard two-dimensional pixel coordinates corresponding to the point. ,in:

[0066]

[0067] Finally, all projected points are filtered: only those points whose calculated pixel coordinates are selected are considered valid. satisfy and Time (of which) and The point cloud data is preserved only after the width and height of the original 2D image are determined (these are the width and height, respectively). This step ensures that the projected points are within the effective visual range of the image.

[0068] After the above calculations and filtering, the system obtains spatially and spatiotemporally aligned 3D point cloud and 2D image data pairs that are strictly matched with the images, providing precisely matched inputs for subsequent feature extraction and fusion.

[0069] S250, 2D image and 3D point cloud input

[0070] The spatiotemporally aligned 2D image obtained in S200 is input to the vision processing branch; simultaneously, the spatiotemporally aligned 3D point cloud is input to the radar processing branch. From this point, the system enters a dual-modal data parallel processing flow.

[0071] S300, dual-branch parallel processing.

[0072] This step executes the visual processing branch and the radar processing branch in parallel, processing the aligned 2D image and 3D point cloud respectively, and simultaneously generating their respective feature maps and environmental assessment weight factors.

[0073] S301, Visual Processing Branch.

[0074] The visual processing branch simultaneously performs visual feature extraction and visual environment assessment (generation of visual weight factor α), and the specific process is as follows:

[0075] Visual Feature Extraction: The received 2D image is input into a pre-trained convolutional neural network for visual feature extraction. In one specific implementation, the convolutional neural network uses a backbone network based on the CSPDarkNet architecture. After the image passes through this network for forward propagation, a multi-scale deep visual feature map is extracted from its third layer, denoted as... .

[0076] Visual environment assessment (generation of visual weighting factor α): Calculate the information entropy of the target region in the two-dimensional image obtained in S200. Specifically, this includes:

[0077] Obtain the grayscale image of the target region.

[0078] Statistically analyze the grayscale histogram of the grayscale image to obtain each grayscale level. (Probability of occurrence is typically between 0 and 255) .

[0079] According to the formula The information entropy is calculated. .

[0080] Subsequently, based on the information entropy Visual weight factors are generated using the first mapping function. In one specific embodiment, a first threshold is set. With the second threshold When the calculated information entropy Higher than (Indicates excessive image noise) or below When the image contrast is too low or too dark, the quality of the two-dimensional image is determined to have degraded. In this case, the first mapping function is calculated using the following Sigmoid function form. :

[0081]

[0082] in, The baseline entropy value under ideal conditions. This is the variance parameter, which is used as an adjustable parameter to adjust the visual weight factor. Information entropy Deviation from baseline entropy value The sensitivity of the image when the image quality is normal. The value approaches 1; when the quality decreases, The value decays towards 0.

[0083] S302, Radar processing branch.

[0084] The radar processing branch simultaneously performs radar geometric feature extraction and radar environment assessment (radar weighting factor β generation), and the specific process is as follows:

[0085] Radar geometric feature extraction: The received 3D point cloud is voxelized, that is, the 3D space is divided into a regular 3D voxel grid, and the point cloud is assigned to the corresponding voxels to obtain voxelized point cloud data. Then, this voxelized point cloud data is input into a point cloud network for geometric feature extraction. In one specific implementation, the point cloud network uses PointNet++. After extracting the local and global geometric features of the point cloud, the network maps them to a deep visual feature map through spatial dimension transformation (such as upsampling or feature projection). A pseudo-feature map that is identical in both height and width is denoted as... .

[0086] Radar environment assessment (radar weighting factor β generation): Calculate clutter density based on the number of clutter points within the target area obtained from the 3D point cloud in S200. Specifically, this includes:

[0087] A preset radius is defined centered on the target location obtained from preliminary detection in the current frame or prediction of historical trajectories. The three-dimensional spherical spatial neighborhood.

[0088] Within this neighborhood, all reflection intensities below a preset intensity threshold are counted. The number of points, as the clutter point count. .

[0089] According to the formula The local clutter density was calculated. .

[0090] Subsequently, based on the clutter density Radar weighting factors are generated using the second mapping function. Specifically, it is calculated using the following formula:

[0091]

[0092] in, The preset upper limit for clutter density tolerance. This is the adjustment coefficient. When the clutter density... At lower levels (when the radar signal is clean). The value approaches 1; when the clutter density Exceed (Severe interference) The value decays towards 0.

[0093] S400, weighted fusion of adaptive fusion module.

[0094] The visual weight factor generated based on S300 and radar weighting factor The deep visual feature map extracted from S300 With pseudo-feature maps The input is weighted and fused by the adaptive fusion module to obtain the fused feature map. .

[0095] The specific process is carried out in three steps:

[0096] Visual feature weighting: based on the aforementioned visual weight factors For the visual feature map Perform weighted processing. Specifically, [the weighted processing will be performed]. Each feature element in the and Multiply to obtain a weighted visual feature map. .

[0097] Radar geometric feature weighting: based on the radar weighting factor For the pseudo feature map Perform weighted processing. Specifically, [the weighted processing will be performed]. Each feature element in the and Multiplying them together yields a weighted pseudo-feature map. .

[0098] Feature map addition and fusion: The weighted visual feature maps are added together. With the weighted pseudo-feature map The adaptive fusion feature map is obtained by adding elements one by one. This process can be expressed as the following weighted summation formula:

[0099]

[0100] in, and All are normalized values ​​between 0 and 1. The higher the value, the higher the reliability of the current two-dimensional image; A higher value indicates a higher level of confidence in the current 3D point cloud. This formula ensures that when the quality of data from any sensor deteriorates, its contribution to the fusion result automatically decreases, thus avoiding noise contamination.

[0101] S500, target detection network detection.

[0102] The fused feature map obtained in S400 Channel integration and dimensionality reduction are performed through a 1×1 convolutional layer to further optimize feature representation and adjust the number of channels.

[0103] Subsequently, the feature map processed by the 1×1 convolutional layer is input into the object detection network (such as the YOLO series detection head based on anchor boxes). The object detection network performs category classification and bounding box regression, and outputs initial detection results including the probability of the UAV target category, the coordinates of the UAV bounding box, and the preliminary confidence level.

[0104] Next, non-maximum suppression processing is performed on the initial detection results to remove highly overlapping redundant detection boxes, resulting in the deduplicated final detection boxes of the UAV.

[0105] S600, Output Category and Location

[0106] Finally, the coordinates of these UAV final detection boxes are mapped from the feature map scale output by the network back to the pixel coordinate system of the original two-dimensional visible light image described in S200, thereby outputting the final detection result containing the UAV target category and precise pixel-level bounding box coordinates for display or further processing.

[0107] Example 2

[0108] Based on the multimodal detection information fusion method based on environmental assessment provided in Embodiment 1 of this application, Embodiment 2 further details the model parameter optimization process involved therein, that is, how to determine the parameters in the convolutional neural network, point cloud network, first mapping function and second mapping function through training so that the system can obtain adaptive fusion capability.

[0109] The optimization of these parameters is achieved through an end-to-end training process, the core of which lies in minimizing a carefully designed total loss function.

[0110] Training data preparation

[0111] Millimeter-wave radar point cloud data and visible light camera image data were simultaneously acquired under various typical environmental scenarios (such as sunny days, cloudy days, nighttime, foggy days, and complex urban backgrounds) to form a training dataset. For each set of data in the dataset, calibration and spatiotemporal alignment processing were performed as described in S100 and S200 of Example 1. Simultaneously, professional personnel labeled the UAV targets in the images, with the labeling information including at least the target category and its precise bounding box coordinates in the image, serving as the "Ground Truth" for model learning.

[0112] Trainable network construction

[0113] Construct a trainable network model that fully corresponds to the detection process described in Example 1. This model includes:

[0114] Visual feature extraction branch: It consists of a convolutional neural network backbone based on the CSPDarkNet architecture.

[0115] Radar geometric feature extraction branch: includes voxelization layer and PointNet++ network.

[0116] Environmental assessment module: Includes a preprocessing unit for calculating information entropy and clutter density, as well as embedded trainable parameters. , , , The first and second mapping functions are used to generate visual weight factors. and radar weighting factor .

[0117] Fusion and detection module: includes a weighted fusion layer, a 1×1 convolutional layer, and an object detection network (such as the YOLO detection head).

[0118] All trainable parameters of the network (including the weights of each network, the adjustment coefficients in the environment assessment module, and the baseline values) are randomly initialized.

[0119] Loss function design

[0120] Total loss function It consists of three parts that collectively guide the learning direction of the network:

[0121]

[0122] Classification loss Cross-entropy loss is typically used to measure the difference between the target category predicted by the network and the true category.

[0123] Balance coefficient Its function is to adjust the weight consistency loss. Its importance in the overall optimization objective. By adjusting... The size of this value can control the model's emphasis between "optimizing detection accuracy" and "learning reasonable weight allocation." In actual training, The value of is usually determined through grid search or cross-validation on a validation set to ensure that the model can generate dynamic weight factors that conform to the logic of environment perception while maintaining high detection performance. and .

[0124] Location loss : This measures the difference between the location and size of the bounding box predicted by the network and the true bounding box, and is usually achieved using the SmoothL1 loss or CIoU loss function.

[0125] Weight Consistency Loss This is a key innovative part of the training process, and its purpose is to enable the network to generate weight factors. and It can accurately and reasonably reflect the quality of the corresponding sensor data. This loss function encourages consistency: when image quality is low (e.g., due to abnormal information entropy caused by haze), the visual weight factor... The radar weighting factor should be minimized when clutter is dense in the radar point cloud. It should be smaller.

[0126] Training and optimization process

[0127] The prepared training data is input into the constructed trainable network for forward propagation to obtain the predicted detection results and weight factors. and .

[0128] Calculate the classification loss between the predicted results and the true labels. and positioning loss .

[0129] Calculate the weight consistency loss by combining environmental labels. .

[0130] Based on total losses It utilizes the backpropagation algorithm and gradient descent optimizer (such as Adam) to simultaneously update all trainable parameters in the entire network (including feature extraction network, environment evaluation parameters, fusion and detection network parameters).

[0131] Iterate through the above process until the total loss converges, and the network model achieves stable optimal performance on the reserved validation set. After training, save the fixed network model and its parameters for use in the real-time inference described in Example 1.

[0132] Through the above training process, this method not only learned how to detect drones, but more importantly, it learned how to dynamically evaluate sensor reliability based on the real-time environment (manifested as generating reasonable...). and This achieves true adaptive fusion and significantly improves the system's robustness in complex and ever-changing environments.

[0133] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0134] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0135] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal detection information fusion method based on environmental assessment, characterized in that, include: Simultaneously acquire radar point clouds and visible light images, and map the radar coordinate system corresponding to the radar point cloud to the image pixel coordinate system corresponding to the visible light image to obtain spatiotemporally aligned three-dimensional point clouds and two-dimensional images; The spatiotemporally aligned 2D image is input into the visual processing branch, and the spatiotemporally aligned 3D point cloud is simultaneously input into the radar processing branch. The visual processing branch's processing flow is as follows: The two-dimensional image is input into a convolutional neural network for visual feature extraction to obtain a deep visual feature map; simultaneously, the information entropy of the target region in the two-dimensional image is calculated, and a visual weight factor is generated based on the information entropy through a first mapping function. ; The radar processing branch process is as follows: After voxelizing the 3D point cloud, it is input into the point cloud network for radar geometric feature extraction to obtain a pseudo-feature map with the same size as the deep visual feature map; simultaneously, the clutter density is calculated based on the number of clutter points in the target area of ​​the 3D point cloud, and a radar weighting factor is generated based on the clutter density through a second mapping function. ; Based on the visual weight factor and radar weighting factor The deep visual feature map and the pseudo feature map are weighted and fused to obtain a fused feature map; The fused feature map is integrated through a 1×1 convolutional layer and then input into the target detection network. The final output includes at least the detection result containing the drone target category and the drone target bounding box coordinates. The clutter density is calculated based on the number of clutter points within the target area of ​​the 3D point cloud, and a radar weighting factor is generated based on the clutter density through a second mapping function. ,include: Centered on the current detection location, within a three-dimensional spatial neighborhood of a preset radius, count the number of points with reflection intensity lower than a preset intensity threshold, and use this count as the number of clutter points. The clutter density is calculated based on the number of clutter points and the volume of the three-dimensional spatial neighborhood. Based on the clutter density, a radar weighting factor is generated using the second mapping function. Specifically: in, clutter density, This represents the upper limit of clutter density tolerance. This is the adjustment coefficient; It also includes optimizing the parameters in the convolutional neural network, point cloud network, first mapping function, and second mapping function by minimizing the total loss function, wherein the total loss function includes classification loss, localization loss, and a term related to the visual weight factor. and radar weighting factor Related weight consistency loss.

2. The multimodal detection information fusion method based on environmental assessment according to claim 1, characterized in that, The process of simultaneously acquiring radar point clouds and visible light images, and mapping the radar coordinate system corresponding to the radar point cloud to the image pixel coordinate system corresponding to the visible light image, to obtain spatiotemporally aligned 3D point clouds and 2D images includes: Multiple sets of radar calibration point clouds and calibration visible light images are collected for a calibration board. The coordinate transformation matrix is ​​calculated based on the radar calibration point clouds and the calibration visible light images. Simultaneously acquire radar point cloud and visible light image, and project the radar point cloud onto the image pixel coordinate system corresponding to the visible light image based on the coordinate transformation matrix to obtain spatiotemporally aligned 3D point cloud and 2D image.

3. The multimodal detection information fusion method based on environmental assessment according to claim 1, characterized in that, The calculation of the information entropy of the target region in the two-dimensional image includes: Obtain the grayscale image of the target region in the two-dimensional image; Statistically analyze the grayscale histogram of the grayscale image to obtain each grayscale level. probability of occurrence ; Based on the probability of occurrence Through formula The information entropy is calculated. .

4. The multimodal detection information fusion method based on environmental assessment according to claim 3, characterized in that, Also includes: Set a first threshold and a second threshold, when the information entropy When the image quality is above the first threshold or below the second threshold, the two-dimensional image quality is determined to have degraded, and a visual weight factor is generated using the first mapping function. Specifically: in, The baseline entropy value under ideal conditions. This is the variance parameter, which is used as an adjustable parameter to adjust the visual weight factor. Information entropy Deviation from baseline entropy value Sensitivity.

5. The multimodal detection information fusion method based on environmental assessment according to claim 1, characterized in that, The visual weight factor and radar weighting factor The deep visual feature map and the pseudo-feature map are weighted and fused to obtain a fused feature map, including: Based on the visual weight factor The deep visual feature map is weighted to obtain a weighted visual feature map; simultaneously, based on the radar weighting factor... The pseudo-feature map is weighted to obtain a weighted pseudo-feature map; The weighted visual feature map and the weighted pseudo-feature map are added element-wise to obtain the fused feature map. The specific process is represented by the following weighted summation formula: in, This is a deep visual feature map. It is a pseudo feature map. To fuse feature maps.

6. The multimodal detection information fusion method based on environmental assessment according to claim 1, characterized in that, The visual weight factor and the radar weighting factor All are normalized values ​​between 0 and 1; wherein, the visual weight factor The higher the value, the higher the confidence level of the current two-dimensional image; the radar weighting factor. The higher the value, the higher the confidence level of the current 3D point cloud.

7. The multimodal detection information fusion method based on environmental assessment according to claim 1, characterized in that, The process of inputting the two-dimensional image into a convolutional neural network for visual feature extraction to obtain a deep visual feature map, and inputting the three-dimensional point cloud into a point cloud network after voxelization for radar geometric feature extraction to obtain a pseudo-feature map with the same size as the deep visual feature map, includes: The two-dimensional image is input into the backbone of a convolutional neural network based on the CSPDarkNet architecture; Multi-scale visual feature maps are extracted from the third layer of the convolutional neural network backbone as the deep visual feature maps; Simultaneously, the three-dimensional point cloud is divided into a regular voxel grid to obtain voxelized point cloud data; The voxelized point cloud data is input into the point cloud network PointNet++ to extract radar geometric features; The extracted radar geometric features are transformed in space dimension and mapped to a pseudo-feature map with the same size as the deep visual feature map.

8. The multimodal detection information fusion method based on environmental assessment according to claim 1, characterized in that, The fused feature map is integrated through a 1×1 convolutional layer and then input into the target detection network. The final output includes detection results containing at least the UAV target category and the UAV target bounding box coordinates, including: The fused feature map is channel-integrated through a 1×1 convolutional layer; The processed fused feature map is input into the target detection network for category classification and bounding box regression to obtain the initial detection result; non-maximum suppression is applied to the initial detection result to obtain the deduplicated final detection box; the coordinates of the final detection box are mapped back to the image pixel coordinate system of the visible light image, and the final output is the detection result containing the UAV target category and the UAV bounding box coordinates.

Images