A severe weather fusion perception method and system based on millimeter wave radar

Abstract

The application provides a kind of based on millimeter wave radar's severe weather fusion perception method and system, the method includes: generating radar point cloud feature set and image feature set, and from two kinds of original data jointly extracted to get the weather disturbance feature vector of current scene;Based on modal degradation weight, the two kinds of feature sets are associated mapping processing to generate corresponding modal associated feature map;According to modal associated feature map, the corresponding layered fusion perception framework is constructed, and the layered fusion perception framework includes multi-level fusion level and the compensation rule of corresponding level;Based on layered fusion perception framework, modal compensation and layered fusion processing are carried out to two kinds of original data respectively, and the corresponding initial perception result is generated, and the matching parameters of degradation mechanism matching library are updated reversely according to the confidence bias of initial perception result.The application can greatly improve the fusion perception efficiency under severe weather.

Description

Technical Field

[0001] This invention relates to the field of radar technology, and in particular to a method and system for fusion perception of severe weather based on millimeter-wave radar. Background Technology

[0002] As autonomous driving technology evolves to higher levels, the all-weather robustness of environmental perception systems has become a core technological bottleneck. In adverse weather conditions such as rain, fog, snow, and sandstorms, camera performance plummets due to obstructed vision and reduced image contrast. LiDAR suffers from beam scattering, leading to point cloud sparsity and a significant reduction in effective detection range. Millimeter-wave radar, with its strong penetrating power at millimeter wavelengths, maintains stable ranging and speed measurement capabilities even in adverse weather, making it an indispensable key modality in multi-sensor fusion perception systems. How to fully leverage the all-weather advantages of millimeter-wave radar and achieve efficient fusion with other modalities such as vision is a core issue in improving the reliability of autonomous driving perception in adverse weather conditions.

[0003] Currently, multi-sensor fusion solutions for severe weather mainly fall into two categories: fixed-weight fusion and adaptive-weighted fusion. Fixed-weight fusion pre-sets the fusion ratio between radar and vision, uniformly increasing the radar weight under severe weather conditions. While simple to implement, this method fails to differentiate the impact of different weather types on each sensor, making it unsuitable for adapting to the varying degradation patterns of different modes in scenarios such as rain, dense fog, and snowfall, thus limiting its performance improvement potential. Adaptive-weighted fusion dynamically adjusts weights based on image clarity or radar signal-to-noise ratio. While responsive to environmental changes, it only assesses the "quality" of a single mode, failing to identify specific weather types and degradation mechanisms. This results in a lack of specificity in the fusion strategy, making it difficult to cope with complex and ever-changing real-world weather scenarios.

[0004] Furthermore, existing adaptive fusion solutions for weather or air quality assessment largely rely on camera image features or external meteorological sensors. The former degrades under severe weather conditions, leading to distorted assessment benchmarks, while the latter increases system hardware costs and deployment complexity, making it difficult to scale up for mass production in automotive scenarios. Meanwhile, millimeter-wave radar echo signals themselves contain rich meteorological features, such as the Doppler distribution of raindrop clutter, frequency-selective attenuation of fog, and non-uniform scattering characteristics of snow. However, current technologies have not fully explored the weather perception potential of radar signals themselves, nor have they established an end-to-end closed-loop mechanism for weather feature extraction, degradation mechanism matching, and differentiated fusion compensation. As a result, the fusion perception performance under severe weather conditions is consistently constrained by the dual bottlenecks of inaccurate modal assessment and insufficient strategy adaptation. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a method and system for fusion perception of severe weather based on millimeter-wave radar, in order to solve the problem that the existing technology has not yet established an end-to-end closed-loop mechanism among weather feature extraction, degradation mechanism matching, and differentiated fusion compensation, which leads to the fusion perception performance under severe weather being always constrained by inaccurate modal evaluation and insufficient strategy adaptation.

[0006] The first aspect of the present invention proposes: A fusion perception method for severe weather based on millimeter-wave radar, wherein the method includes: The original point cloud data from the millimeter-wave radar and the original image data from the visible light camera were preprocessed separately to generate radar point cloud feature sets and image feature sets, and the weather disturbance feature vector of the current scene was jointly extracted from the two types of original data. The modal degradation type corresponding to the weather disturbance feature vector is matched by a preset degradation mechanism matching library, and the modal degradation weights corresponding to the radar point cloud feature set and the image feature set are calculated based on the modal degradation type. The two feature sets are then associated and mapped based on the modal degradation weights to generate the corresponding modal association feature map. A corresponding hierarchical fusion perception framework is constructed based on the modal association feature map. The hierarchical fusion perception framework includes multiple fusion levels and corresponding compensation rules. Based on the hierarchical fusion perception framework, modal compensation and hierarchical fusion processing are performed on the two types of raw data respectively to generate corresponding initial perception results, and the matching parameters of the degradation mechanism matching library are updated in reverse according to the confidence deviation of the initial perception results.

[0007] The beneficial effects of this invention are as follows: This solution preprocesses the raw point cloud data from millimeter-wave radar and the raw image data from visible light cameras respectively, and jointly extracts weather disturbance feature vectors, which can accurately quantify the degree of weather interference in the current scene; relying on a preset degradation mechanism matching library to match the corresponding modal degradation type and calculate the degradation weights of the two types of features, and combining feature association mapping to generate a modal association feature map, it can achieve accurate quantitative assessment of the degree of modal degradation, effectively solving the problem of inaccurate modal assessment in traditional solutions; based on the modal association feature map, a hierarchical fusion perception framework containing multi-level fusion layers and corresponding compensation rules is constructed, which can perform differentiated fusion compensation for different degradation scenarios, making up for the defects of insufficient strategy adaptation; finally, the matching parameters of the matching library are updated in reverse by the confidence deviation of the initial perception results, establishing an end-to-end closed-loop mechanism for weather feature extraction, degradation mechanism matching, and differentiated fusion compensation, which can continuously iterate and optimize the fusion logic, significantly improving the accuracy and environmental robustness of multimodal fusion perception under severe weather conditions.

[0008] Furthermore, the step of matching the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, and calculating the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type includes: Multiple weather feature parameters contained in the weather disturbance feature vector are extracted, and the coupling relationship between the multiple weather feature parameters is identified by a graph neural network. A corresponding weather coupling feature map is generated based on the coupling relationship. The radar point cloud feature set and the image feature set are respectively standardized and encoded to generate radar feature sequences and image feature sequences with the same corresponding dimensions. The two types of feature sequences are then mapped to the feature space corresponding to the weather coupled feature map through a feature mapping algorithm. The radar degradation weight value corresponding to the radar feature sequence and the weather coupled feature map, and the image degradation weight value corresponding to the image feature sequence and the weather coupled feature map are calculated respectively by a preset weight calculation function, and the two types of degradation weight values ​​are output as the modal degradation weight.

[0009] Furthermore, the step of performing association mapping processing on the two types of feature sets based on the modality degradation weights to generate corresponding modality association feature maps includes: The common spatial feature fields between the radar point cloud feature set and the image feature set are extracted in real time, and the association confidence of each common spatial feature field is calculated in real time based on the modal degradation weight. For shared spatial feature fields with association confidence scores higher than a preset threshold, the radar point cloud feature set and the corresponding feature vectors in the image feature set are fused in spatial dimension to generate corresponding fused feature units. For shared spatial feature fields with association confidence scores below a preset threshold, a coordinate mapping link between the two types of features is established using a spatial projection algorithm. Using the fusion feature unit as the core node and the coordinate mapping link as the connecting edge, the modal association feature map is formed accordingly.

[0010] Furthermore, the step of constructing the corresponding hierarchical fusion perception framework based on the modality association feature map includes: The node features in the modal association feature graph are labeled with types to distinguish spatial location nodes, semantic attribute nodes, motion state nodes and weather disturbance nodes, and the number of each type of node and the density of associated edges are counted. Based on the edge density of the spatial location nodes and the motion state nodes, the corresponding fusion level sorting is determined, and the key parameter information contained in the semantic attribute nodes and the weather disturbance nodes is extracted. Several levels of compensation sub-rules are then divided according to the degree of influence of the key parameter information. Based on the hierarchical relationship between the fusion hierarchy sorting and several hierarchical compensation sub-rules, a corresponding fusion framework structure tree is constructed, and the node ID of the modality association feature map is marked at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework.

[0011] Furthermore, the step of marking the node ID of the modality association feature map at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework includes: Each branch node of the fusion framework structure tree is hierarchically and deeply encoded to form a three-dimensional positioning code containing the root node identifier, the parent node identifier, and the node position. The node IDs in the modal association feature map are grouped and clustered according to node type, and the mapping relationship between each node ID and the three-dimensional positioning code is constructed in real time. According to the mapping relationship, the corresponding node ID is written into the attribute field of each branch node in the fusion framework structure tree, and the number of associated nodes and weight value of the node in the modal association feature map are marked on each node ID to generate the hierarchical fusion perception framework.

[0012] Furthermore, the step of performing modal compensation and hierarchical fusion processing on the two types of raw data based on the hierarchical fusion sensing framework to generate corresponding initial sensing results includes: The original point cloud data from the millimeter-wave radar is divided into spatial grids to generate a point cloud structure index; the original image data from the visible light camera is segmented into multi-scale semantic segments to generate an image feature label set. The point cloud structure index is matched with the spatial location class nodes in the hierarchical fusion perception framework to determine the compensation weight value corresponding to each grid; the image feature label set is compared with the semantic attribute class nodes to retain feature labels with matching degree exceeding the preset matching threshold. Clutter suppression and effective point enhancement are performed on the point cloud raster with high compensation weight values ​​to generate corresponding compensated point cloud data; contrast enhancement and detail restoration are performed on the image region corresponding to the feature label to generate corresponding compensated image data; and multi-level fusion processing is performed on the compensated point cloud data and the compensated image data to generate the initial perception result.

[0013] Furthermore, the step of performing multi-level fusion processing on the compensated point cloud data and the compensated image data to generate the initial perception result includes: Add corresponding spatial coordinate meta-labels to the compensated point cloud data, and add appropriate semantic attribute tags to the compensated image data; A corresponding fusion index table is created in real time based on the spatial coordinate meta-labels and the semantic attribute tags, and a corresponding spatial correlation matrix is ​​generated based on the compensated point cloud data and the compensated image data. According to the fusion index table, the corresponding target perception features are extracted in real time from the spatial correlation matrix, and the target perception features are input into a preset detection head for decoding processing to generate the initial perception result.

[0014] The second aspect of the present invention proposes: A severe weather fusion sensing system based on millimeter-wave radar, wherein the system includes: The processing module is used to preprocess the synchronously acquired millimeter-wave radar raw point cloud data and visible light camera raw image data respectively, generate radar point cloud feature sets and image feature sets respectively, and jointly extract the weather disturbance feature vector of the current scene from the two types of raw data. The matching module is used to match the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, calculate the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type, and perform association mapping processing on the two feature sets based on the modal degradation weights to generate the corresponding modal association feature map. The construction module is used to construct a corresponding hierarchical fusion perception framework based on the modality association feature map. The hierarchical fusion perception framework includes multiple fusion levels and corresponding compensation rules. The verification module is used to perform modal compensation and hierarchical fusion processing on the two types of raw data based on the hierarchical fusion perception framework, generate corresponding initial perception results, and update the matching parameters of the degradation mechanism matching library in reverse according to the confidence deviation of the initial perception results.

[0015] Furthermore, the step of matching the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, and calculating the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type includes: Multiple weather feature parameters contained in the weather disturbance feature vector are extracted, and the coupling relationship between the multiple weather feature parameters is identified by a graph neural network. A corresponding weather coupling feature map is generated based on the coupling relationship. The radar point cloud feature set and the image feature set are respectively standardized and encoded to generate radar feature sequences and image feature sequences with the same corresponding dimensions. The two types of feature sequences are then mapped to the feature space corresponding to the weather coupled feature map through a feature mapping algorithm. The radar degradation weight value corresponding to the radar feature sequence and the weather coupled feature map, and the image degradation weight value corresponding to the image feature sequence and the weather coupled feature map are calculated respectively by a preset weight calculation function, and the two types of degradation weight values ​​are output as the modal degradation weight.

[0016] Furthermore, the step of performing association mapping processing on the two types of feature sets based on the modality degradation weights to generate corresponding modality association feature maps includes: The common spatial feature fields between the radar point cloud feature set and the image feature set are extracted in real time, and the association confidence of each common spatial feature field is calculated in real time based on the modal degradation weight. For shared spatial feature fields with association confidence scores higher than a preset threshold, the radar point cloud feature set and the corresponding feature vectors in the image feature set are fused in spatial dimension to generate corresponding fused feature units. For shared spatial feature fields with association confidence scores below a preset threshold, a coordinate mapping link between the two types of features is established using a spatial projection algorithm. Using the fused feature unit as the core node and the coordinate mapping link as the connecting edge, the modal association feature map is formed accordingly.

[0017] Furthermore, the step of constructing the corresponding hierarchical fusion perception framework based on the modality association feature map includes: The node features in the modal association feature graph are labeled with types to distinguish spatial location nodes, semantic attribute nodes, motion state nodes and weather disturbance nodes, and the number of each type of node and the density of associated edges are counted. Based on the edge density of the spatial location nodes and the motion state nodes, the corresponding fusion level sorting is determined, and the key parameter information contained in the semantic attribute nodes and the weather disturbance nodes is extracted. Several levels of compensation sub-rules are then divided according to the degree of influence of the key parameter information. Based on the hierarchical relationship between the fusion hierarchy sorting and several hierarchical compensation sub-rules, a corresponding fusion framework structure tree is constructed, and the node ID of the modality association feature map is marked at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework.

[0018] Furthermore, the step of marking the node ID of the modality association feature map at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework includes: Each branch node of the fusion framework structure tree is hierarchically and deeply encoded to form a three-dimensional positioning code containing the root node identifier, the parent node identifier, and the node position. The node IDs in the modal association feature map are grouped and clustered according to node type, and the mapping relationship between each node ID and the three-dimensional positioning code is constructed in real time. According to the mapping relationship, the corresponding node ID is written into the attribute field of each branch node in the fusion framework structure tree, and the number of associated nodes and weight value of the node in the modal association feature map are marked on each node ID to generate the hierarchical fusion perception framework.

[0019] Furthermore, the step of performing modal compensation and hierarchical fusion processing on the two types of raw data based on the hierarchical fusion sensing framework to generate corresponding initial sensing results includes: The original point cloud data from the millimeter-wave radar is divided into spatial grids to generate a point cloud structure index; the original image data from the visible light camera is segmented into multi-scale semantic segments to generate an image feature label set. The point cloud structure index is matched with the spatial location class nodes in the hierarchical fusion perception framework to determine the compensation weight value corresponding to each grid; the image feature label set is compared with the semantic attribute class nodes to retain feature labels with matching degree exceeding the preset matching threshold. Clutter suppression and effective point enhancement are performed on the point cloud raster with high compensation weight values ​​to generate corresponding compensated point cloud data; contrast enhancement and detail restoration are performed on the image region corresponding to the feature label to generate corresponding compensated image data; and multi-level fusion processing is performed on the compensated point cloud data and the compensated image data to generate the initial perception result.

[0020] Furthermore, the step of performing multi-level fusion processing on the compensated point cloud data and the compensated image data to generate the initial perception result includes: Add corresponding spatial coordinate meta-labels to the compensated point cloud data, and add appropriate semantic attribute tags to the compensated image data; A corresponding fusion index table is created in real time based on the spatial coordinate meta-labels and the semantic attribute tags, and a corresponding spatial correlation matrix is ​​generated based on the compensated point cloud data and the compensated image data. According to the fusion index table, the corresponding target perception features are extracted in real time from the spatial correlation matrix, and the target perception features are input into a preset detection head for decoding processing to generate the initial perception result.

[0021] The third aspect of the present invention proposes: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the severe weather fusion perception method based on millimeter-wave radar as described above.

[0022] The fourth aspect of the present invention proposes: A readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the severe weather fusion sensing method based on millimeter-wave radar as described above.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] Figure 1 A flowchart illustrating the severe weather fusion perception method based on millimeter-wave radar provided in the first embodiment of the present invention; Figure 2 This is a structural block diagram of a severe weather fusion sensing system based on millimeter-wave radar provided in the third embodiment of the present invention.

[0025] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0026] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0027] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0029] Please see Figure 1 The image shows a severe weather fusion perception method based on millimeter-wave radar provided in the first embodiment of the present invention. The severe weather fusion perception method based on millimeter-wave radar provided in this embodiment can establish an end-to-end closed-loop mechanism for weather feature extraction, degradation mechanism matching, and differentiated fusion compensation. It can continuously iterate and optimize the fusion logic, and significantly improve the accuracy and environmental robustness of multimodal fusion perception under severe weather conditions.

[0030] Specifically, this embodiment provides: A fusion perception method for severe weather based on millimeter-wave radar, wherein the method includes: Step S10: Preprocess the original point cloud data of the millimeter-wave radar and the original image data of the visible light camera that were acquired simultaneously to generate radar point cloud feature set and image feature set respectively, and extract the weather disturbance feature vector of the current scene from the two types of original data. It should be noted that targeted preprocessing was performed on the millimeter-wave radar point cloud data and the visible light camera image data after time synchronization and spatial joint calibration. The radar point cloud data underwent direct-pass filtering to remove road surface noise, statistical filtering to remove outliers, and coordinate transformation to unify to the world coordinate system, extracting a structured radar point cloud feature set containing position, intensity, and velocity information. The visible light image data underwent lens distortion correction, white balance calibration, and grayscale normalization, and multi-dimensional image feature sets such as edge, texture, and semantic features were extracted through multi-scale convolution. At the same time, a 12-dimensional weather disturbance feature vector was jointly extracted from the two types of raw data. The radar side included four terms: point cloud clutter rate, effective target point ratio, average reflection intensity attenuation rate, and distance measurement error standard deviation. The image side included five terms: global contrast, atmospheric light intensity estimate, edge retention rate, depth of field attenuation coefficient, and color saturation. The visibility estimate, meteorological particle equivalent diameter, and disturbance uniformity were jointly derived. The two types of features complement each other to form a complete weather disturbance characterization. The joint extraction method avoids the systematic bias of judging weather by a single mode. For example, in dense fog, image features degrade significantly but radar features change only slightly. Relying solely on images will overestimate the overall degradation level. Dual-mode joint extraction can more accurately characterize the true level of disturbance.

[0031] Step S20: Match the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, calculate the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type, and perform correlation mapping processing on the two feature sets based on the modal degradation weights to generate the corresponding modal correlation feature map. It should be noted that, through a pre-defined degradation mechanism matching library, the modal degradation type corresponding to the weather disturbance feature vector is matched to determine whether the current weather is fog, moderate rain, blizzard, dust storm, or a mixed type, and the corresponding electromagnetic wave and optical degradation mechanisms. Then, based on the degradation type, the modal degradation weights corresponding to the radar point cloud feature set and the image feature set are calculated separately to quantify the reliability and degradation magnitude of the two types of sensors under the current weather conditions. Subsequently, based on the modal degradation weights, spatial correlation mapping is performed on the two feature sets to distinguish between high-reliability, strongly correlated regions and low-reliability, weakly correlated regions, generating a structured modal correlation feature map. This step overcomes the limitations of traditional fixed-weight fusion. For example, in foggy weather where image degradation is severe but radar is relatively reliable, the fusion weight of radar features is automatically increased while the image weight is decreased, preventing severely degraded modes from lowering the overall perception accuracy.

[0032] Step S30: Construct a corresponding hierarchical fusion perception framework based on the modal association feature map. The hierarchical fusion perception framework includes multiple fusion levels and corresponding compensation rules. It should be noted that a hierarchical fusion perception framework is built based on the generated modal association feature maps. The framework includes three levels of fusion and corresponding compensation rules for each level. Different levels correspond to different types of feature fusion and different intensities of degradation compensation. The hierarchical framework follows the logic of "bottom-level features supplementing space, middle-level features supplementing semantics, and high-level features supplementing decision-making," and integrates four types of features—spatial location, motion state, semantic attributes, and weather disturbances—in an orderly manner according to the hierarchy. This avoids noise diffusion caused by the mixing of different types of features and allows the compensation strategy to accurately match the degradation features of the corresponding level. For example, the spatial layer focuses on radar clutter suppression, the semantic layer focuses on image dehazing and restoration, and the decision layer focuses on confidence calibration.

[0033] Step S40: Based on the hierarchical fusion perception framework, modal compensation and hierarchical fusion processing are performed on the two types of raw data respectively to generate corresponding initial perception results, and the matching parameters of the degradation mechanism matching library are updated in reverse according to the confidence deviation of the initial perception results.

[0034] It should be noted that, based on the hierarchical fusion perception framework, targeted modal compensation processing is performed on both radar and image raw data, and then multi-level feature fusion is completed in a hierarchical order to output initial perception results such as target detection, classification, and tracking. Simultaneously, based on the confidence deviation of the initial perception results, the matching parameters of the degradation mechanism matching library are updated in reverse, correcting the correspondence between different weather features and degradation types and the weight calculation coefficients. For example, in a blizzard, if the initial weight calculation deviates significantly from the actual perception, the system automatically adjusts the degradation weight coefficients for the snow scene, allowing the matching library to continuously optimize with data accumulation, forming a closed-loop iterative mechanism of "perception—verification—optimization," which leads to a sustained improvement in perception accuracy over long-term use.

[0035] Second Embodiment Furthermore, the step of matching the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, and calculating the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type includes: Multiple weather feature parameters contained in the weather disturbance feature vector are extracted, and the coupling relationship between the multiple weather feature parameters is identified by a graph neural network. A corresponding weather coupling feature map is generated based on the coupling relationship. The radar point cloud feature set and the image feature set are respectively standardized and encoded to generate radar feature sequences and image feature sequences with the same corresponding dimensions. The two types of feature sequences are then mapped to the feature space corresponding to the weather coupled feature map through a feature mapping algorithm. The radar degradation weight value corresponding to the radar feature sequence and the weather coupled feature map, and the image degradation weight value corresponding to the image feature sequence and the weather coupled feature map are calculated respectively by a preset weight calculation function, and the two types of degradation weight values ​​are output as the modal degradation weight.

[0036] It should be noted that 12 weather feature parameters are extracted from the weather disturbance feature vector, including raindrop clutter density, fog attenuation coefficient, snow particle reflection intensity, image atmospheric light value, edge degradation rate, and visibility estimate. A graph neural network is then used to identify the coupling relationships between these weather feature parameters. Under severe weather conditions, various disturbance factors do not act independently. For example, heavy rain accompanied by low temperatures easily creates a mixed rain and fog environment. Radar raindrop scattering and image fog attenuation can produce a coupling amplification effect, and simple linear superposition will underestimate the overall degradation degree. The graph neural network can capture the nonlinear coupling relationships between features, output the coupling gain coefficient under different feature combinations, and finally generate the corresponding weather coupling feature map, comprehensively representing the comprehensive intensity and mode of action of weather disturbances in the current scene. Taking foggy weather as an example, the coupling feature map shows a coupling mode of "image degradation as the main factor and radar degradation as the minor factor". The atmospheric light intensity, edge degradation features and visibility features on the image side have a high coupling degree, with a coupling gain coefficient of 1.3. On the radar side, only the clutter density increases slightly, with a coupling gain coefficient of only 1.05. This accurately reflects the physical law that fog has a great impact on optical imaging but a small impact on millimeter waves.

[0037] The radar point cloud feature set and image feature set are standardized and encoded separately. A shared feature embedding network maps both types of heterogeneous features into 256-dimensional feature vectors, generating radar feature sequences and image feature sequences with consistent dimensions, eliminating dimensional and scalar differences between the two types of features. Then, a feature mapping algorithm maps the two standardized feature sequences to a unified feature space corresponding to the weather coupled feature map, placing radar features, image features, and weather coupled features under the same feature measurement system. This provides a comparable feature basis for subsequent calculation of degradation weights, avoiding weight calculation distortion caused by inconsistent feature spaces.

[0038] By using a preset weight calculation function, the radar degradation weight value corresponding to the radar feature sequence and the weather coupled feature map, and the image degradation weight value corresponding to the image feature sequence and the weather coupled feature map are calculated respectively. A higher weight value indicates that the mode is more severely affected by weather disturbances and the original feature reliability is lower. The calculation combines prior coefficients from the degradation mechanism library with the current feature coupling strength for a comprehensive output. Taking a typical scenario as an example, in a foggy scenario, the calculated image degradation weight is 0.78, representing a 78% decrease in image feature reliability and a significant loss of effective information; the radar degradation weight is 0.22, indicating that the radar is less affected by fog and the feature reliability remains at a high level. However, in a moderate to heavy rain scenario, the scattering of radar electromagnetic waves by raindrops is enhanced, and the radar degradation weight rises to 0.51, while the image degradation weight is 0.63. Both modes show significant degradation, and their weights are closer. The calculated radar degradation weight and image degradation weight together constitute the mode degradation weight, which is output to the subsequent association mapping stage as the core basis for calculating feature fusion confidence.

[0039] Furthermore, the step of performing association mapping processing on the two types of feature sets based on the modality degradation weights to generate corresponding modality association feature maps includes: The common spatial feature fields between the radar point cloud feature set and the image feature set are extracted in real time, and the association confidence of each common spatial feature field is calculated in real time based on the modal degradation weight. For shared spatial feature fields with association confidence scores higher than a preset threshold, the radar point cloud feature set and the corresponding feature vectors in the image feature set are fused in spatial dimension to generate corresponding fused feature units. For shared spatial feature fields with association confidence scores below a preset threshold, a coordinate mapping link between the two types of features is established using a spatial projection algorithm. Using the fusion feature unit as the core node and the coordinate mapping link as the connecting edge, the modal association feature map is formed accordingly.

[0040] It should be noted that the common spatial feature fields between the radar point cloud feature set and the image feature set are extracted in real time. Specifically, these include 12 common spatial features across three categories: vertices of the target's bounding rectangle, edge corners, and vertical projection contours. The association confidence of each common spatial feature field is calculated in real time based on modality degradation weights. The confidence comprehensively reflects the matching degree and credibility of the corresponding features between the two modalities. The lower the degradation degree of the two modalities and the higher the feature matching degree, the higher the association confidence. When a certain modality is severely degraded, feature distortion leads to a decrease in matching degree, and the association confidence decreases accordingly. The confidence value ranges from 0 to 1, providing a quantitative basis for subsequent hierarchical processing.

[0041] The preset association confidence threshold is 0.6. For shared spatial feature fields with a confidence level higher than the threshold, it indicates that the corresponding features of both modalities are reliable and have a high matching degree. Deep spatial fusion is then performed on the corresponding feature vectors in the radar point cloud feature set and the image feature set to generate a fused feature unit that combines radar spatial accuracy with image semantic details, serving as the core anchor point for subsequent fusion. For shared spatial feature fields with an association confidence level lower than the threshold, it indicates that at least one modality is severely degraded and the feature matching reliability is insufficient. In this case, feature fusion is not forced to avoid degraded features contaminating effective features. Instead, a coordinate mapping link between the two types of features is established through a spatial projection algorithm. That is, the correspondence between spatial coordinates is preserved through the homography matrix from radar to image, and lightweight fusion is performed at an appropriate level in the subsequent hierarchical framework. Taking a foggy scene as an example, for vehicle targets within 50 meters of the sensing node, the matching degree of the two modal features is high, and the association confidence can reach 0.82, which is higher than the threshold, generating a deep fusion feature unit; for targets more than 150 meters away, the image features are severely blurred, the feature matching degree drops significantly, and the association confidence is only 0.35, which is lower than the threshold. At this time, only the coordinate mapping relationship is retained, and the image features are not forcibly fused, effectively avoiding the blurry features from reducing the detection accuracy of targets at a great distance.

[0042] Using high-confidence fusion feature units as core nodes and low-confidence coordinate mapping links as connecting edges, a structured modal association feature graph is formed. This graph structure not only preserves the strong feature information of the core fusion nodes but also maintains the spatial association relationships of all features through edges. It clearly presents the association strength and fusion depth of different regions and supports fast indexing of corresponding features by node ID, providing a structured feature carrier for node matching in the subsequent hierarchical fusion framework.

[0043] Furthermore, the step of constructing the corresponding hierarchical fusion perception framework based on the modality association feature map includes: The node features in the modal association feature graph are labeled with types to distinguish spatial location nodes, semantic attribute nodes, motion state nodes and weather disturbance nodes, and the number of each type of node and the density of associated edges are counted. Based on the edge density of the spatial location nodes and the motion state nodes, the corresponding fusion level sorting is determined, and the key parameter information contained in the semantic attribute nodes and the weather disturbance nodes is extracted. Several levels of compensation sub-rules are then divided according to the degree of influence of the key parameter information. Based on the hierarchical relationship between the fusion hierarchy sorting and several hierarchical compensation sub-rules, a corresponding fusion framework structure tree is constructed, and the node ID of the modality association feature map is marked at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework.

[0044] It should be noted that all node features in the modal association feature map are categorized into four types: spatial location nodes, semantic attribute nodes, motion state nodes, and weather disturbance nodes. Spatial location nodes correspond to geometric features such as the target's 3D coordinates, size, and contour, accounting for 35% of the total nodes. Semantic attribute nodes correspond to semantic labels such as target category, vehicle type, and color, accounting for 28%. Motion state nodes correspond to dynamic parameters such as velocity, acceleration, and heading angle, accounting for 22%. Weather disturbance nodes correspond to meteorological parameters such as local fog concentration, raindrop flux, and snow particle density, accounting for 15%. The proportion of each type of node and the density of associated edges are simultaneously calculated. A higher density of associated edges indicates stronger coupling between that type of node and other nodes, making it more suitable for fusion at an earlier level. Statistics show that the associated edge density of spatial location and motion state nodes reaches 0.82, indicating the strongest coupling.

[0045] Based on the high correlation edge density of spatial location nodes and motion state nodes, these two types of nodes are preferentially fused at the bottom layer to form the first-level bottom-layer fusion layer, responsible for outputting accurate target spatial positioning and motion trajectory. Semantic attribute nodes have a medium correlation edge density and are fused at the middle layer to supplement the semantic classification information of the target. Weather disturbance nodes have the lowest correlation edge density and are fused at the top layer to be responsible for global degradation compensation scheduling. Thus, the three-level fusion hierarchy of "spatial motion layer - semantic attribute layer - disturbance decision layer" is determined. At the same time, key parameter information from semantic attribute nodes and weather disturbance nodes is extracted, and several hierarchical compensation sub-rules are divided according to the degree of influence of parameters on the perception results: the bottom layer corresponds to spatial domain clutter suppression and point cloud enhancement rules, the middle layer corresponds to semantic domain deblurring and detail restoration rules, and the top layer corresponds to decision-level confidence weighting rules. Each compensation sub-rule is matched one-to-one with the corresponding fusion layer.

[0046] Based on the hierarchical order of fusion and the subordinate relationships of compensation sub-rules at each level, a tree-like fusion framework structure tree is constructed: the root corresponds to the global fusion output, the first-level branches correspond to the three-level fusion hierarchy, and the second-level branches correspond to the compensation sub-rules and feature nodes at each level. At each branch node of the fusion framework structure tree, the node ID of the modality-associated feature map is marked, establishing a one-to-one correspondence between the framework tree nodes and the feature map nodes, ultimately generating a complete hierarchical fusion perception framework. The tree structure has a clear hierarchy and explicit subordinate relationships, supporting hierarchical scheduling of corresponding fusion and compensation operations, and also facilitating the optimization of strategies at a single level without affecting the overall framework.

[0047] Furthermore, the step of marking the node ID of the modality association feature map at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework includes: Each branch node of the fusion framework structure tree is hierarchically and deeply encoded to form a three-dimensional positioning code containing the root node identifier, the parent node identifier, and the node position. The node IDs in the modal association feature map are grouped and clustered according to node type, and the mapping relationship between each node ID and the three-dimensional positioning code is constructed in real time. According to the mapping relationship, the corresponding node ID is written into the attribute field of each branch node in the fusion framework structure tree, and the number of associated nodes and weight value of the node in the modal association feature map are marked on each node ID to generate the hierarchical fusion perception framework.

[0048] It's important to note that a unified hierarchical depth encoding is performed on all branch nodes of the fusion framework's tree structure, generating a 3D location code containing the root node identifier, parent node identifier, and node position. The encoding follows the format "root node-level number-node sequence number," for example, "R01-L01-N03" represents the 1st fusion framework, the 1st level (bottom spatial layer), and node number 3. The root node identifier distinguishes fusion frameworks for different scenarios and configurations, the parent node identifier clarifies the parent level to which the current node belongs, and the node position indicates the order of nodes at the same level. The 3D location code uniquely determines the hierarchical position and execution order of each node in the framework tree, supporting rapid location of all nodes at a specific level using the encoding prefix, eliminating the need to traverse the entire tree and significantly improving node indexing efficiency during fusion scheduling.

[0049] The node IDs in the modal association feature map are grouped and clustered according to node type: spatial location and motion state node IDs are clustered into bottom-level groups, semantic attribute node IDs into middle-level groups, and weather disturbance node IDs into top-level groups. A bidirectional mapping relationship between each node ID and the 3D positioning code is constructed in real time, using a dual hash table for storage. The forward table uses the feature node ID as the key and the 3D positioning code as the value, while the reverse table does the opposite. This bidirectional mapping achieves decoupling and linkage between feature data and the fusion framework. Updating feature nodes does not require adjusting the framework structure, and adjusting the fusion framework does not require modifying the feature data, thus improving the maintainability and iteration efficiency of the entire perception system.

[0050] Based on the mapping relationship, the corresponding feature node ID is written into the attribute field of each branch node in the fusion framework structure tree. Simultaneously, two core attributes are added next to each node ID: the number of associated nodes in the modal association feature map and the fusion weight value of the node. For example, a spatial node at the bottom layer labeled "Number of associated nodes: 6, Fusion weight: 0.72" indicates that this node has feature associations with 6 neighboring nodes and contributes a weight of 0.72 in this level of fusion. After writing the attributes of all nodes, the final hierarchical fusion perception framework is generated. Each node in the framework carries complete feature identifiers, association information, and weight parameters, and can be directly scheduled to execute corresponding fusion and compensation operations.

[0051] Furthermore, the step of performing modal compensation and hierarchical fusion processing on the two types of raw data based on the hierarchical fusion sensing framework to generate corresponding initial sensing results includes: The original point cloud data from the millimeter-wave radar is divided into spatial grids to generate a point cloud structure index; the original image data from the visible light camera is segmented into multi-scale semantic segments to generate an image feature label set. The point cloud structure index is matched with the spatial location class nodes in the hierarchical fusion perception framework to determine the compensation weight value corresponding to each grid; the image feature label set is compared with the semantic attribute class nodes to retain feature labels with matching degree exceeding the preset matching threshold. Clutter suppression and effective point enhancement are performed on the point cloud raster with high compensation weight values ​​to generate corresponding compensated point cloud data; contrast enhancement and detail restoration are performed on the image region corresponding to the feature label to generate corresponding compensated image data; and multi-level fusion processing is performed on the compensated point cloud data and the compensated image data to generate the initial perception result.

[0052] It should be noted that the raw point cloud data from the millimeter-wave radar is uniformly divided into spatial grids using a 0.5m × 0.5m × 0.2m three-dimensional voxel grid. The three-dimensional space within the radar's detection range is divided into a 200 × 200 × 50 grid array. For each grid, three parameters are statistically analyzed: the number of points, average reflection intensity, and average radial velocity, generating a point cloud structure index. This rasterization process transforms disordered point clouds into structured raster data, significantly improving the computational efficiency of subsequent matching and compensation. Multi-scale semantic segmentation is performed on the raw image data from the visible light camera using a three-layer feature pyramid structure, outputting semantic labels at three scales: 1 / 4, 1 / 8, and 1 / 16, covering targets of different sizes and generating an image feature label set. The multi-scale labels can take into account both the overall semantics of large targets and the detailed features of small targets.

[0053] The point cloud structure index is matched with spatial location nodes in the hierarchical fusion perception framework at the grid level. Based on the degradation weight and compensation rules of the corresponding nodes, the compensation weight value for each grid is determined. Grids with more severe degradation receive higher compensation weights, resulting in stronger clutter suppression and effective point enhancement. Taking heavy rain as an example, distant grids (100-150 meters from the radar) have dense raindrop clutter, so the compensation weight is set to 0.75 after matching, performing strong clutter suppression. Closer grids (0-50 meters) have less clutter, so the compensation weight is 0.15, performing only mild smoothing. Simultaneously, the image feature label set is compared with semantic attribute nodes, retaining feature labels with a matching degree exceeding a preset threshold and filtering out low-confidence semantic labels caused by weather degradation to avoid erroneous semantic interference with the fusion results.

[0054] Targeted clutter suppression and effective point enhancement are performed on point cloud raster with high compensation weight values: In rainy and foggy weather, random clutter formed by raindrops and fog particles is filtered out, and outliers are filtered out using adaptive thresholding. Simultaneously, the point cloud reflectance intensity and spatial continuity of the real target are enhanced to generate compensated point cloud data. Contrast enhancement and detail restoration are performed on image regions corresponding to high-confidence feature labels: In foggy weather, dark channel prior dehazing combined with contrast-limited histogram equalization is used to improve scene contrast and restore edge details; in rainy weather, rain streak removal algorithms are used to repair areas obscured by raindrops, generating compensated image data. After completing single-modal compensation, multi-level feature fusion is performed from the bottom to the top layers of the hierarchical fusion perception framework: the spatial layer, semantic layer, and decision layer are fused sequentially. The bottom layer fuses spatial location and motion state, outputting the target's 3D bounding box and velocity information; the middle layer fuses semantic attributes, supplementing target category and fine-grained attributes; the top layer combines weather perturbation features to calibrate the final confidence level, ultimately outputting an initial perception result containing information such as target location, category, and motion state.

[0055] Furthermore, the step of performing multi-level fusion processing on the compensated point cloud data and the compensated image data to generate the initial perception result includes: Add corresponding spatial coordinate meta-labels to the compensated point cloud data, and add appropriate semantic attribute tags to the compensated image data; A corresponding fusion index table is created in real time based on the spatial coordinate meta-labels and the semantic attribute tags, and a corresponding spatial correlation matrix is ​​generated based on the compensated point cloud data and the compensated image data. According to the fusion index table, the corresponding target perception features are extracted in real time from the spatial correlation matrix, and the target perception features are input into a preset detection head for decoding processing to generate the initial perception result.

[0056] To compensate the point cloud data, corresponding spatial coordinate metadata labels are added. These labels include four pieces of information for each point cloud raster: raster ID, world coordinate system (XYZ) coordinates, average radar cross section, and average velocity. All labels use UTM projection coordinates under the WGS84 coordinate system, ensuring consistency with the image coordinate system. Adapted semantic attribute tags are added to the compensated image data. These tags include target category ID, classification confidence score, and pixel-level boundary mask. This tagging process provides a unified identification benchmark for the spatial alignment of the two types of data, avoiding fusion misalignment caused by inconsistencies in coordinate systems.

[0057] Based on spatial coordinate meta-labels and semantic attribute tags, a corresponding fusion index table is created in real time. The index table adopts a key-value pair structure, with the spatial raster ID as the primary key. Each record stores the corresponding point cloud feature address, image feature address, and correlation value. The query time is less than 1 millisecond, supporting fast queries with constant time complexity. Simultaneously, based on the spatial projection relationship between the compensated point cloud data and the compensated image data, a spatial correlation matrix is ​​generated. The matrix rows correspond to point cloud rasters, the columns correspond to image pixel blocks, and the matrix elements represent the feature correlation strength at corresponding locations, clearly presenting the spatial correspondence between the two types of data.

[0058] Guided by the fusion index table, the fusion perception features corresponding to the target region are extracted from the spatial correlation matrix. These multi-dimensional fusion features are then input into a pre-defined multi-task detection head network for decoding. The detection head employs a parallel branch structure, comprising three branches: 3D target detection, multi-target tracking, and attribute classification. These branches decode and output the target's 3D bounding box coordinates and category, trajectory ID and motion parameters, and vehicle type and color attributes, respectively. These are then integrated to form a complete initial perception result. Taking a foggy scene as an example, the final output perception result can achieve stable detection of vehicle targets within a 150-meter range, with a ranging error controlled within 1.5 meters and a classification accuracy maintained above 85%. The structured perception result can be directly input into the downstream decision-making and planning module, or used to reverse-verify the accuracy of the degradation mechanism matching library, supporting closed-loop iterative optimization of the entire perception system.

[0059] Please see Figure 2 The third embodiment of the present invention provides: A severe weather fusion sensing system based on millimeter-wave radar, wherein the system includes: The processing module is used to preprocess the synchronously acquired millimeter-wave radar raw point cloud data and visible light camera raw image data respectively, generate radar point cloud feature sets and image feature sets respectively, and jointly extract the weather disturbance feature vector of the current scene from the two types of raw data. The matching module is used to match the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, calculate the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type, and perform association mapping processing on the two feature sets based on the modal degradation weights to generate the corresponding modal association feature map. The construction module is used to construct a corresponding hierarchical fusion perception framework based on the modality association feature map. The hierarchical fusion perception framework includes multiple fusion levels and corresponding compensation rules. The verification module is used to perform modal compensation and hierarchical fusion processing on the two types of raw data based on the hierarchical fusion perception framework, generate corresponding initial perception results, and update the matching parameters of the degradation mechanism matching library in reverse according to the confidence deviation of the initial perception results.

[0060] Furthermore, the step of matching the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, and calculating the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type includes: Multiple weather feature parameters contained in the weather disturbance feature vector are extracted, and the coupling relationship between the multiple weather feature parameters is identified by a graph neural network. A corresponding weather coupling feature map is generated based on the coupling relationship. The radar point cloud feature set and the image feature set are respectively standardized and encoded to generate radar feature sequences and image feature sequences with the same corresponding dimensions. The two types of feature sequences are then mapped to the feature space corresponding to the weather coupled feature map through a feature mapping algorithm. The radar degradation weight value corresponding to the radar feature sequence and the weather coupled feature map, and the image degradation weight value corresponding to the image feature sequence and the weather coupled feature map are calculated respectively by a preset weight calculation function, and the two types of degradation weight values ​​are output as the modal degradation weight.

[0061] Furthermore, the step of performing association mapping processing on the two types of feature sets based on the modality degradation weights to generate corresponding modality association feature maps includes: The common spatial feature fields between the radar point cloud feature set and the image feature set are extracted in real time, and the association confidence of each common spatial feature field is calculated in real time based on the modal degradation weight. For shared spatial feature fields with association confidence scores higher than a preset threshold, the radar point cloud feature set and the corresponding feature vectors in the image feature set are fused in spatial dimension to generate corresponding fused feature units. For shared spatial feature fields with association confidence scores below a preset threshold, a coordinate mapping link between the two types of features is established using a spatial projection algorithm. Using the fusion feature unit as the core node and the coordinate mapping link as the connecting edge, the modal association feature map is formed accordingly.

[0062] Furthermore, the step of constructing the corresponding hierarchical fusion perception framework based on the modality association feature map includes: The node features in the modal association feature graph are labeled with types to distinguish spatial location nodes, semantic attribute nodes, motion state nodes and weather disturbance nodes, and the number of each type of node and the density of associated edges are counted. Based on the edge density of the spatial location nodes and the motion state nodes, the corresponding fusion level sorting is determined, and the key parameter information contained in the semantic attribute nodes and the weather disturbance nodes is extracted. Several levels of compensation sub-rules are then divided according to the degree of influence of the key parameter information. Based on the hierarchical relationship between the fusion hierarchy sorting and several hierarchical compensation sub-rules, a corresponding fusion framework structure tree is constructed, and the node ID of the modality association feature map is marked at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework.

[0063] Furthermore, the step of marking the node ID of the modality association feature map at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework includes: Each branch node of the fusion framework structure tree is hierarchically and deeply encoded to form a three-dimensional positioning code containing the root node identifier, the parent node identifier, and the node position. The node IDs in the modal association feature map are grouped and clustered according to node type, and the mapping relationship between each node ID and the three-dimensional positioning code is constructed in real time. According to the mapping relationship, the corresponding node ID is written into the attribute field of each branch node in the fusion framework structure tree, and the number of associated nodes and weight value of the node in the modal association feature map are marked on each node ID to generate the hierarchical fusion perception framework.

[0064] Furthermore, the step of performing modal compensation and hierarchical fusion processing on the two types of raw data based on the hierarchical fusion sensing framework to generate corresponding initial sensing results includes: The original point cloud data from the millimeter-wave radar is divided into spatial grids to generate a point cloud structure index; the original image data from the visible light camera is segmented into multi-scale semantic segments to generate an image feature label set. The point cloud structure index is matched with the spatial location class nodes in the hierarchical fusion perception framework to determine the compensation weight value corresponding to each grid; the image feature label set is compared with the semantic attribute class nodes to retain feature labels with matching degree exceeding the preset matching threshold. Clutter suppression and effective point enhancement are performed on the point cloud raster with high compensation weight values ​​to generate corresponding compensated point cloud data; contrast enhancement and detail restoration are performed on the image region corresponding to the feature label to generate corresponding compensated image data; and multi-level fusion processing is performed on the compensated point cloud data and the compensated image data to generate the initial perception result.

[0065] Furthermore, the step of performing multi-level fusion processing on the compensated point cloud data and the compensated image data to generate the initial perception result includes: Add corresponding spatial coordinate meta-labels to the compensated point cloud data, and add appropriate semantic attribute tags to the compensated image data; A corresponding fusion index table is created in real time based on the spatial coordinate meta-labels and the semantic attribute tags, and a corresponding spatial correlation matrix is ​​generated based on the compensated point cloud data and the compensated image data. According to the fusion index table, the corresponding target perception features are extracted in real time from the spatial correlation matrix, and the target perception features are input into a preset detection head for decoding processing to generate the initial perception result.

[0066] The fourth embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the severe weather fusion perception method based on millimeter-wave radar as described above.

[0067] The fifth embodiment of the present invention provides a readable storage medium storing a computer program thereon, wherein the program, when executed by a processor, implements the severe weather fusion perception method based on millimeter-wave radar as described above.

[0068] In summary, the severe weather fusion perception method and system based on millimeter-wave radar provided in the above embodiments of the present invention can establish an end-to-end closed-loop mechanism for weather feature extraction, degradation mechanism matching, and differentiated fusion compensation, and can continuously iteratively optimize the fusion logic, significantly improving the accuracy and environmental robustness of multimodal fusion perception under severe weather conditions.

[0069] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0070] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0071] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0072] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0073] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0074] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for fusion sensing of severe weather based on millimeter-wave radar, characterized in that, The method includes: The original point cloud data from the millimeter-wave radar and the original image data from the visible light camera were preprocessed separately to generate radar point cloud feature sets and image feature sets, and the weather disturbance feature vector of the current scene was jointly extracted from the two types of original data. By matching the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, it is determined whether the current weather belongs to fog, moderate rain, blizzard, sandstorm or mixed type. Based on the modal degradation type, the modal degradation weights corresponding to the radar point cloud feature set and the image feature set are calculated. Based on the modal degradation weights, the two types of feature sets are associated and mapped to generate corresponding modal association feature maps. A corresponding hierarchical fusion perception framework is constructed based on the modal association feature map. The hierarchical fusion perception framework includes multiple fusion levels and corresponding compensation rules. Based on the aforementioned hierarchical fusion perception framework, modal compensation and hierarchical fusion processing are performed on the two types of raw data respectively to generate corresponding initial perception results, and the matching parameters of the degradation mechanism matching library are updated in reverse according to the confidence deviation of the initial perception results. The steps of matching the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, and calculating the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type include: Multiple weather feature parameters contained in the weather disturbance feature vector are extracted, and the coupling relationship between the multiple weather feature parameters is identified by a graph neural network. A corresponding weather coupling feature map is generated based on the coupling relationship. The radar point cloud feature set and the image feature set are respectively standardized and encoded to generate radar feature sequences and image feature sequences with the same corresponding dimensions. The two types of feature sequences are then mapped to the feature space corresponding to the weather coupled feature map through a feature mapping algorithm. The radar degradation weight value corresponding to the radar feature sequence and the weather coupled feature map, and the image degradation weight value corresponding to the image feature sequence and the weather coupled feature map are calculated respectively by a preset weight calculation function, and the two types of degradation weight values ​​are output as the modal degradation weight.

2. The severe weather fusion perception method based on millimeter-wave radar according to claim 1, characterized in that, The step of performing association mapping processing on the two types of feature sets based on the modality degradation weights to generate corresponding modality association feature maps includes: The common spatial feature fields between the radar point cloud feature set and the image feature set are extracted in real time, and the association confidence of each common spatial feature field is calculated in real time based on the modal degradation weight. For shared spatial feature fields with association confidence scores higher than a preset threshold, the radar point cloud feature set and the corresponding feature vectors in the image feature set are fused in spatial dimension to generate corresponding fused feature units. For shared spatial feature fields with association confidence scores below a preset threshold, a coordinate mapping link between the two types of features is established using a spatial projection algorithm. Using the fusion feature unit as the core node and the coordinate mapping link as the connecting edge, the modal association feature map is formed accordingly.

3. The severe weather fusion perception method based on millimeter-wave radar according to claim 1, characterized in that, The step of constructing the corresponding hierarchical fusion perception framework based on the modality association feature map includes: The node features in the modal association feature graph are labeled with types to distinguish spatial location nodes, semantic attribute nodes, motion state nodes and weather disturbance nodes, and the number of each type of node and the density of associated edges are counted. Based on the edge density of the spatial location nodes and the motion state nodes, the corresponding fusion level sorting is determined, and the key parameter information contained in the semantic attribute nodes and the weather disturbance nodes is extracted. Several levels of compensation sub-rules are then divided according to the degree of influence of the key parameter information. Based on the hierarchical relationship between the fusion hierarchy sorting and several hierarchical compensation sub-rules, a corresponding fusion framework structure tree is constructed, and the node ID of the modality association feature map is marked at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework.

4. The severe weather fusion perception method based on millimeter-wave radar according to claim 3, characterized in that, The step of marking the node ID of the modality association feature map at each branch node of the fusion framework structure tree to generate the hierarchical fusion perception framework includes: Each branch node of the fusion framework structure tree is hierarchically and deeply encoded to form a three-dimensional positioning code containing the root node identifier, the parent node identifier, and the node position. The node IDs in the modal association feature map are grouped and clustered according to node type, and the mapping relationship between each node ID and the three-dimensional positioning code is constructed in real time. According to the mapping relationship, the corresponding node ID is written into the attribute field of each branch node in the fusion framework structure tree, and the number of associated nodes and weight value of the node in the modal association feature map are marked on each node ID to generate the hierarchical fusion perception framework.

5. The severe weather fusion perception method based on millimeter-wave radar according to claim 1, characterized in that, The steps of performing modal compensation and hierarchical fusion processing on the two types of raw data based on the hierarchical fusion sensing framework to generate corresponding initial sensing results include: The original point cloud data from the millimeter-wave radar is divided into spatial grids to generate a point cloud structure index; the original image data from the visible light camera is segmented into multi-scale semantic segments to generate an image feature label set. The point cloud structure index is matched with the spatial location class nodes in the hierarchical fusion perception framework to determine the compensation weight value corresponding to each grid; the image feature label set is compared with the semantic attribute class nodes to retain feature labels with matching degree exceeding the preset matching threshold. Clutter suppression and effective point enhancement are performed on the point cloud raster with high compensation weight values ​​to generate corresponding compensated point cloud data; contrast enhancement and detail restoration are performed on the image region corresponding to the feature label to generate corresponding compensated image data; and multi-level fusion processing is performed on the compensated point cloud data and the compensated image data to generate the initial perception result.

6. The severe weather fusion perception method based on millimeter-wave radar according to claim 5, characterized in that, The step of performing multi-level fusion processing on the compensated point cloud data and the compensated image data to generate the initial perception result includes: Add corresponding spatial coordinate meta-labels to the compensated point cloud data, and add appropriate semantic attribute tags to the compensated image data; A corresponding fusion index table is created in real time based on the spatial coordinate meta-labels and the semantic attribute tags, and a corresponding spatial correlation matrix is ​​generated based on the compensated point cloud data and the compensated image data. According to the fusion index table, the corresponding target perception features are extracted in real time from the spatial correlation matrix, and the target perception features are input into a preset detection head for decoding processing to generate the initial perception result.

7. A severe weather fusion perception system based on millimeter-wave radar, characterized in that, The system is used to implement the severe weather fusion perception method based on millimeter-wave radar as described in any one of claims 1 to 6, the system comprising: The processing module is used to preprocess the synchronously acquired millimeter-wave radar raw point cloud data and visible light camera raw image data respectively, generate radar point cloud feature sets and image feature sets respectively, and jointly extract the weather disturbance feature vector of the current scene from the two types of raw data. The matching module is used to match the modal degradation type corresponding to the weather disturbance feature vector through a preset degradation mechanism matching library, calculate the modal degradation weights corresponding to the radar point cloud feature set and the image feature set based on the modal degradation type, and perform association mapping processing on the two feature sets based on the modal degradation weights to generate the corresponding modal association feature map. The construction module is used to construct a corresponding hierarchical fusion perception framework based on the modality association feature map. The hierarchical fusion perception framework includes multiple fusion levels and corresponding compensation rules. The verification module is used to perform modal compensation and hierarchical fusion processing on the two types of raw data based on the hierarchical fusion perception framework, generate corresponding initial perception results, and update the matching parameters of the degradation mechanism matching library in reverse according to the confidence deviation of the initial perception results.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the severe weather fusion perception method based on millimeter-wave radar as described in any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the severe weather fusion perception method based on millimeter-wave radar as described in any one of claims 1 to 6.

Images