A multi-source heterogeneous sensor anti-interference fusion perception system

Abstract

The application relates to the technical field of information processing, in particular to a multi-source heterogeneous sensor anti-interference fusion perception system, which aims to solve the problem of decreased perception performance of multi-source heterogeneous sensors in adverse weather such as rain and fog; the system comprises a data acquisition and synchronization module, a multi-modal feature extraction and weather interference suppression module, an adaptive multi-modal fusion module, a target detection and tracking module and a perception result output module; the system extracts robust features from the interfered laser radar and camera data in combination with millimeter wave radar information through adversarial robust feature learning; and then the adaptive fusion module is used to dynamically adjust the weights of various modes according to the weather; the application effectively improves the perception accuracy and reliability of vehicles in complex environments such as rain and fog, and guarantees driving safety.

Description

Technical Field

[0001] This invention belongs to the field of information processing technology, specifically relating to a multi-source heterogeneous sensor anti-interference fusion sensing system. Background Technology

[0002] The rapid development of modern information technology and the Internet of Things has enabled sensor technology to play a central role in many fields such as environmental monitoring, industrial control, intelligent transportation, and healthcare. As a key entry point for acquiring information about the physical world, the accuracy, completeness, and reliability of the sensor data directly determine the decision-making quality of upper-level applications. To overcome the limitations of single sensors, multi-source heterogeneous information fusion technology has emerged, aiming to form a more comprehensive, accurate, and robust understanding of targets or environments by integrating data from different types and modalities of sensors.

[0003] Among them, multi-source heterogeneous sensor fusion sensing systems are key to achieving efficient information integration and intelligent decision-making. By comprehensively utilizing sensor data from various physical quantities such as light, sound, electricity, heat, and force, they can achieve information complementarity and redundancy, thereby significantly improving the accuracy and reliability of target detection, state estimation, and environmental modeling. These systems, through the effective processing and fusion of massive amounts of heterogeneous data, aim to build a digital model that more closely resembles the real world, providing a solid data foundation for intelligent applications.

[0004] However, existing multi-source heterogeneous sensor fusion sensing systems still face a series of severe challenges. First, the diversity of data formats, differences in acquisition frequencies, and spatiotemporal synchronization difficulties among heterogeneous sensors result in high costs for data preprocessing and alignment. Second, traditional fusion algorithms are highly sensitive to data quality in complex and variable environments, struggling to effectively handle missing or anomalous data, thus limiting the accuracy and robustness of the fusion results. More critically, sensor systems are highly susceptible to environmental noise, electromagnetic interference, sensor drift, and even deliberate attacks in practical applications. These interferences significantly reduce the signal-to-noise ratio of the original data, causing information distortion and severely impacting the accuracy and reliability of fusion sensing, even leading to system decision failure. These problems prevent existing fusion sensing systems from performing at their full potential in environments with strong interference or adversarial forces, necessitating urgent solutions. Summary of the Invention

[0005] This invention relates to a multi-source heterogeneous sensor anti-interference fusion perception system, aiming to solve the problem of significant performance degradation of multi-source heterogeneous sensors such as LiDAR, cameras, and millimeter-wave radar in existing autonomous driving or assisted driving systems under adverse weather conditions such as rain and fog. Existing solutions typically fail to effectively extract and fuse reliable information from various modalities when sensor data is interfered with by weather, leading to reduced accuracy and robustness in target detection and tracking, and consequently seriously affecting vehicle driving safety and decision-making.

[0006] To overcome the shortcomings of existing technologies, this invention proposes a multi-source heterogeneous sensor anti-interference fusion perception system. This system introduces an adversarial weather robust feature learning mechanism. Under adverse weather conditions, it learns and extracts robust features unaffected by weather interference from LiDAR point cloud data and camera image data. Simultaneously, it combines the inherent robustness of millimeter-wave radar to adverse weather conditions, achieving deep fusion of multi-source heterogeneous data. This system can significantly improve the vehicle's environmental perception capabilities in complex environments such as rain and fog, ensuring high accuracy and high reliability of the autonomous driving system.

[0007] According to one embodiment of the present invention, a multi-source heterogeneous sensor anti-interference fusion sensing system includes: a data acquisition and synchronization module, a multimodal feature extraction and weather interference suppression module, an adaptive multimodal fusion module, a target detection and tracking module, and a sensing result output module.

[0008] In one embodiment of the present invention, a data acquisition and synchronization module is used to acquire sensor data of the vehicle's surrounding environment in real time, and to timestamp and align this data with a spatial coordinate system. Specifically, this module acquires 3D point cloud data through a LiDAR sensor mounted on the vehicle platform, acquires visible light or infrared image data through a camera sensor, and acquires radar point target data, including information such as target distance, azimuth, radial velocity, and radar cross-section, through a millimeter-wave radar sensor. The data acquisition and synchronization module ensures precise temporal synchronization of the data from each sensor through a hardware triggering mechanism. Furthermore, the module performs preprocessing operations on the acquired raw sensor data, including removing sensor noise, performing distortion correction, and transforming all sensor data to a unified vehicle coordinate system, thereby providing standardized and consistent data input for subsequent processing.

[0009] In one embodiment of the present invention, the multimodal feature extraction and weather interference suppression module is the core component of the system. Its function is to extract multimodal features from preprocessed multi-source heterogeneous sensor data and effectively suppress interference caused by weather conditions such as rain and fog on lidar and camera data through an adversarial learning mechanism, thereby generating a weather-robust feature representation. This module includes a millimeter-wave radar feature extraction unit, a lidar feature extraction unit, a camera feature extraction unit, a weather feature perception unit, and an adversarial weather-robust feature learning unit.

[0010] In one embodiment of the present invention, a millimeter-wave radar feature extraction unit is used to extract high-level features from raw point target data of millimeter-wave radar. Specifically, the unit clusters radar point targets to identify potential target entities. Then, for each clustered target, features such as its geometric center, envelope size, average radial velocity, and maximum radar cross-section are extracted. Due to its wavelength characteristics, millimeter-wave radar data is relatively less affected by attenuation and scattering in rainy and foggy weather. The extracted features have high reliability and stability under adverse weather conditions and can serve as a reference information source for subsequent robust feature learning in adverse weather conditions. The millimeter-wave radar feature extraction unit encodes the extracted radar features into fixed-dimensional vector representations or grid-based bird's-eye view features for unified processing with other modal features.

[0011] In one embodiment of the present invention, a lidar feature extraction unit is used to extract geometric and intensity features from the three-dimensional point cloud data of a lidar system. Specifically, this unit separates the foreground target point cloud from the background point cloud using a point cloud segmentation algorithm. For the foreground target point cloud, geometric attributes such as the three-dimensional coordinates, reflection intensity information, normal vector, and local point density of each point are extracted. Under severe weather conditions, lidar point clouds become sparse and may contain false point clouds due to scattering from raindrops or fog droplets. The lidar feature extraction unit utilizes deep neural network structures, such as point cloud convolutional networks or graph neural networks, to perform feature learning on the point cloud data to generate feature vectors that can capture the shape, size, and spatial distribution of the target.

[0012] In one embodiment of the present invention, a camera feature extraction unit is used to extract visual semantic features and depth features from image data acquired by a camera. Specifically, this unit uses a deep convolutional neural network model to extract features from the image, generating feature maps at different scales. These feature maps capture edges, textures, colors, and high-level semantic information in the image. Under adverse weather conditions, images may suffer from reduced contrast due to fog or haze, or information loss due to raindrops or mist. The camera feature extraction unit further performs preliminary defogging or deraining processing on the image using image processing techniques or deep learning models to mitigate the direct impact of weather interference on feature extraction. Simultaneously, this unit can also infer scene depth information from the image using stereo vision or monocular depth estimation techniques, serving as auxiliary features for 3D perception.

[0013] In one embodiment of the present invention, a weather feature perception unit is used to assess the current environmental weather conditions in real time. Specifically, this unit can estimate weather parameters by analyzing fog, raindrop density, or brightness distribution in camera images. For example, the intensity of rain and fog can be quantified by image contrast, pixel grayscale distribution in the sky region, and detection of rain streaks. Furthermore, the unit can also combine the sparsity of lidar point clouds, the degree of reflection intensity attenuation, and the attenuation of millimeter-wave radar signals for a comprehensive judgment. The weather feature perception unit encodes the estimated weather conditions into a weather parameter vector, which serves as an important input to the adversarial weather robust feature learning unit and the adaptive multimodal fusion module, guiding the system to adjust its perception strategy under different weather conditions.

[0014] As one embodiment of the present invention, an adversarial weather-robust feature learning unit is key to achieving high-precision perception under adverse weather conditions. This unit aims to convert rain- and fog-affected LiDAR and camera features into weather-independent, robust feature representations. The unit comprises two cooperative and adversarial network components. The first network component, acting as a feature transformation module, receives features potentially affected by weather from the LiDAR and camera feature extraction units. The goal of this feature transformation module is to learn a mapping function that converts these affected features into weather-insensitive, semantically rich, and robust features. The second network component, acting as a feature discrimination module, receives inputs that can be the output features of the feature transformation module, relatively weather-independent radar features from the millimeter-wave radar feature extraction unit, or real, interference-free LiDAR or camera features acquired under clear weather conditions. The goal of the feature discrimination module is to distinguish whether the received features originate from the output of the feature transformation module or from a real, interference-free source. Through this adversarial training mechanism, the feature transformation module is forced to learn and generate weather-robust features sufficient to "deceive" the feature discrimination module; that is, the generated features are indistinguishable from real, interference-free features in the eyes of the discrimination module. During training, the features provided by the millimeter-wave radar feature extraction unit serve as stable reference anchors, assisting the feature discrimination module in making more accurate judgments. This guides the feature conversion module to learn feature representations that more closely resemble real-world environmental conditions. This adversarial training process minimizes the generation loss of the feature conversion module and the discrimination loss of the feature discrimination module, ultimately enabling the feature conversion module to stably output robust weather features under severe weather conditions.

[0015] In one embodiment of the present invention, an adaptive multimodal fusion module is used to deeply fuse LiDAR features, camera features, and millimeter-wave radar features after weather interference suppression to generate a comprehensive, high-dimensional environmental perception feature representation. This module employs a deep fusion network structure that effectively handles the heterogeneity between different modal features and dynamically adjusts the fusion weights of each modality feature based on real-time perceived weather conditions. Specifically, the deep fusion network contains multiple fusion layers, each embedding a cross-modal attention mechanism. The attention mechanism calculates and assigns contribution weights to each modality feature based on the importance of the input features and the current weather conditions. For example, in rainy or foggy weather, the fusion module automatically assigns higher weights to millimeter-wave radar features while relatively reducing the weights of LiDAR and camera features. In clear weather, the weights are adjusted accordingly to fully utilize the high-resolution detail information provided by LiDAR and cameras. This adaptive fusion strategy ensures that the system outputs optimal perception features under various weather conditions. Deeply fused networks also include a gating mechanism that dynamically controls the transmission and fusion of information flow based on learned intermodal correlations and weather parameters, thereby maximizing the retention of effective information and suppressing redundant or noisy information.

[0016] In one embodiment of the present invention, the target detection and tracking module receives comprehensive environmental perception features output by the adaptive multimodal fusion module and performs 3D target detection and multi-target tracking based on these features. Specifically, the target detection unit utilizes a deep learning-based detection network, such as a 3D point cloud detection network or a multi-view fusion detection network, to identify various target entities in the scene, such as vehicles, pedestrians, and bicycles, from the fused features. The detection results include the target's category, 3D bounding box (including length, width, and height), position coordinates, pose angle, and confidence score. The target tracking unit employs a multi-target tracking algorithm, such as a Kalman filter, extended Kalman filter, or deep relational network method, to continuously track the detected targets. This unit assigns a unique identifier to each target by associating and matching the detection results of the current frame with the tracking trajectories of historical frames, and estimates the target's motion state (including velocity and acceleration). When a target is occluded or temporarily lost, the tracking unit can use historical motion information for prediction and association to maintain tracking stability.

[0017] In one embodiment of the present invention, the perception result output module is used to output the perception results generated by the target detection and tracking module to the vehicle's decision-making and planning system or other driver assistance systems in a predefined standard format. The output information includes the unique ID, category, precise three-dimensional position, size, attitude, speed, acceleration, and tracking status of each identified target. Furthermore, this module can also output the current environmental weather condition assessment results, road drivable area information, and potential obstacle risk levels, providing comprehensive and accurate environmental perception support for the upper-level decision-making system.

[0018] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0019] The present invention proposes a multi-source heterogeneous sensor anti-interference fusion perception system, which significantly improves the environmental perception capability of autonomous vehicles under adverse weather conditions such as rain and fog.

[0020] First, through an adversarial weather-robust feature learning unit, this system can effectively learn and extract weather-independent, highly robust semantic features from weather-affected LiDAR point clouds and camera images. This mechanism fundamentally solves the problem of perception performance degradation caused by the decline in sensor data quality under adverse weather conditions in traditional fusion methods. Adversarial learning makes the feature extraction process highly adaptable to weather changes and resistant to interference, ensuring the stability and reliability of the features.

[0021] Second, this system fully utilizes the inherent robustness of millimeter-wave radar under adverse weather conditions. The relatively stable perception information provided by millimeter-wave radar is cleverly integrated into the adversarial learning process as a weather-insensitive feature, serving as a "real" or "reference" to guide and constrain the training of the adversarial network, thereby further improving the accuracy and effectiveness of lidar and camera feature de-interference.

[0022] Third, the adaptive multimodal fusion module can dynamically adjust the fusion weights of each modality feature based on real-time perceived weather conditions. In severe weather, the system will automatically increase the contribution weight of millimeter-wave radar features while appropriately reducing the weight of LiDAR and camera features, which are more susceptible to weather conditions, and vice versa. This intelligent adaptive fusion strategy ensures that the system can achieve optimal sensing performance under different weather conditions, effectively balancing the strengths and weaknesses of each sensor.

[0023] Fourth, the end-to-end deep learning architecture adopted by this system enables seamless and efficient processing from raw sensor data to the final target detection and tracking results. The overall system structure is compact, the processing flow is optimized, and the computational efficiency is high, meeting the stringent real-time requirements of autonomous driving systems.

[0024] Fifth, through the above-mentioned innovative mechanism, the system of this invention has achieved significant improvements over existing technologies in key indicators such as target detection accuracy, tracking stability, and false alarm rate under adverse weather conditions such as rain and fog. This greatly enhances the environmental perception reliability and safety of autonomous vehicles, and provides a solid perception foundation for the application of advanced driver assistance and fully autonomous driving functions in complex environments. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention;

[0026] Figure 2 This is a schematic diagram of the robust feature learning unit architecture for adversarial weather in this invention. Detailed Implementation

[0027] This embodiment provides a multi-source heterogeneous sensor anti-interference fusion perception system, aiming to solve the problem of significant performance degradation of multi-source heterogeneous sensors such as LiDAR, cameras, and millimeter-wave radar in existing autonomous driving or assisted driving systems under adverse weather conditions such as rain and fog. This system introduces an adversarial weather robust feature learning mechanism to learn and extract robust features unaffected by weather interference from LiDAR point cloud data and camera image data under adverse weather conditions. Simultaneously, it combines the inherent robustness of millimeter-wave radar to adverse weather conditions to achieve deep fusion of multi-source heterogeneous data. This system significantly improves the vehicle's environmental perception capability in complex environments such as rain and fog, ensuring high accuracy and high reliability of the autonomous driving system.

[0028] Please refer to Figure 1 and Figure 2 This system includes a data acquisition and synchronization module, a multimodal feature extraction and weather interference suppression module, an adaptive multimodal fusion module, a target detection and tracking module, and a perception result output module.

[0029] The data acquisition and synchronization module is used to acquire sensor data of the vehicle's surrounding environment in real time and to timestamp and align this data with the spatial coordinate system. This module works in concert with a high-resolution LiDAR sensor, a high dynamic range camera sensor, and a high-precision millimeter-wave radar sensor mounted on the vehicle platform to comprehensively capture environmental information. The LiDAR sensor acquires 3D point cloud data, which includes the precise 3D spatial coordinates and reflection intensity information of each point, typically acquired at a frequency of tens of thousands or even hundreds of thousands of points per second. The camera sensor acquires visible light or infrared image data, with image resolution typically reaching high-definition or even ultra-high-definition standards and frame rates exceeding thirty frames per second, used to capture rich visual texture, color, and semantic information. The millimeter-wave radar sensor acquires radar point target data, including key information such as the target's distance, azimuth, radial velocity, and radar cross-section. Its detection range typically reaches hundreds of meters and has strong penetration capabilities even in adverse weather conditions.

[0030] The data acquisition and synchronization module ensures precise temporal synchronization of data from all sensors through a hardware triggering mechanism. For example, a central synchronization unit sends a unified trigger signal to all sensors, causing them to begin data acquisition simultaneously, thus ensuring that the data acquired by different sensors correspond to the same snapshot. This time synchronization accuracy is typically required to be at the millisecond or even microsecond level to ensure the accuracy of subsequent data fusion. After data acquisition, this module performs preprocessing operations on the raw sensor data. Preprocessing first includes removing sensor noise, such as outliers in LiDAR point clouds, salt-and-pepper noise or Gaussian noise in camera images, and clutter interference in millimeter-wave radar. Noise removal can employ statistical filtering, median filtering, or deep learning-based denoising algorithms. Next, distortion correction is performed, such as radial and tangential distortion correction for camera images, and mechanical mounting error correction for LiDAR and millimeter-wave radar, ensuring the accuracy of the data's geometric relationships. Finally, this module transforms all sensor data into a unified vehicle coordinate system. This is typically accomplished through precise sensor intrinsic and extrinsic parameter calibration, transforming the coordinate systems of each sensor to the coordinate system of the vehicle center or a fixed reference point, such as the rear axle center point coordinate system. Calibration data, such as the three-dimensional translation vectors and three-dimensional rotation matrices of the sensor mounting positions, is usually measured and stored offline with high precision during vehicle manufacturing or repair, and can be adaptively calibrated online during system operation to compensate for long-term drift. This standardized and consistent data input provides a solid foundation for subsequent multimodal feature extraction and fusion processing, ensuring the uniformity and availability of the data source.

[0031] The multimodal feature extraction and weather interference suppression module is the core component of this system. Its function is to extract multimodal features from preprocessed multi-source heterogeneous sensor data and effectively suppress interference from weather conditions such as rain and fog on lidar and camera data through an adversarial learning mechanism, thereby generating a weather-robust feature representation. This module includes a millimeter-wave radar feature extraction unit, a lidar feature extraction unit, a camera feature extraction unit, a weather feature perception unit, and an adversarial weather-robust feature learning unit.

[0032] The millimeter-wave radar feature extraction unit is used to extract high-level features from raw point target data of millimeter-wave radar. This unit first clusters the radar point targets to identify potential target entities. Clustering algorithms can employ density-based methods such as DBSCAN or model-based methods such as Gaussian mixture models, grouping spatially close point targets with similar radial velocities into the same cluster. Next, for each cluster, features such as its geometric center, envelope dimensions (e.g., length, width, height), average radial velocity, and maximum radar cross-section are extracted. The geometric center is determined by calculating the average of the three-dimensional coordinates of all point targets within the cluster. The envelope size is determined by the maximum and minimum range differences of the point targets within the cluster along the three axes. The average radial velocity is a weighted average of the radial velocities of all point targets within the cluster, with weights determined based on the point target's strength or reliability. The maximum radar cross-section reflects the target's reflection intensity of radar waves and is closely related to the target's material and orientation. Due to its wavelength characteristics, millimeter-wave radar data is less affected by attenuation and scattering in rainy and foggy weather. The extracted features exhibit high reliability and stability under adverse weather conditions, serving as a valuable reference source for subsequent robust feature learning in adverse weather conditions. The millimeter-wave radar feature extraction unit encodes the extracted radar features into fixed-dimensional vector representations or grid-based bird's-eye view features. For example, vector representations can concatenate the target's geometric center coordinates, size, velocity, etc., into a one-dimensional feature vector. Bird's-eye view features project radar point targets onto a two-dimensional grid map, encoding radar features such as point density and average velocity in each grid cell for unified processing with other modal features, providing standardized input for subsequent fusion.

[0033] The lidar feature extraction unit extracts geometric and intensity features from the lidar's 3D point cloud data. This unit first separates the foreground target point cloud from the background point cloud using a point cloud segmentation algorithm. The segmentation algorithm can employ deep learning-based methods such as semantic segmentation networks, or traditional geometric analysis methods such as RANSAC plane fitting to remove ground points, followed by connected component analysis to identify the foreground target. For the foreground target point cloud, geometric attributes such as 3D coordinates, reflection intensity information, normal vector, and local point density are extracted for each point. The 3D coordinates are directly provided by the sensor. Reflection intensity reflects the reflectivity of the object's surface. The normal vector is calculated through principal component analysis of the local point's neighborhood, representing the surface orientation. Local point density is measured by calculating the number of points within each point's neighborhood sphere, reflecting the density of the target surface. Under adverse weather conditions, such as raindrops or fog droplets, the lidar point cloud absorbs and scatters the laser beam, causing it to become sparse and potentially resulting in false point clouds due to scattering by raindrops or fog droplets. The lidar feature extraction unit utilizes deep neural network structures, such as point cloud convolutional networks or graph neural networks, to learn features from point cloud data, generating feature vectors that capture the shape, size, and spatial distribution of the target. For example, networks such as PointNet++ or DGCNN can effectively process unstructured point cloud data and capture the geometric characteristics of the target at different granularities through multi-scale feature learning mechanisms, generating fixed-dimensional feature descriptors for subsequent fusion processing.

[0034] The camera feature extraction unit extracts visual semantic features and depth features from image data acquired by the camera. This unit employs a deep convolutional neural network model to extract features from the image, generating feature maps at different scales. These feature maps capture edges, textures, colors, and high-level semantic information in the image. For example, using backbone networks such as ResNet and EfficientNet, through multi-layer convolution and pooling operations, different levels of feature representations are gradually abstracted from the original pixel information, from small-scale edges and textures to large-scale high-level semantic concepts such as vehicles and pedestrians. In adverse weather conditions, images may suffer from reduced contrast due to fog or haze, or information loss due to raindrops or mist. The camera feature extraction unit further performs preliminary dehazing or deraining processing on the image using image processing techniques or deep learning models to mitigate the direct impact of weather interference on feature extraction. For example, a physics-based dehazing algorithm inverts a fog-free image using an atmospheric scattering model, while a deep learning-based deraining algorithm learns and removes rain stripe features using a residual network. Simultaneously, this unit can also infer scene depth information from the image using stereo vision or monocular depth estimation techniques, serving as auxiliary features for 3D perception. Stereo vision calculates depth by the disparity between two images, while monocular depth estimation directly predicts depth maps from a single image using a deep learning network, thus supplementing areas lacking LiDAR information.

[0035] The weather feature perception unit is used to assess the current environmental weather conditions in real time. This unit can estimate weather parameters by analyzing fog, raindrop density, or brightness distribution in camera images. For example, fog intensity can be quantified by image contrast, based on the dark channel prior theory proposed by He et al.; brightness changes can be detected by the pixel grayscale distribution in the sky region; and rain intensity and density can be quantified by detecting rain streaks through Hough transform or deep learning models. Furthermore, this unit can also combine the sparsity of lidar point clouds, the degree of reflection intensity attenuation, and the attenuation of millimeter-wave radar signals for a comprehensive judgment. For example, lidar point clouds become sparser and reflection intensity is generally reduced in fog. While millimeter-wave radar is robust in heavy rainfall, it still experiences some attenuation. The weather feature perception unit encodes the estimated weather conditions into a weather parameter vector. This vector typically contains multiple dimensions of weather indicators, such as fog intensity, rain intensity, visibility estimates, and weather type classifications, such as sunny, light rain, and heavy fog. These parameters serve as important inputs to the adversarial weather robust feature learning unit and the adaptive multimodal fusion module, guiding the system's perception strategy adjustment under different weather conditions and ensuring that the system can respond optimally to the actual environment.

[0036] The adversarial weather-robust feature learning unit is crucial for achieving high-precision perception under adverse weather conditions. This unit aims to transform rain- and fog-affected LiDAR and camera features into weather-independent, robust feature representations. It comprises two cooperative and adversarial network components: a feature transformation module and a feature discrimination module.

[0037] The feature transformation module receives features from the LiDAR feature extraction unit and the camera feature extraction unit that may be affected by weather conditions. These features include, for example, feature vectors or feature maps extracted by a deep neural network. The goal of the feature transformation module is to learn a mapping function that transforms these affected features into weather-insensitive, semantically robust features. This module typically consists of one or more deep neural network layers, such as convolutional layers, fully connected layers, and residual blocks, whose parameters are optimized during training to achieve the feature transformation.

[0038] The input to the feature discrimination module can be the output features of the feature transformation module, radar features from the millimeter-wave radar feature extraction unit that are relatively unaffected by weather, or real, interference-free features from LiDAR or cameras collected under clear weather conditions. The goal of the feature discrimination module is to distinguish whether the features it receives come from the output of the feature transformation module or from a real, interference-free source. This module typically consists of a binary classification deep neural network that outputs a probability value representing the source of the feature.

[0039] Through this adversarial training mechanism, the feature transformation module is forced to learn to generate weather-robust features sufficient to "deceive" the feature discrimination module; that is, the generated features are indistinguishable from real, interference-free features in the discriminator's view. During training, features provided by the millimeter-wave radar feature extraction unit serve as stable reference anchors, assisting the feature discrimination module in making more accurate judgments, thereby guiding the feature transformation module to learn feature representations that more closely resemble real-world environmental conditions. This reference anchor mechanism effectively solves the problem of lacking ground truth data under severe weather conditions, leveraging the inherent robustness of millimeter-wave radar to provide reliable supervisory signals for de-interference of lidar and camera features. This adversarial training process minimizes the generation loss of the feature transformation module and the discrimination loss of the feature discrimination module.

[0040] The total loss function for adversarial training can be expressed as:

[0041]

[0042] in, Represents the total loss of the entire sensing system. This represents the perceptual loss generated by object detection and tracking tasks, such as classification loss and localization loss. The weighted hyperparameters are used to balance the task loss and the adversarial loss. It is the adversarial loss between the feature transformation module and the feature discrimination module, and its form is similar to the loss function in generative adversarial networks. It can be defined as:

[0043]

[0044] Here, Expressing expectations, This represents genuine, interference-free features, such as features from millimeter-wave radar or features from lidar / camera in clear weather. Features of lidar or camera systems that are affected by weather interference; The mapping function representing the feature transformation module; This represents the discriminant function of the feature discrimination module. The goal of the feature discrimination module is to maximize... This is to accurately distinguish between real features and transformed features, and the goal of the feature transformation module is to minimize... This generates features capable of deceiving the discrimination module. Ultimately, this enables the feature transformation module to stably output weather-robust features under severe weather conditions, which are semantically consistent with features under clear weather conditions and are unaffected by changes in weather conditions.

[0045] The adaptive multimodal fusion module deeply fuses features from LiDAR, camera, and millimeter-wave radar (after weather interference suppression) to generate a comprehensive, high-dimensional environmental perception feature. This module employs a deep fusion network structure that effectively handles the heterogeneity between different modal features and dynamically adjusts the fusion weights of each modality based on real-time perceived weather conditions. The deep fusion network contains multiple fusion layers, each embedding a cross-modal attention mechanism. This attention mechanism calculates and assigns contribution weights to each modality feature based on its importance and the current weather conditions. For example, in rainy or foggy weather, the fusion module automatically assigns higher weights to millimeter-wave radar features while relatively reducing the weights of LiDAR and camera features. In clear weather, the weights are adjusted accordingly to fully utilize the high-resolution detail information provided by LiDAR and camera.

[0046] Specifically, it integrates environmental perception features It can be represented by the following weighted sum with weather perception weights:

[0047]

[0048] in, , These are the robust features of the LiDAR and camera after adversarial learning. This is a characteristic of millimeter-wave radar. It is a weather parameter vector output by the weather feature sensing unit. It is an attention weight calculation function implemented by a deep neural network, which dynamically generates modalities based on all input modal features and real-time weather conditions. Fusion weights These weights satisfy ,and The value reflects the reliability and importance of the corresponding mode in the current environment. For example, in heavy fog, The value will be significantly increased, while and The value will decrease accordingly. This adaptive fusion strategy ensures that the system can output optimal sensing features under various weather conditions, effectively balancing the strengths and weaknesses of each sensor, thereby improving overall sensing performance. The deep fusion network also includes a gating mechanism, which dynamically controls the transmission and fusion of information flow based on the learned inter-modal correlations and weather parameters, thereby maximizing the retention of effective information and suppressing redundant or noisy information, further improving the quality of fused features. For example, the gating unit can decide which specific features of which modalities should be strengthened or weakened under specific weather conditions to optimize information flow and prevent low-quality data from contaminating high-quality data.

[0049] The target detection and tracking module receives comprehensive environmental perception features output by the adaptive multimodal fusion module and performs 3D target detection and multi-target tracking based on these features. The target detection unit utilizes deep learning-based detection networks, such as 3D point cloud detection networks or multi-view fusion detection networks, to identify various target entities in the scene, including vehicles, pedestrians, and bicycles, from the fused features. The detection network can employ a single-stage or two-stage architecture, such as PointRCNN or CenterPoint, to directly predict targets from the fused features. Detection results include the target category (e.g., car, truck, pedestrian, bicycle), a precise 3D bounding box containing length, width, and height, 3D position coordinates, attitude angles such as yaw, pitch, and roll, and confidence scores. These output parameters provide accurate spatial localization and geometric description of the target.

[0050] The target tracking unit employs multi-target tracking algorithms, such as Kalman filtering, extended Kalman filtering, or deep relational networks, to continuously track detected targets. This unit correlates the detection results of the current frame with the tracking trajectories of historical frames. Correlation matching can be based on the target's 3D position, motion state, feature similarity (e.g., appearance features or fusion features), and matching algorithms such as the Hungarian algorithm. Through correlation matching, a unique identifier is assigned to each target, ensuring consistency in target identity across consecutive frames. Simultaneously, the tracking unit estimates the target's motion state, including velocity and acceleration. Kalman filtering or extended Kalman filtering is used to predict and update the target state, smoothing target motion and reducing the impact of measurement noise. When a target is occluded or briefly lost, the tracking unit can utilize historical motion information for prediction and correlation, such as predicting future positions from the target trajectory, and attempt to re-detect and correlate in subsequent frames, thereby maintaining tracking stability. Furthermore, deep relational networks can learn more complex feature correlations, improving the accuracy of re-identification even under long-term occlusion.

[0051] The perception result output module outputs the perception results generated by the target detection and tracking module to the vehicle's decision-making and planning system or other driver assistance systems in a predefined standard format. Output information includes the unique ID, category, precise 3D position, size, attitude, velocity, acceleration, and tracking status (e.g., active, lost, newly created) for each identified target. This information is transmitted in real-time via high-speed communication interfaces such as Ethernet or PCIe in structured data formats such as JSON, Protobuf, or ROS messages. Additionally, this module can output current environmental weather condition assessments, such as quantified fog intensity, rain intensity, and visibility values. It also outputs information on drivable road areas, such as free space areas generated through semantic segmentation or drivable area detection algorithms, and potential obstacle risk levels, such as risk scores calculated based on target collision probability or hazard level. This comprehensive and accurate environmental perception support information provides ample basis for the upper-level decision-making and planning system, enabling it to make safe and efficient driving decisions based on real-time environmental conditions and perception results, including path planning, behavior prediction, and emergency braking, thereby greatly enhancing the reliability and safety of autonomous vehicles in complex environments.

[0052] The system in this embodiment, through an adversarial weather-robust feature learning unit, can effectively learn and extract weather-independent, highly robust semantic features from weather-affected LiDAR point clouds and camera images. This mechanism fundamentally solves the problem of perception performance degradation caused by the decline in sensor data quality under severe weather conditions in traditional fusion methods. Adversarial learning makes the feature extraction process highly adaptable and anti-interference-resistant to weather changes, ensuring the stability and reliability of the features. This system fully utilizes the inherent robustness of millimeter-wave radar under severe weather conditions. The relatively stable perception information provided by millimeter-wave radar is integrated into the adversarial learning process as a real or reference weather-insensitive feature, used to guide and constrain the training of the adversarial network, thereby further improving the accuracy and effectiveness of LiDAR and camera feature de-interference. The adaptive multimodal fusion module can dynamically adjust the fusion weights of each modality feature according to the real-time perceived weather conditions. Under severe weather conditions, the system will automatically increase the contribution ratio of millimeter-wave radar features, while appropriately reducing the ratio of LiDAR and camera features that are more affected by weather. Conversely, in clear weather, the system increases the weighting of LiDAR and camera features to utilize their high-resolution detail. This intelligent adaptive fusion strategy ensures optimal perception performance under various weather conditions, effectively balancing the strengths and weaknesses of each sensor. The end-to-end deep learning architecture employed in this system enables seamless and efficient processing from raw sensor data to final target detection and tracking results. The overall system structure is compact, with optimized processing flow and high computational efficiency, meeting the stringent real-time requirements of autonomous driving systems. Through these innovative mechanisms, this system achieves significant improvements over existing technologies in key indicators such as target detection accuracy, tracking stability, and false alarm rate under adverse weather conditions like rain and fog. This greatly enhances the environmental perception reliability and safety of autonomous vehicles, providing a solid perception foundation for the application of advanced driver assistance systems (ADAS) and fully autonomous driving functions in complex environments.

Claims

1. A multi-source heterogeneous sensor anti-interference fusion sensing system, characterized in that, include: The data acquisition and synchronization module is used to acquire LiDAR data, camera data, and millimeter-wave radar data in real time, and to perform timestamp synchronization, spatial coordinate system alignment, and preprocessing on the LiDAR data, camera data, and millimeter-wave radar data. The multimodal feature extraction and weather interference suppression module is used to extract multimodal features from the lidar data and the camera data, suppress weather interference through an adversarial learning mechanism to generate weather robust features, and extract millimeter-wave radar features from the millimeter-wave radar data. An adaptive multimodal fusion module is used to deeply fuse the weather robust features and the millimeter-wave radar features to generate comprehensive environmental perception features; The target detection and tracking module performs target detection and tracking based on the comprehensive environmental perception features. The perception result output module is used to output the recognition results of the target detection and tracking module to an external system; The multimodal feature extraction and weather interference suppression module, It includes a feature transformation module and a feature discrimination module; The feature conversion module is used to receive weather-affected multimodal features from the lidar feature extraction unit and the camera feature extraction unit, and learn a mapping function to convert them into weather-insensitive, semantically robust weather features. The feature discrimination module is used to distinguish whether the feature it receives is the output of the feature conversion module, or the millimeter-wave radar feature from the millimeter-wave radar feature extraction unit, or the real, interference-free feature collected under clear weather conditions. Through an adversarial training mechanism between the feature conversion module and the feature discrimination module, the feature conversion module is guided to learn and generate weather-robust features sufficient to deceive the feature discrimination module. The millimeter-wave radar features serve as a stable reference anchor point to assist the feature discrimination module in making judgments.

2. The multi-source heterogeneous sensor anti-interference fusion sensing system according to claim 1, characterized in that, The data acquisition and synchronization module is used to acquire 3D point cloud data generated by the lidar sensor, image data generated by the camera sensor, and radar point target data generated by the millimeter-wave radar sensor. The preprocessing includes removing sensor noise, performing distortion correction, and transforming all sensor data into a unified vehicle coordinate system.

3. The multi-source heterogeneous sensor anti-interference fusion sensing system according to claim 1, characterized in that, The multimodal feature extraction and weather interference suppression module includes: a millimeter-wave radar feature extraction unit, a lidar feature extraction unit, a camera feature extraction unit, a weather feature perception unit, and an adversarial weather robust feature learning unit.

4. The multi-source heterogeneous sensor anti-interference fusion sensing system according to claim 3, characterized in that, The millimeter-wave radar feature extraction unit is used to perform clustering processing on the original radar point target data of the millimeter-wave radar to identify potential target entities, and extract the geometric center, envelope size, average radial velocity and maximum radar cross-section features of each clustered target, and encode the extracted radar features into a fixed-dimensional vector representation or a grid-based bird's-eye view feature.

5. The multi-source heterogeneous sensor anti-interference fusion sensing system according to claim 3, characterized in that, The lidar feature extraction unit is used to separate the foreground target point cloud from the background point cloud in the three-dimensional point cloud data generated by the lidar sensor through a point cloud segmentation algorithm, and extract the three-dimensional coordinates, reflection intensity information, normal vector and local point density geometric attributes of each point from the foreground target point cloud. It also uses a deep neural network structure to perform feature learning on the point cloud data to generate feature vectors that can capture the shape, size and spatial distribution of the target.

6. The multi-source heterogeneous sensor anti-interference fusion sensing system according to claim 3, characterized in that, The camera feature extraction unit is used to extract visual semantic features and depth features from the image data acquired by the camera using a deep convolutional neural network model, and to perform preliminary defogging or deraining processing on the image data using image processing technology or a deep learning model to reduce weather interference, and further infer the depth information of the scene from the image using stereo vision or monocular depth estimation technology.

7. The multi-source heterogeneous sensor anti-interference fusion sensing system according to claim 3, characterized in that, The weather feature sensing unit is used to estimate weather parameters by analyzing the fog, raindrop density, or brightness distribution in the camera image, and to comprehensively judge the current environmental weather conditions by combining the sparsity of the lidar point cloud, the degree of reflection intensity attenuation, and the signal attenuation of the millimeter-wave radar, and to encode the estimated weather conditions into a weather parameter vector.

8. The multi-source heterogeneous sensor anti-interference fusion sensing system according to claim 7, characterized in that, The adaptive multimodal fusion module employs a deep fusion network structure to deeply fuse the weather robust features, the millimeter-wave radar features, and the weather parameter vector output by the weather feature perception unit to generate the comprehensive environmental perception features. The deep fusion network structure contains multiple fusion layers, each embedding a cross-modal attention mechanism. The attention mechanism dynamically calculates and adjusts the fusion weights of each modality feature based on the importance of the input features and the weather parameter vector to ensure that the system can output the optimal perception features under various weather conditions.

9. The multi-source heterogeneous sensor anti-interference fusion sensing system according to claim 8, characterized in that, The target detection and tracking module includes a target detection unit and a target tracking unit; The target detection unit is used to identify various target entities in the scene from the comprehensive environmental perception features using a deep learning-based detection network, and form a detection result. The detection result includes the target's category, three-dimensional bounding box, position coordinates, pose angle, and confidence level. The target tracking unit is used to continuously track detected targets using a multi-target tracking algorithm. By associating and matching the detection results of the current frame with the tracking trajectories of historical frames, a unique identifier is assigned to each target, and the motion state of the target is estimated. The motion state includes velocity and acceleration. When the target is occluded or briefly lost, the target tracking unit can use historical motion information for prediction and association to maintain the stability of tracking.

Images