Sensor data fusion system based on attention mechanism dynamic weight distribution

Abstract

The invention relates to a sensor data fusion system based on attention mechanism dynamic weight distribution, the system comprises a multi-sensor space-time synchronization module, the multi-sensor space-time synchronization module comprises a time synchronization unit and a space synchronization unit, the time synchronization unit is used for a GPS / IMU to obtain a unified time reference, interpolation calibration is carried out on sampling timestamps of all sensors, and the space synchronization unit is used for carrying out space synchronization; and the space synchronization unit is used for converting the laser radar point cloud and the 4D radar data to a camera coordinate system based on sensor external parameter calibration so as to achieve unification of the space dimension. According to the method, the target detection accuracy is improved by 15%-30%, the space-time deviation of multi-sensor data is reduced, and a high-precision data basis is provided for fusion.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor fusion technology, and in particular to a sensor data fusion system based on dynamic weight allocation using an attention mechanism. Background Technology

[0002] With the development of autonomous driving technology, the requirements for the accuracy and reliability of vehicle perception in complex environments are increasing. Single sensors (such as cameras and LiDAR) have significant limitations in complex weather conditions (rain, fog, snow) or sudden changes in lighting: cameras are susceptible to light effects, and image noise surges in rainy or foggy weather; LiDAR suffers from sparse point clouds and large fluctuations in reflectivity in rainy or snowy weather; while 4D radar has all-weather capabilities, its target resolution is relatively low. Therefore, multi-sensor fusion has become a key technology for improving the robustness of environmental perception.

[0003] Existing multi-sensor fusion solutions have the following shortcomings: Insufficient spatiotemporal synchronization accuracy: Differences in sampling frequency and installation location of different sensors lead to deviations in data timestamps and spatial coordinate systems, directly affecting the fusion effect; Fixed feature fusion strategy: Most solutions use static weight allocation (such as a fixed fusion ratio of camera and LiDAR), which cannot dynamically adjust the weight of each sensor according to real-time environmental conditions (such as rain and snow intensity, light intensity), resulting in a significant decrease in accuracy under extreme weather conditions; Poor adaptability to complex weather: The lack of a dynamic compensation mechanism for sensor performance degradation, such as relying on camera features in rainy and foggy weather, leads to increased false detection and missed detection rates. Summary of the Invention

[0004] In view of the above problems, the present invention provides a sensor data fusion system based on dynamic weight allocation of attention mechanism, which not only improves the target detection accuracy by 15%-30%, but also reduces the spatiotemporal deviation of multi-sensor data, providing a high-precision data foundation for fusion.

[0005] To achieve the above and other related objectives, the present invention provides the following technical solution: A sensor data fusion system based on dynamic weight allocation using an attention mechanism, the system comprising: The multi-sensor spatiotemporal synchronization module includes a time synchronization unit and a spatial synchronization unit. The time synchronization unit is used to obtain a unified time reference from GPS / IMU and to interpolate and calibrate the sampling timestamps of each sensor to achieve data alignment in the time dimension. The spatial synchronization unit is used to convert LiDAR point cloud and 4D radar data to the camera coordinate system based on sensor extrinsic parameter calibration to achieve spatial dimension unification. The environmental perception and sensor performance evaluation module is connected to the multi-sensor spatiotemporal synchronization module. It is used to collect environmental parameters in real time, analyze visibility and precipitation intensity through camera images, determine the degree of snow particle interference through the reflectivity fluctuation of lidar point cloud, and quantify the effectiveness of each sensor based on environmental parameters. The feature extraction module, connected to the environmental perception and sensor performance evaluation module, is used to extract features from the synchronized sensor data. A dynamic weight fusion module based on an attention mechanism is connected to the feature extraction module and is used to fuse data from each sensor using a feature fusion algorithm that constructs a dual-channel attention network. The target detection module, connected to the attention-based dynamic weight fusion module, is used to input the fused global features into the target detection model and output target category, location, and velocity information.

[0006] Furthermore, the feature extraction of the synchronized sensor data includes extracting the visual features of the target through a deep learning model and extracting the three-dimensional geometric features of the target through point cloud clustering and segmentation algorithms.

[0007] Furthermore, the feature extraction of the synchronized sensor data also includes extracting the target's velocity features and radar cross-section features based on the synchronized 4D radar data.

[0008] Furthermore, the feature fusion algorithm employing a dual-channel attention network to fuse data from each sensor includes: L1. Input the environmental parameters and the performance scores of each sensor into the environmental perception channel attention network to characterize the initial weight coefficients of the data fusion of each sensor and obtain the data information of the initial weight coefficients of the data fusion of each sensor. L2. Input the feature extraction data from each sensor into the feature interaction channel attention network, perform cross-attention calculation on the features of multiple sensors, explore the correlation between features, and output the dynamically adjusted weights for the fusion of each sensor. L3. Sum the initial weight coefficients of the fused data from each sensor and the dynamically adjusted weights of the fused data from each sensor to obtain the final fusion weight; L4. Based on the final fusion weight, the features from multiple sensors are weighted and fused to generate a global target feature vector.

[0009] Furthermore, the environmental parameters include visibility and precipitation intensity, and the performance score of each sensor is a quantitative score of the effectiveness of each sensor.

[0010] Furthermore, the target detection model is an improved CenterNet model.

[0011] Furthermore, the system also includes an initialization module, which is used to complete the calibration of sensor extrinsic parameters, store the installation position and attitude angle of each sensor, and pre-train the dual-channel attention network in the feature fusion algorithm for constructing the dual-channel attention network to optimize the network weights and biases.

[0012] Furthermore, the sensor extrinsic parameter calibration is performed by obtaining the sensor installation position and attitude parameters in advance through a checkerboard or calibration board.

[0013] Furthermore, the scoring function for quantifying the effectiveness of each sensor based on environmental parameters is as follows: Score = 100 - λ × Interference Intensity Where λ is the scene coefficient.

[0014] Furthermore, the system also includes a model iteration optimization module connected to an attention-based dynamic weight fusion module, used to fine-tune the parameters of the dual-channel attention network online based on actual detection results.

[0015] The present invention has the following positive effects: 1. This invention controls the spatiotemporal deviation of multi-sensor data to within 1ms and 5cm through GPS / IMU time calibration and extrinsic spatial conversion, providing high-precision basic data for fusion. The dynamic weight allocation based on the attention mechanism can adjust the contribution ratio of sensors in real time according to environmental conditions (such as prioritizing the use of 4D radar data in rainy or snowy weather), improving the target detection accuracy by 15%-30% (especially significant in foggy scenes with visibility <50m).

[0016] 2. This invention mines the correlation between multi-sensor data through feature interaction branches, achieving complementary advantages of "visual features supplementing details, laser point clouds supplementing structure, and radar data supplementing dynamics". It supports the rapid integration of new sensors (such as infrared cameras) and only requires expanding the feature extraction module and the input dimension of the attention network without reconstructing the overall framework. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system framework of the present invention; Figure 2 This is a flowchart illustrating the feature fusion algorithm for constructing a dual-channel attention network according to the present invention. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] Example 1: As Figure 1 As shown, a sensor data fusion system based on dynamic weight allocation using an attention mechanism is disclosed. The system includes: The multi-sensor spatiotemporal synchronization module includes a time synchronization unit and a spatial synchronization unit. The time synchronization unit is used to obtain a unified time reference from GPS / IMU and to interpolate and calibrate the sampling timestamps of each sensor to achieve data alignment in the time dimension. The spatial synchronization unit is used to convert LiDAR point cloud and 4D radar data to the camera coordinate system based on sensor extrinsic parameter calibration to achieve spatial dimension unification. The environmental perception and sensor performance evaluation module is connected to the multi-sensor spatiotemporal synchronization module. It is used to collect environmental parameters in real time, analyze visibility and precipitation intensity through camera images, determine the degree of snow particle interference through the reflectivity fluctuation of lidar point cloud, and quantify the effectiveness of each sensor based on environmental parameters. The feature extraction module, connected to the environmental perception and sensor performance evaluation module, is used to extract features from the synchronized sensor data. A dynamic weight fusion module based on an attention mechanism is connected to the feature extraction module and is used to fuse data from each sensor using a feature fusion algorithm that constructs a dual-channel attention network. The target detection module, connected to the attention-based dynamic weight fusion module, is used to input the fused global features into the target detection model and output target category, location, and velocity information.

[0020] In this embodiment, the feature extraction of the synchronized sensor data includes extracting the visual features of the target through a deep learning model and extracting the three-dimensional geometric features of the target through point cloud clustering and segmentation algorithms.

[0021] In this embodiment, the feature extraction of the synchronized sensor data also includes extracting the target's velocity features and radar cross-section features based on the synchronized 4D radar data.

[0022] In this embodiment, as Figure 2 As shown, the feature fusion algorithm that uses a dual-channel attention network to fuse data from various sensors includes: L1. Input the environmental parameters and the performance scores of each sensor into the environmental perception channel attention network to characterize the initial weight coefficients of the data fusion of each sensor and obtain the data information of the initial weight coefficients of the data fusion of each sensor. L2. Input the feature extraction data from each sensor into the feature interaction channel attention network, perform cross-attention calculation on the features of multiple sensors, explore the correlation between features, and output the dynamically adjusted weights for the fusion of each sensor. L3. Sum the initial weight coefficients of the fused data from each sensor and the dynamically adjusted weights of the fused data from each sensor to obtain the final fusion weight; L4. Based on the final fusion weight, the features from multiple sensors are weighted and fused to generate a global target feature vector.

[0023] In this embodiment, the environmental parameters include visibility and precipitation intensity, and the performance score of each sensor is a quantitative score of the effectiveness of each sensor.

[0024] In this embodiment, the target detection model is an improved CenterNet model.

[0025] In this embodiment, the system further includes an initialization module, which is used to complete the calibration of sensor extrinsic parameters, store the installation position and attitude angle of each sensor, and pre-train the dual-channel attention network in the feature fusion algorithm for constructing the dual-channel attention network to optimize the weights and biases of the network.

[0026] In this embodiment, the sensor extrinsic parameter calibration is performed by obtaining the sensor installation position and attitude parameters in advance through a checkerboard or calibration board.

[0027] In this embodiment, the scoring function for quantifying the effectiveness of each sensor based on environmental parameters is: Score = 100 - λ × Interference Intensity Where λ is the scene coefficient.

[0028] In this embodiment, the system further includes a model iteration optimization module connected to an attention-based dynamic weight fusion module, used to fine-tune the parameters of the dual-channel attention network online based on actual detection results.

[0029] Example 2: Based on the sensor data fusion system based on dynamic weight allocation of attention mechanism in Example 1, the present invention will be further described and explained below.

[0030] like Figure 1 As shown, a sensor data fusion system based on dynamic weight allocation using an attention mechanism is disclosed. The system includes: The multi-sensor spatiotemporal synchronization module includes a time synchronization unit and a spatial synchronization unit. The time synchronization unit is used to obtain a unified time reference from GPS / IMU and to interpolate and calibrate the sampling timestamps of each sensor to achieve data alignment in the time dimension. The spatial synchronization unit is used to convert LiDAR point cloud and 4D radar data to the camera coordinate system based on sensor extrinsic parameter calibration to achieve spatial dimension unification. The environmental perception and sensor performance evaluation module is connected to the multi-sensor spatiotemporal synchronization module. It is used to collect environmental parameters in real time, analyze visibility and precipitation intensity through camera images, determine the degree of snow particle interference through the reflectivity fluctuation of lidar point cloud, and quantify the effectiveness of each sensor based on environmental parameters. The feature extraction module, connected to the environmental perception and sensor performance evaluation module, is used to extract features from the synchronized sensor data. A dynamic weight fusion module based on an attention mechanism is connected to the feature extraction module and is used to fuse data from each sensor using a feature fusion algorithm that constructs a dual-channel attention network. The target detection module, connected to the attention-based dynamic weight fusion module, is used to input the fused global features into the target detection model and output target category, location, and velocity information.

[0031] In this embodiment, the multi-sensor spatiotemporal synchronization module Time synchronization: A unified time reference is obtained through GPS / IMU (Inertial Measurement Unit), and the sampling timestamps of each sensor are interpolated and calibrated to ensure that the data are aligned in the time dimension (error ≤ 1ms). Spatial synchronization: Based on sensor extrinsic parameter calibration (obtaining installation position and attitude parameters in advance through tools such as checkerboard and calibration board), the LiDAR point cloud and 4D radar data are converted to the camera coordinate system to achieve spatial dimension unification.

[0032] In this embodiment, the environmental perception and sensor performance evaluation module Real-time acquisition of environmental parameters: Visibility (such as fog concentration) and precipitation intensity (such as raindrop density) are analyzed through camera images, and the degree of snow particle interference is judged by the fluctuation of point cloud reflectivity of lidar. Sensor performance rating: The effectiveness of each sensor is quantitatively rated (0-100 points) based on environmental parameters. For example, the camera score drops to 30 points in rainy or foggy weather, while the 4D radar score remains at 80 points.

[0033] In this embodiment, the feature extraction module Feature extraction is performed on the synchronized sensor data: Camera: Extracts visual features (such as contours, colors, and textures) of the target through deep learning models (such as YOLO and FPN); LiDAR: Extracts the three-dimensional geometric features of a target (such as size, distance, and surface normals) through point cloud clustering and segmentation algorithms. 4D radar: Extracts the target's velocity characteristics and radar cross-section (RCS) characteristics.

[0034] In this embodiment, the dynamic weight fusion module based on the attention mechanism Constructing a dual-channel attention network: Environmental perception branch: Input environmental parameters (such as visibility, precipitation intensity) and sensor performance scores, and output initial weighting coefficients; Feature interaction branch: Perform cross-attention calculation on multi-sensor features (e.g., through a Transformer encoder) to explore the correlation between features and output dynamically adjusted weights; Fusion weight calculation: The initial weight and the dynamically adjusted weight are weighted and summed to obtain the final fusion weight (e.g., the weight of 4D radar is increased to 0.6 in rainy and foggy weather, while the weight of camera is reduced to 0.1). Feature fusion: The features from multiple sensors are weighted and fused according to the final weights to generate a global target feature vector.

[0035] 5. Target Detection Module The fused global features are input into the object detection model (such as the improved CenterNet) and output information such as object category, location, and velocity.

[0036] Technical Implementation Steps 1. System initialization: Complete the sensor extrinsic parameter calibration and store the installation position (x, y, z) and attitude angle (roll, pitch, yaw) of each sensor. Pre-trained attention network model (based on a dataset containing complex weather scenarios).

[0037] 2. Real-time data acquisition and preprocessing: Simultaneously collect raw data from 4D radar, lidar, and cameras, and record their respective timestamps; Noise reduction processing is performed on the raw data (such as noise reduction of LiDAR point clouds and rain / fog removal of camera images).

[0038] 3. Spatiotemporal synchronization: Time synchronization: Interpolate the data from each sensor based on GPS timestamps to unify them to the same time reference; Spatial synchronization: Based on external parameters, LiDAR and 4D radar data are converted to the camera coordinate system to generate spatiotemporally aligned multi-source data.

[0039] 4. Environmental parameters and sensor performance evaluation: Extract environmental features from camera images (such as fog concentration calculated using a dark channel prior algorithm); The performance of each sensor is scored based on environmental characteristics (e.g., the formula is: score = 100 - λ × interference intensity, where λ is the scene coefficient).

[0040] 5. Feature extraction and dynamic weight calculation: Extract the target features from each sensor separately; Environmental parameters, sensor scores, and features are input into the attention network, which outputs dynamic fusion weights.

[0041] 6. Feature fusion and target detection: Multi-sensor features are fused using dynamic weights to generate global features; Input the object detection model and output the final detection results (category, bounding box, speed, etc.).

[0042] 7. Model Iterative Optimization: Based on actual detection results (such as missed detections and false detections), the attention network parameters are fine-tuned online to improve the accuracy of weight allocation.

[0043] In summary, this invention not only improves the target detection accuracy by 15%-30%, but also reduces the spatiotemporal bias of multi-sensor data, providing a high-precision data foundation for fusion.

[0044] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A sensor data fusion system based on dynamic weight allocation using an attention mechanism, characterized in that, The system includes: The multi-sensor spatiotemporal synchronization module includes a time synchronization unit and a spatial synchronization unit. The time synchronization unit is used to obtain a unified time reference from GPS / IMU and to interpolate and calibrate the sampling timestamps of each sensor to achieve data alignment in the time dimension. The spatial synchronization unit is used to convert LiDAR point cloud and 4D radar data to the camera coordinate system based on sensor extrinsic parameter calibration to achieve spatial dimension unification. The environmental perception and sensor performance evaluation module is connected to the multi-sensor spatiotemporal synchronization module. It is used to collect environmental parameters in real time, analyze visibility and precipitation intensity through camera images, determine the degree of snow particle interference through the reflectivity fluctuation of lidar point cloud, and quantify the effectiveness of each sensor based on environmental parameters. The feature extraction module, connected to the environmental perception and sensor performance evaluation module, is used to extract features from the synchronized sensor data. The dynamic weight fusion module based on the attention mechanism is connected to the feature extraction module and is used to fuse the data from each sensor using a feature fusion algorithm that constructs a dual-channel attention network. The target detection module, connected to the attention-based dynamic weight fusion module, is used to input the fused global features into the target detection model and output target category, location, and velocity information.

2. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 1, characterized in that: The feature extraction of the synchronized sensor data includes extracting the visual features of the target through a deep learning model and extracting the three-dimensional geometric features of the target through point cloud clustering and segmentation algorithms.

3. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 2, characterized in that: The feature extraction of the synchronized sensor data also includes extracting the target's velocity features and radar cross-section features based on the synchronized 4D radar data.

4. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 1, characterized in that, The feature fusion algorithm that uses a dual-channel attention network to fuse data from various sensors includes: L1. Input the environmental parameters and the performance scores of each sensor into the environmental perception channel attention network to characterize the initial weight coefficients of the data fusion of each sensor and obtain the data information of the initial weight coefficients of the data fusion of each sensor. L2. Input the feature extraction data from each sensor into the feature interaction channel attention network, perform cross-attention calculation on the features of multiple sensors, explore the correlation between features, and output the dynamically adjusted weights for the fusion of each sensor. L3. Sum the initial weight coefficients of the fused data from each sensor and the dynamically adjusted weights of the fused data from each sensor to obtain the final fusion weight; L4. Based on the final fusion weight, the features from multiple sensors are weighted and fused to generate a global target feature vector.

5. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 4, characterized in that: The environmental parameters include visibility and precipitation intensity, and the performance score of each sensor is a quantitative score of the effectiveness of each sensor.

6. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 1, characterized in that: The target detection model is an improved CenterNet model.

7. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 1, characterized in that, The system also includes an initialization module, which is used to complete the calibration of sensor extrinsic parameters, store the installation position and attitude angle of each sensor, and pre-train the dual-channel attention network in the feature fusion algorithm for constructing the dual-channel attention network to optimize the weights and biases of the network.

8. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 7, characterized in that: The sensor extrinsic parameter calibration is performed by obtaining the sensor installation position and attitude parameters in advance through a checkerboard or calibration board.

9. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 1, characterized in that: The scoring function for quantifying the effectiveness of each sensor based on environmental parameters is as follows: Score = 100 - λ × Interference Intensity Where λ is the scene coefficient.

10. The sensor data fusion system based on dynamic weight allocation using an attention mechanism according to claim 1, characterized in that, The system also includes a model iteration optimization module, which is connected to a dynamic weight fusion module based on an attention mechanism, for fine-tuning the parameters of the dual-channel attention network online based on actual detection results.

Images