Multi-sensor fusion processing method, sensing method and equipment based on LiDAR point cloud pseudo image conversion

Abstract

The invention discloses a multi-sensor fusion processing method, perception method and equipment based on LiDAR point cloud pseudo image conversion, a multi-view projection strategy is adopted to convert a preprocessed LiDAR point cloud into an aerial view, a front view and a side view, the aerial view, the front view and the side view are subjected to feature coding and then fused through a SENet attention mechanism to generate a multi-view fusion pseudo image, and a large amount of space information is reserved. Meanwhile, in order to effectively improve the multi-sensor feature fusion efficiency, after a visual image is preprocessed, an improved CNN network is adopted to extract the features of a LiDAR pseudo image and the visual image, feature fusion is achieved through a cross-modal attention mechanism, attention weights are generated by calculating a similarity matrix, and the feature fusion efficiency is improved. And after the features are enhanced, the features are fused in a channel splicing and element-level adding mode. The method can effectively improve the precision and robustness of automatic driving environment perception, enhances the performance in a complex scene, and is suitable for tasks such as target detection and semantic segmentation in automatic driving.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology and involves interdisciplinary technologies such as multi-sensor data fusion, 3D point cloud processing, computer vision and deep learning. It focuses on the visual representation of LiDAR point cloud data and multi-sensor collaborative perception, and is applicable to key tasks such as target detection, semantic segmentation and path planning in autonomous driving environments. It can be widely used in autonomous driving platforms such as passenger cars and commercial vehicles. Background Technology

[0002] In the environmental perception layer of autonomous driving systems, a single sensor is insufficient to handle complex and ever-changing real-world scenarios. LiDAR, as a core sensor providing three-dimensional spatial information, can accurately measure target distances, generating point cloud data with centimeter-level ranging accuracy, and is unaffected by lighting conditions, exhibiting stable performance in scenarios such as darkness and tunnels. However, LiDAR point cloud data suffers from issues such as sparsity, non-uniform distribution, and large data volume. For example, a 128-line LiDAR at a 10Hz acquisition frequency produces approximately 2 million points per frame, placing extremely high demands on hardware computing power for direct real-time processing.

[0003] Currently, mainstream point cloud processing methods mainly fall into two categories: one is direct processing based on point clouds, such as the PointNet series models which process unordered point clouds using symmetric functions, but this has high computational complexity and is difficult to meet the real-time requirements of autonomous driving; the other is to convert point clouds into pseudo-images and process them using mature 2D CNN architectures. However, existing pseudo-image conversion methods have shortcomings. Single-view projection loses a lot of spatial information, bird's-eye view is not accurate enough in processing low obstacles, and front view has defects in representing distant targets; moreover, the feature alignment accuracy between pseudo-images and visual images is low, the fusion process lacks an adaptive adjustment mechanism, and the fusion effect drops significantly in adverse weather or complex traffic scenarios.

[0004] Visual sensors can provide rich texture, color, and semantic information, and CNN-based image processing technology has achieved remarkable results in areas such as object detection and semantic segmentation. However, visual sensors are greatly affected by lighting and weather conditions. In environments such as strong light, backlight, rain, and fog, they are prone to overexposure and reduced contrast, leading to decreased perception accuracy. Therefore, achieving efficient fusion of LiDAR point clouds and visual images is crucial to improving the perception performance of autonomous driving systems.

[0005] In existing multi-sensor fusion methods, early fusion strategies focused on the data layer or decision layer. Data layer fusion suffers from data heterogeneity issues, while decision layer fusion struggles to fully utilize underlying feature information. Although feature layer fusion can achieve deep information integration, existing methods have shortcomings in feature alignment and modal weight allocation, resulting in limited fused feature representation capabilities. Therefore, there is an urgent need for a method that can effectively preserve LiDAR point cloud spatial information and achieve efficient cross-modal feature fusion to improve the perception accuracy and robustness of autonomous driving systems. Summary of the Invention

[0006] This invention aims to address the problems of severe spatial information loss, low efficiency of multi-sensor feature fusion, and insufficient robustness of perception systems in the conversion of LiDAR point cloud pseudo-images in existing technologies. It provides a multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion. This method enhances the representational ability of pseudo-images through multi-view projection and attention mechanisms, and employs an improved CNN network and cross-modal attention mechanism to achieve efficient fusion of LiDAR pseudo-image features and visual image features, thereby improving the environmental perception accuracy and real-time performance of autonomous driving systems in complex scenarios.

[0007] The technical solution adopted in this invention is as follows:

[0008] Step 1: LiDAR point cloud data preprocessing

[0009] 1.1 Noise Reduction Processing

[0010] The statistical filtering algorithm is used to denoise the original LiDAR point cloud. The specific steps are as follows:

[0011] (1) Calculate the average distance between each point and all points in its neighborhood. The number of neighborhood points is dynamically adjusted according to the point cloud density. For high-density point clouds, the number of neighborhood points is set to 50; for low-density point clouds, the number of neighborhood points is set to 30.

[0012] (2) Calculate the standard deviation of the average distance of all points, and identify and remove points whose average distance is greater than (mean + 1.5 × standard deviation) as noise points, retaining the valid point cloud data. This method can remove more than 95% of noise points while retaining more than 98% of valid points.

[0013] 1.2 Point Cloud Registration

[0014] The spatiotemporal synchronization and coordinate transformation of point clouds are achieved based on IMU and GPS data. The specific process is as follows:

[0015] (1) Using the angular velocity and acceleration data of the IMU, the motion state of the vehicle is predicted by the Kalman filter algorithm;

[0016] (2) Correct the prediction results by combining GPS positioning data to obtain the vehicle's precise position and attitude information;

[0017] (3) Based on the vehicle position and attitude, the LiDAR point clouds at different times are unified to the world coordinate system, and the registration error is controlled within 0.1m.

[0018] 1.3 Downsampling Processing

[0019] Voxel mesh downsampling is used to reduce the number of point clouds. Specific parameter settings are as follows:

[0020] (1) The voxel grid size is dynamically adjusted according to the point cloud density. For 128-line LiDAR, the voxel size is set to 0.1m×0.1m×0.1m; for 64-line LiDAR, the voxel size is set to 0.2m×0.2m×0.2m.

[0021] (2) For each voxel grid, calculate the centroid coordinates of all points within the grid, and use the centroid point to represent the voxel to achieve point cloud downsampling. After downsampling, the number of points in the point cloud can be reduced by 60%-70%, while retaining key feature points.

[0022] Step 2: LiDAR point cloud pseudo-image generation

[0023] 2.1 Multi-view projection

[0024] 2.1.1 Bird's-eye view BEV projection

[0025] (1) Projection range: Taking the vehicle as the origin, a rectangular area of ​​120m×100m is formed with 100m in front, 20m behind, and 50m to the left and right.

[0026] (2) Projection method: Project the 3D point cloud (x,y,z) onto the 2D plane (x,y), and use the z coordinate for subsequent feature encoding;

[0027] (3) Resolution: 512×512 pixels, each pixel corresponds to an actual distance of 0.234m×0.195m.

[0028] 2.1.2 Front View FV Projection

[0029] (1) Projection range: The vertical pitch angle is -5° to 20°, covering the ground to the sky, and the horizontal azimuth angle is -90° to 90°, achieving 180° horizontal scanning;

[0030] (2) Projection method: Convert the 3D point cloud (x,y,z) into polar coordinates. Where r is the distance and θ is the azimuth angle. The pitch angle is projected onto... flat;

[0031] (3) Resolution: 256×512 pixels, with each pixel corresponding to 0.351° in the azimuth direction and 0.098° in the pitch direction.

[0032] 2.1.3 Side View SV Projection

[0033] (1) Projection range: 50m in front, 50m behind, and 30m above and below, forming a rectangular area of ​​100m×60m;

[0034] (2) Projection method: Project the three-dimensional point cloud (x,y,z) onto the (y,z) plane;

[0035] (3) Resolution: 256×512 pixels, each pixel corresponds to an actual distance of 0.391m×0.117m.

[0036] 2.2 Feature Encoding

[0037] 2.2.1 Distance Information Encoding

[0038] The distance r from the point cloud to the vehicle is normalized using the following formula:

[0039] I r =225×(rr) min ) / (r max -r min (1)

[0040] Where, r min For a minimum distance of 0m, r max For a maximum distance of 100m, I r It is the normalized gray value, with a value range of 0-255.

[0041] 2.2.2 Reflection Intensity Coding

[0042] The reflection intensity i of LiDAR point clouds is typically in the range of 0-255. The reflection intensity is weighted and adjusted according to the reflection characteristics of different materials.

[0043] I i =i×ω (2)

[0044] Among them, I i The reflection intensity is the weighted adjustment value, where ω is the weighting coefficient. For metallic targets (such as vehicles), ω = 1.2; for non-metallic targets (such as pedestrians and roads), ω = 0.8.

[0045] 2.2.3 Altitude Information Encoding

[0046] The z-coordinate of the point cloud is normalized using the following formula:

[0047] Iz =225×(zz) min ) / (z max -z min (3)

[0048] Among them, z min Minimum height -2m, z max With a maximum height of 5m, I z The z-coordinates of the point cloud are normalized and used to supplement the height information of the bird's-eye view.

[0049] 2.2.4 Multi-feature fusion coding

[0050] The pseudo-images for each viewpoint employ a three-channel design. The three channels for the bird's-eye view are distance grayscale value, reflection intensity weighted value, and height normalized value, respectively. The three channels for the front view and side view are distance grayscale value, reflection intensity weighted value, and point cloud density, respectively.

[0051] 2.3 Multi-view Fusion

[0052] The SENet attention mechanism is used to fuse pseudo-images from three perspectives. The specific steps are as follows:

[0053] (1) Reduce the dimensionality of the pseudo images from the three perspectives to the same number of channels, 64, through a convolutional layer;

[0054] (2) Perform global average pooling on the feature map of each viewpoint to obtain a 1×1×64 feature vector;

[0055] (3) Calculate the attention weights (α1, α2, α3) of the three views through the fully connected layer so that α1+α2+α3=1, where α1 represents the bird's-eye view weight, α2 represents the front view weight, and α3 represents the side view weight.

[0056] (4) The feature maps of the three perspectives are weighted and summed according to the weights to generate a multi-view fused pseudo-image;

[0057] (5) Scene adaptive adjustment: In the highway scene, the weight of the bird's-eye view α1 is increased to 0.4-0.5; in the urban road scene, the weight of the front view α2 is increased to 0.4-0.5; in the narrow road scene, the weight of the side view α3 is increased to 0.3-0.4.

[0058] Step 3: Visual Image Preprocessing

[0059] 3.1 Distortion Correction

[0060] The camera intrinsic parameter matrix K and distortion coefficients D are obtained through Zhang Zhengyou's calibration method, and then distortion correction is achieved using the undistort function of the OpenCV library to eliminate the influence of lens distortion on the image.

[0061]

[0062] D=[k1,k2,p1,p2,k3] (5)

[0063] Among them, f x ,f y c is the focal length. x ,c y The coordinates of the main point are k1, k2, and k3; k1, k2, and k3 are the radial distortion coefficients, and p1 and p2 are the tangential distortion coefficients.

[0064] 3.2 Size Adjustment

[0065] The corrected image size was adjusted to 512×256 pixels to match the size of the LiDAR pseudo-image, which facilitates subsequent feature fusion.

[0066] 3.3 Light Enhancement

[0067] An adaptive histogram equalization (CLAHE) algorithm is used to enhance image contrast. This method can enhance image details while avoiding overexposure, and can increase the image information entropy by 15%-20% in low-light scenes. Specific parameters are as follows:

[0068] (1) Block size: 8×8 pixels;

[0069] (2) Contrast Limit: 2.0;

[0070] (3) Interpolation method: bilinear interpolation.

[0071] Step 4: Feature Extraction and Fusion

[0072] 4.1 Improved CNN Feature Extraction

[0073] Based on the ResNet50 architecture, the following improvements were made:

[0074] (1) Add a channel attention module (CAM) and a spatial attention module (SAM) after the convolutional layer of each residual block;

[0075] (2) The CAM module generates channel attention weights by performing global average pooling and fully connected layer processing on the channel dimension of the feature map, thereby enhancing the feature representation of key channels;

[0076] (3) The SAM module performs max pooling and average pooling on the spatial dimension of the feature map and generates spatial attention weights by combining convolutional layers to highlight the features of the target region.

[0077] (4) The network output feature map has a length, width, and number of channels of 32×16×1024, which correspond to the features of LiDAR pseudo image and visual image, respectively.

[0078] 4.2 Feature Alignment

[0079] Based on the rotation matrix R and translation vector T of the camera and LiDAR, a mapping relationship between pixel coordinates and point cloud coordinates is established; then, an affine transformation is performed on the LiDAR pseudo-image features to make them consistent with the visual image features in spatial position, with the alignment error controlled within 1-2 pixels, thus achieving spatial alignment between the LiDAR pseudo-image features and the visual image features.

[0080] Feature alignment is a prerequisite for the effective fusion of LiDAR pseudo-image features and visual image features. The core is to solve the problem of "misalignment" in spatial location between the two types of sensor data. The specific process is as follows:

[0081] (1) Establish coordinate mapping relationship

[0082] Based on the rotation matrix R and translation vector T of the camera and LiDAR, a mapping relationship between pixel coordinates and point cloud coordinates can be established:

[0083] P new =R×P LiDAR +T (6)

[0084] P′=P new ×K (7)

[0085] Among them, P LiDAR P represents the 3D coordinates (x, y, z) of a LiDAR point cloud. new It is a 3D coordinate transformation to the camera coordinate system. The rotation matrix R can be used to correct the attitude difference between the two sensors (such as the installation angle deviation), and the translation vector T can be used to compensate for the position offset (such as the spatial distance of the installation position). Then, it is further transformed into image pixel coordinates P′(u,v) through the camera intrinsic parameter matrix K, thus realizing the complete mapping from LiDAR point cloud coordinates to camera coordinates and then to image pixel coordinates.

[0086] (2) Affine transformation correction

[0087] Because the projection rules of LiDAR pseudo-images differ from those of visual images, even after coordinate mapping, the feature maps of the two may still have slight spatial offsets. Therefore, it is necessary to perform an affine transformation on the LiDAR pseudo-image features according to the established mapping relationship, fine-tuning the position of each pixel in the feature map, and ultimately aligning the two features in space.

[0088] (3) Precision control

[0089] Excessive alignment error can lead to disordered feature information in subsequent fusion. Therefore, the alignment error needs to be strictly controlled within 2 pixels. This high-precision alignment can ensure the effectiveness of cross-modal feature fusion.

[0090] 4.3 Cross-modal attention fusion

[0091] Cross-modal attention mechanism is the core of the fusion process. Its purpose is to enable the model to select the more reliable and relevant information from the two features, achieving a fusion effect of 1+1>2. The specific steps are as follows:

[0092] (1) The LiDAR pseudo-image features F l and visual image features F v Mapping to the same dimension through convolutional layers:

[0093] F l ′=Conv(F l (8)

[0094] F v ′=Conv(F v (9)

[0095] Among them, F l F' is a LiDAR pseudo-image feature. l F is a dimensionally unified LiDAR pseudo-image feature obtained through mapping by convolutional layers. v ′ represents visual image feature F v Visual image features with unified dimensions obtained through mapping via convolutional layers;

[0096] (2) Calculate the similarity matrix:

[0097]

[0098] Where d is the number of feature channels. As a normalization factor, to avoid numerical inflation due to excessive dimensionality, and to ensure the comparability of similarities at different positions, S(i,j) represents F. l The i-th spatial position of ′ and F v The similarity of the j-th spatial location is such that the higher the similarity, the greater the likelihood that the two describe the same target or scene.

[0099] (3) Perform row and column normalization on the similarity matrix S(i,j) to obtain the LiDAR attention weights A for vision. lv And visual attention weights A for LiDAR vl A lv (i,j) represents F l The i-th position of F v The level of attention at position j, A vl (j,i) represents F v The j-th position of F l The level of attention given to the i-th position;

[0100] (4) Generate attention enhancement features, including LiDAR attention enhancement feature F l "and visual attention enhancement features F v ":

[0101] F l " = F l +Proj(A vl ×F v ′) (11)

[0102] F v " = F v +Proj(A lv ×F l ′) (12)

[0103] The dimensional projection function Proj() maps the product of attention weights and features to the same dimension as the original features, ensuring that the information after cross-modal attention weighting is accurately integrated into the original features, achieving effective complementary enhancement. l "Using the portion A of visual features that is related to LiDAR features" lv ×F v Enhance LiDAR features to compensate for the lack of LiDAR texture information, F v "Using the visually relevant portion A of LiDAR features" vl ×F l Enhance visual features to compensate for the lack of robustness of vision in adverse weather conditions;

[0104] (5) Enhance the feature F l "and F v "By concatenating and adding elements, the final fused feature is obtained, thereby achieving cross-modal enhanced feature fusion:"

[0105] F2 = Concat(F l ",F v ")+ElementSum(F l ",F v (13)

[0106] In the formula, Concat() represents the concatenation operation, which preserves the spatial geometric features of LiDAR and the semantic texture features of vision; ElementSum() represents the element-wise addition of the enhanced features of the two modalities to strengthen the consensus features of the two modalities.

[0107] Step 5: Perception Task Processing

[0108] 5.1 Target Detection

[0109] The YOLOv5 model is used as the basic detection framework, and the fused features F2 are input into the neck and head networks of YOLOv5. The neck network adopts a PANet structure, achieving multi-scale feature fusion through upsampling and downsampling. The head network includes three detection heads, corresponding to large, medium, and small target sizes, respectively, to meet the detection needs of different-sized targets such as vehicles, pedestrians, and cyclists in autonomous driving scenarios, achieving multi-scale target detection. The loss function adopts CIoU loss, combined with class loss and confidence loss, to improve detection accuracy.

[0110] 5.2 Semantic Segmentation

[0111] Semantic segmentation is performed using the DeepLabv3+ model, with the fused features F2 input to the ASPP module. The ASPP module obtains multi-scale contextual information through dilated convolutions with different dilation rates, and uses skip connections to fuse low-level and high-level features to improve the accuracy of segmentation boundaries. The loss function uses cross-entropy loss combined with Dice loss to address the class imbalance problem.

[0112] The beneficial effects of this invention are:

[0113] (1) Existing pseudo-image conversion methods suffer from severe spatial information loss: single-view bird's-eye view is insufficient in representing the height information of low obstacles such as manhole covers and stone blocks, leading to a decrease in detection accuracy; the front view has a large error in the spatial positioning of distant targets, with a position deviation usually exceeding 1m; and the side view is difficult to cover a wide field of view in front, resulting in insufficient information integrity. This invention proposes a multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion. By fusing multi-view projection with the SENet attention mechanism to generate pseudo-images, it solves the problem of severe spatial information loss in LiDAR point cloud pseudo-image conversion, retaining more than 90% of the original point cloud spatial information, and can also dynamically adjust the view weight according to the scene. For example, in highway scenes, increasing the bird's-eye view weight to 0.5 can enhance the spatial positioning of distant lane lines and vehicles; in urban road scenes, increasing the front view weight to 0.5 can optimize the detail capture of close targets such as pedestrians and traffic lights; in narrow road scenes, increasing the side view weight to 0.4 can enhance the distance perception of obstacles on both sides. This adaptive mechanism can overcome the scene limitations of single-view methods and provide more comprehensive and accurate spatial basis data for subsequent fusion.

[0114] (2) Existing multi-sensor fusion methods have two major bottlenecks: low feature alignment accuracy and rigid weight allocation. Due to sensor installation deviations and differences in projection rules, the spatial misalignment between LiDAR and visual features is usually 3-5 pixels, which leads to semantic conflicts after fusion and problems such as mismatch between the vehicle position detected by LiDAR and the vehicle outline in the visual image. Using a fixed ratio to fuse features cannot cope with complex scenes and fully utilize the advantages of each feature, resulting in limited expressive power of the fused features. This invention employs an improved CNN network to extract features during multi-sensor fusion, combined with a cross-modal attention mechanism. This controls the feature alignment error to within 1-2 pixels, significantly lower than the 3-5 pixels of existing methods. Through coordinate mapping and affine transformation, it ensures a strict match between LiDAR spatial coordinates and visual pixel positions. Furthermore, based on a similarity matrix, the correlation between LiDAR and visual features is calculated, generating dynamic attention weights to enhance more reliable modal information. For example, in rainy conditions, the LiDAR feature weight is increased to 0.6, while in bright light conditions, the visual feature weight is increased to 0.7. This enhances the precise spatial information of LiDAR and compensates for the insufficient robustness of visual information in adverse weather conditions. The fused features possess both the centimeter-level spatial accuracy of LiDAR and the rich textural semantics of vision, exhibiting higher perceptual accuracy in complex scenes. This fusion method improves feature representation capability by more than 40% compared to existing methods, providing higher-quality input features for perception tasks.

[0115] (3) This invention demonstrates significant advantages in core perception tasks such as target detection and semantic segmentation: it not only greatly improves detection accuracy and recall, but also increases the detection recall to 85%, far exceeding the 60% of existing methods, especially for traditional challenges such as distant vehicles and low obstacles. The average accuracy (mAP) of target detection is 15%-20% higher than existing technologies. At the same time, it effectively enhances the stability of the perception system in harsh environments such as rain, fog, and strong light. The decrease in semantic segmentation accuracy in rain and fog scenes is controlled within 10%, and the target false detection rate is reduced to below 5% in strong light environments, solving the problem of traditional vision sensors being prone to failure in such scenarios. In terms of engineering practicality, this invention controls the processing time of a single frame to within 80ms through point cloud downsampling and improved lightweight design of CNN, fully meeting the real-time requirements of 10Hz for autonomous driving. Moreover, the model size is adapted to automotive-grade chip deployment, overcoming the shortcomings of existing methods such as high computational complexity and difficulty in engineering implementation. It provides reliable technical support for autonomous driving environmental perception with high accuracy, strong robustness, and real-time performance. Attached Figure Description

[0116] Figure 1 The overall flowchart of the method of this invention. Detailed Implementation

[0117] The invention will now be further described with reference to the accompanying drawings.

[0118] The multi-sensor fusion processing method and sensing method based on LiDAR point cloud pseudo-image conversion of this invention are as follows: Figure 1 As shown, pseudo-images are generated through multi-view projection and fused with visual images to improve the accuracy of environmental perception in autonomous driving. The specific implementation steps are as follows:

[0119] (1) Data preparation: The datasets used in this invention are the KITTI 3D object detection dataset and the nuScenes full-scene dataset. The KITTI dataset contains 7481 frames of training data and 7518 frames of test data. Each frame contains a 64-line LiDAR point cloud and a 2048×1024 resolution stereo camera image. The labeled categories include vehicles, pedestrians, cyclists, etc. The nuScenes dataset covers 1000 scenes, including a 128-line LiDAR point cloud and 1600×900 resolution images from 6 cameras. It is labeled with 23 object categories and semantic segmentation labels.

[0120] (2) Data preprocessing: Time calibration was performed on all data to ensure that the timestamp deviation of data from different sources did not exceed 10 milliseconds; at the same time, external parameter calibration between devices was completed to control the rotation angle error within 0.5 degrees and the position offset error within 5 centimeters. The processed data was divided into three parts in a ratio of 7:1.5:1.5:3.

[0121] (3) LiDAR Point Cloud Preprocessing Module Construction: 1. Statistical filtering is used to denoise LiDAR point clouds of different densities. For high-density point clouds such as 128 lines, 50 neighboring points around each point are selected for analysis; for low-density point clouds such as 16 lines, 30 neighboring points are selected. By calculating the average distance between each point and its neighboring points, outliers with an average distance exceeding "mean plus 1.5 times the standard deviation" are removed. The retention rate of effective points after processing can reach more than 98%. 2. Point cloud registration. Combining IMU data sampled 100 times per second and GPS data with a positioning accuracy of 1 meter, the extended Kalman filter method is used to estimate the vehicle's position and attitude. On this basis, the point clouds collected at different times are uniformly converted to the world coordinate system to ensure that the position deviation after registration is controlled within 0.1 meters. 3. Downsampling processing. To simplify the LiDAR, different voxel grid sizes are used depending on the line count: a 0.1m × 0.1m × 0.1m cubic grid is used for a 128-line LiDAR, while a 0.2m × 0.2m × 0.2m grid is used for a 64-line LiDAR. Each grid is represented by its centroid point. After processing, the number of point clouds can be reduced by 60%-70%, while the retention rate of key features can be maintained at over 95%.

[0122] (4) Building a LiDAR point cloud pseudo-image conversion module:

[0123] 1. Multi-view projection. The bird's-eye view projection range is 100m in front, 20m behind, and 50m to the left and right, with a resolution of 512×512 pixels, and each pixel corresponds to an actual distance of 0.234m×0.195m; the front view has a pitch angle of -5° to 20° and an azimuth angle of -90° to 90°, with a polar coordinate projection of 256×512 pixels, and each azimuth angle corresponds to 0.351°; the side view projection range is 50m in front, 50m behind, and 30m up and down, with a projection of 256×512 pixels, and each pixel corresponds to 0.391m×0.117m.

[0124] 2. Feature encoding.

[0125] Distance encoding:

[0126] I r =225×(rr) min ) / (r max -r min (14)

[0127] Where r is the distance from the point cloud to the vehicle, and the minimum distance r min The maximum distance is 0m, r max For 100m, I r It is the normalized gray value, with a value range of 0-255.

[0128] Reflection intensity encoding:

[0129] I i =i×ω (15)

[0130] Where i is the reflection intensity of the LiDAR point cloud, the weighting coefficient ω = 1.2 for metallic targets, and the weighting coefficient ω = 0.8 for non-metallic targets.

[0131] Height encoding:

[0132] I z =225×(zz) min ) / (z max -z min (16)

[0133] Where z is the z-coordinate of the point cloud, z min Minimum height -2m, z max With a maximum height of 5m, I z Used to supplement the height information of the bird's-eye view. The feature combination for the BEV view is (distance, reflectance intensity, height), while the feature combination for the FV and SV views is (distance, reflectance intensity, point cloud density).

[0134] 3. Multi-view fusion. The pseudo-images from each viewpoint are reduced to 64 channels via 1×1 convolution. The feature maps from each viewpoint are then subjected to global average pooling and attention weights (α1, α2, α3) are calculated through a fully connected layer, ensuring α1 + α2 + α3 = 1. Weighted fusion formula:

[0135] F1=α1×F BEV +α2×F FV +α3×F SV (17)

[0136] Where F1 is the final feature obtained after weighted fusion of pseudo-image features from different perspectives, F BEV For pseudo-image features from a bird's-eye view perspective, F FV For pseudo-image features of the front view perspective, F SV The pseudo-image features are for side view perspectives. The weight of the bird's-eye view for highway scenes is α1 = 0.5, the weight of the front view for urban road scenes is α2 = 0.5, and the weight of the side view for narrow road scenes is α3 = 0.4.

[0137] (5) Build a visual image preprocessing module:

[0138] 1. Distortion Correction. The intrinsic parameter matrix K and distortion coefficients D are obtained using the Zhang Zhengyou calibration method.

[0139]

[0140] D = [k1,k2,p1,p2,k3] (19)

[0141] Among them, f x ,f y Let c be the focal length of the camera in the x and y directions. x ,c y Here, k1, k2, and k3 are the principal point coordinates on the image plane; k1, k2, and k3 are the radial distortion coefficients, and p1 and p2 are the tangential distortion coefficients. Distortion correction is achieved using the `undistort` function from the OpenCV library.

[0142] r 2 =x 2 +y 2 (20)

[0143] x′=x×(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 ) (twenty one)

[0144] y′=y×(1+k1r 2 +k2r 4 +k3r6 )+2p2xy+p1(r 2 +2y 2 ) (twenty two)

[0145] Where x, y are the original coordinates of the image pixels, r is the radial distance from the point to the principal point, reflecting the effect of the distance from the pixel to the principal point on distortion, and x′, y′ are the coordinates of the image pixels after distortion correction. Through these correction terms, the distorted pixel coordinates are restored to the ideal distortion-free position, thus achieving image distortion correction.

[0146] 2. Resizing. Use bilinear interpolation to resize the image to 512×256 pixels to match the size of the pseudo image.

[0147] 3. Illumination Enhancement. Set the CLAHE algorithm parameters to block size 8×8 and contrast limit 2.0 to increase image information entropy by 15%-20% in low-light scenes.

[0148] (6) Construct a feature extraction and fusion module:

[0149] 1. Construct an improved CNN network. Based on the ResNet50 architecture, add a Channel Attention (CAM) module and a Spatial Attention (SAM) module to each residual block. The CAM module generates channel attention weights by performing global average pooling and fully connected layers on the channel dimension of the feature map, thereby enhancing the feature representation of key channels. The SAM module generates spatial attention weights by performing max pooling and average pooling on the spatial dimension of the feature map, combined with convolutional layers, to highlight the features of the target region. Finally, the output feature map has a feature size of 32×16×1024, corresponding to the features of the LiDAR pseudo image and the visual image, respectively.

[0150] 2. Feature Alignment. First, establish the mapping relationship between pixel coordinates and point cloud coordinates:

[0151] P new =R×P LiDAR +T (23)

[0152] P′=P new ×K (24)

[0153] Where R is the rotation matrix, T is the translation vector, and P LiDAR P represents the 3D coordinates (x, y, z) of a LiDAR point cloud. new It transforms the coordinates to the camera coordinate system into three-dimensional coordinates, and then further transforms them into image pixel coordinates (u,v) through the camera intrinsic parameter matrix K, realizing a complete mapping from LiDAR point cloud coordinates to camera coordinates and then to image pixel coordinates. Finally, it fine-tunes the spatial position through affine transformation to make the alignment error less than 2 pixels.

[0154] 3. Cross-modal attention mechanism. First, the LiDAR pseudo-image features F... l and visual image features F v Mapping to the same dimension through convolutional layers:

[0155] F l ′=Conv(F l (25)

[0156] F v ′=Conv(F v (26)

[0157] At this time, F l ′ and F v The number of channels for each feature is 512. Next, the degree of correlation between each spatial location in the two features is measured using the inner product operation of the similarity matrix:

[0158]

[0159] Where the number of feature channels d = 512, and S(i,j) represents F l The i-th spatial position of ′ and F v The similarity of the j-th spatial location of ′ is such that the higher the value, the stronger the correlation between the two in describing the same target.

[0160] The similarity matrix is ​​transposed and normalized to obtain the bidirectional attention weights:

[0161] A lv =Softmax(S T (28)

[0162] A vl =Softmax(S) (29)

[0163] In formulas 28 and 29, S is the same as S(i,j) in formula 27, and A lv A represents the attention weight of LiDAR features on visual features. vl This represents the attention weight of visual features on LiDAR features.

[0164] Next, attention weights are used to achieve cross-modal information complementarity, as shown in the formula:

[0165] F l " = F l +Proj(A vl ×F v ′) (30)

[0166] F v " = F v +Proj(A lv ×Fl ′) (31) F l "and F v "These are the enhanced LiDAR features and visual features, respectively. The visual features supplement the texture information defects of LiDAR, while the LiDAR features enhance the robustness of vision in harsh environments. The dimensional projection function Proj() maps the 512-dimensional supplementary features to the same dimensions as the original features.

[0167] Finally, F l "and F v "The concatenated features are split into 1024 channels, preserving all details of both features. The concatenated features are then added element-by-element to enhance consistent information between the two features, resulting in the final fused feature."

[0168] F2 = Concat(F l ",F v ")+ElementSum(F l ",F v (32)

[0169] First, Concat(,) is used to stitch the channels together, preserving the spatial geometric features of LiDAR and the semantic texture features of vision. Then, ElementSum(,) is used to perform channel-by-channel addition on the enhanced features of the two modalities, strengthening the consensus features of the two modalities.

[0170] (7) Build a perception task processing module:

[0171] 1. Object Detection Module. An object detection network is built based on the YOLOv5 framework. The neck section uses a PANet structure, fusing high-resolution shallow features and low-resolution deep features through bottom-up feature pyramids and top-down path aggregation to ensure effective recognition of objects at different scales. The head section has three detection heads, corresponding to large, medium, and small object sizes. The small-scale detection head targets small, near-distance objects such as cyclists, using high-resolution features to capture detailed contours. The medium-scale detection head targets medium-sized objects such as cars, balancing detail and semantic information. The large-scale detection head targets large, distant objects such as buses, improving recognition stability through deep semantic features. The loss function L consists of bounding box loss, class loss, and confidence loss.

[0172] L = L CIoU +L cls +L conf (33)

[0173] Bounding box loss L CIoUTaking into account bounding box overlap, center point distance, and aspect ratio, this method improves localization accuracy compared to traditional IoU loss, and is particularly suitable for bounding box convergence of occluded targets. Class Loss L cls Cross-entropy loss is used to penalize misclassification of the target class, ensuring the accuracy of class identification. Confidence loss L conf Cross-entropy loss is used to measure the confidence that the predicted bounding box contains the target, thus suppressing false detections in the background region and false negatives in the target region.

[0174] 2. Semantic Segmentation Module. Based on the DeepLabv3+ framework, semantic segmentation of road scenes is implemented, focusing on optimizing the segmentation effect of key regions such as roads, lane lines, and traffic signs. Dilated convolutions with dilation rates of 1, 6 / 12, and 18 are configured in the ASPP module to capture local details such as lane line edges, medium-range context such as road areas, and global scene information such as intersection layout information, respectively. The high-level semantic features output by ASPP are fused with the low-level detail features output by the encoder, and upsampling is used to restore the image to its input size, improving the clarity of segmentation boundaries. To address class imbalance issues in the scene, such as the large proportion of road areas and the small proportion of traffic signs, the total loss L′ is a weighted combination of cross-entropy loss and Dice loss.

[0175] L′=0.5×L ce +0.5×L Dice (34)

[0176] Cross-entropy loss L ce Pixel-by-pixel classification errors are penalized to ensure overall segmentation accuracy. Dice loss L Dice Based on the intersection-union ratio (IUU) of the predicted and true masks, the segmentation capability for small targets is enhanced, mitigating the impact of class imbalance.

[0177] Based on the above-mentioned multi-sensor fusion method and perception method, this embodiment of the invention also proposes a device for autonomous driving, such as a controller for an autonomous vehicle, which can execute the above-mentioned multi-sensor fusion processing method or perception method based on LiDAR point cloud pseudo-image conversion.

[0178] In summary, the method of the present invention demonstrates high perception accuracy, robustness, and real-time performance in practical applications, and can meet the environmental perception requirements of autonomous driving systems, thus having broad application value.

[0179] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion, characterized in that, Includes the following steps: Step 1: LiDAR point cloud data preprocessing, including: Step 1.1 Denoising; Step 1.2 Point cloud registration; Step 1.3 Downsampling. Step 2: LiDAR point cloud pseudo-image generation, including: Step 2.1 multi-view projection; Step 2.2 feature encoding; Step 2.3 multi-view fusion; Step 3: Visual image preprocessing, including: Step 3.1 Distortion correction; Step 3.2 Size adjustment; Step 3.3 Illumination enhancement; Step 4: Feature extraction and fusion, including: Step 4.1 Building an improved CNN network for feature extraction; Step 4.2 Feature alignment; Step 4.3 Cross-modal attention fusion.

2. The multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion according to claim 1, characterized in that, Step 1.1, the denoising process, employs a statistical filtering algorithm to denoise the original LiDAR point cloud. The specific steps are as follows: Step 1.1.1 Calculate the average distance between each point and all points in its neighborhood. The number of neighborhood points is dynamically adjusted according to the point cloud density. For high-density point clouds, the number of neighborhood points is set to 50; for low-density point clouds, the number of neighborhood points is set to 30. Step 1.1.2 Calculate the standard deviation of the average distance of all points, identify points with an average distance greater than (mean + 1.5 × standard deviation) as noise points and remove them, retaining the valid point cloud data; Step 1.2, point cloud registration, uses IMU and GPS data to achieve spatiotemporal synchronization and coordinate transformation of the point cloud. The specific steps are as follows: Step 1.2.1 Using the angular velocity and acceleration data from the IMU, the motion state of the vehicle is predicted using the Kalman filter algorithm; Step 1.2.2 Correct the prediction results by combining GPS positioning data to obtain the vehicle's precise position and attitude information; Step 1.2.3 Unify the LiDAR point clouds at different times to the world coordinate system according to the vehicle's position and attitude, and control the registration error within 0.1m; Step 1.3, the downsampling process, uses a voxel grid downsampling method to reduce the number of point clouds, as detailed below: Step 1.3.1 The voxel mesh size is dynamically adjusted according to the point cloud density. For 128-line LiDAR, the voxel size is set to 0.1m×0.1m×0.1m; for 64-line LiDAR, the voxel size is set to 0.2m×0.2m×0.2m. Step 1.3.2 For each voxel grid, calculate the centroid coordinates of all points within the grid, and use the centroid point to represent the voxel to achieve downsampling of the point cloud.

3. The multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion according to claim 1, characterized in that, The specific steps of step 2.1, multi-view projection, are as follows: Step 2.1.1 Bird's-eye view BEV projection (1) Projection range: Taking the vehicle as the origin, a rectangular area of ​​120m×100m is formed with 100m in front, 20m behind, and 50m to the left and right. (2) Projection method: Project the three-dimensional point cloud coordinates (x,y,z) onto the two-dimensional plane coordinates (x,y), and the z coordinate is used for subsequent feature encoding; (3) Resolution: 512×512 pixels, each pixel corresponds to an actual distance of 0.234m×0.195m; Step 2.1.2 Front View FV Projection (1) Projection range: The vertical pitch angle is -5° to 20°, covering the ground to the sky, and the horizontal azimuth angle is -90° to 90°, achieving 180° horizontal scanning; (2) Projection method: Convert the three-dimensional point cloud coordinates (x, y, z) to polar coordinates. Where r is the distance and θ is the azimuth angle. The pitch angle is projected onto... flat; (3) Resolution: 256×512 pixels, 0.351° per pixel in the azimuth direction and 0.098° per pixel in the pitch direction; Step 2.1.3 Side View SV Projection (1) Projection range: 50m in front, 50m behind, and 30m above and below, forming a rectangular area of ​​100m×60m; (2) Projection method: Project the three-dimensional point cloud coordinates (x,y,z) onto the two-dimensional (y,z) plane; (3) Resolution: 256×512 pixels, each pixel corresponds to an actual distance of 0.391m×0.117m.

4. The multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion according to claim 3, characterized in that, The specific steps of feature encoding in step 2.2 are as follows: Step 2.2.1 Distance Information Encoding The distance r from the point cloud to the vehicle is normalized using the following formula: I r ＝225×(r-r min ) / (r max -r min ) (1) Where r min For a minimum distance of 0m, r max For a maximum distance of 100m, I r It is the normalized gray value, with a value range of 0-255; Step 2.2.2 Reflection Intensity Encoding The reflection intensity i of LiDAR point clouds is in the range of 0-255. The reflection intensity is weighted and adjusted according to the reflection characteristics of different materials. I i =i×ω (2) Where ω is the weighting coefficient, ω = 1.2 for metallic targets and ω = 0.8 for non-metallic targets; Step 2.2.3 Height Information Encoding The z-coordinate of the point cloud is normalized using the following formula: AND z =225×(zz min ) / (With max -With min ) (3) Where z min For the minimum height, z max For maximum height, I z The normalized z-coordinate is used to supplement the height information of the bird's-eye view. Step 2.2.4 Multi-feature fusion coding The pseudo-images for each viewpoint employ a three-channel design. The three channels for the bird's-eye view are distance grayscale value, reflection intensity weighted value, and height normalized value, respectively. The three channels for the front view and side view are distance grayscale value, reflection intensity weighted value, and point cloud density, respectively.

5. The multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion according to claim 4, characterized in that, Step 2.3, multi-view fusion, uses the SENet attention mechanism to fuse the pseudo-images from three views. The specific steps are as follows: Step 2.3.1 Reduce the dimensionality of the three pseudo-images from the three perspectives to the same number of channels (64) through a convolutional layer; Step 2.3.2 Perform global average pooling on the feature map of each viewpoint to obtain a 1×1×64 feature vector; Step 2.3.3 Calculate the attention weights (α1, α2, α3) for the three views through a fully connected layer, so that α1 + α2 + α3 = 1; Step 2.3.4: Perform a weighted summation of the feature maps from the three perspectives according to their weights to generate a multi-view fused pseudo-image. Step 2.3.5 Scene Adaptive Adjustment: In highway scenes, the weight of the bird's-eye view α1 is increased to 0.4-0.5; in urban road scenes, the weight of the front view α2 is increased to 0.4-0.5; in narrow road scenes, the weight of the side view α3 is increased to 0.3-0.

4.

6. The multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion according to claim 1, characterized in that, Step 3.1, distortion correction, involves obtaining the camera intrinsic parameter matrix K and distortion coefficients D using the Zhang Zhengyou calibration method, and then using the undistort function from the OpenCV library to perform distortion correction, thereby eliminating the impact of lens distortion on the image. D=[k1,k2,p1,p2,k3] (5) Among them, f x ,f y c is the focal length. x ,c y The coordinates of the principal point are k1, k2, and k3; k1, k2, and k3 are the radial distortion coefficients, and p1 and p2 are the tangential distortion coefficients. Step 3.2, size adjustment, involves adjusting the corrected image size to 512×256 pixels to match the size of the LiDAR pseudo-image, facilitating subsequent feature fusion. Step 3.3, illumination enhancement, employs the adaptive histogram equalization CLAHE algorithm to enhance image contrast. This enhances image details while avoiding overexposure, increasing image information entropy by 15%-20% in low-light scenes. Specific settings are as follows: (1) Block size: 8×8 pixels; (2) Contrast Limit: 2.0; (3) Interpolation method: bilinear interpolation.

7. The multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion according to claim 1, characterized in that, Step 4.1 describes the construction of an improved CNN network, using ResNet50 as the basic architecture, with the following improvements: (1) Add a channel attention module (CAM) and a spatial attention module (SAM) after the convolutional layer of each residual block; (2) The CAM module generates channel attention weights by performing global average pooling and fully connected layer processing on the channel dimension of the feature map, thereby enhancing the feature representation of key channels; (3) The SAM module performs max pooling and average pooling on the spatial dimension of the feature map and generates spatial attention weights by combining convolutional layers to highlight the features of the target region. (4) The network output feature map has a length, width and number of channels of 32×16×1024, which correspond to the features of LiDAR pseudo image and visual image, respectively. The specific steps of feature alignment in step 4.2 are as follows: Step 4.2.1 Establish coordinate mapping relationship Based on the rotation matrix R and translation vector T of the camera and LiDAR, a mapping relationship between pixel coordinates and point cloud coordinates can be established: P new =R×P LiDAR +T (6) Among them, P LiDAR P represents the three-dimensional coordinates (x, y, z) of a LiDAR point cloud. new This represents the 3D coordinates transformed into the camera coordinate system. The rotation matrix R is used to correct the attitude difference between the two sensors, and the translation vector T is used to compensate for the position offset. Then, it is further transformed into image pixel coordinates (u,v) through the camera intrinsic parameter matrix K, thus realizing the complete mapping from LiDAR point cloud coordinates to camera coordinates and then to image pixel coordinates. Step 4.2.2 Affine Transformation Correction Based on the established mapping relationship, an affine transformation is performed on the LiDAR pseudo-image features to fine-tune the position of each pixel in the feature map, ultimately aligning the two features in space. Step 4.2.3 Precision Control Alignment error is controlled within 2 pixels to ensure the effectiveness of cross-modal feature fusion.

8. The multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion according to claim 7, characterized in that, The specific steps of cross-modal attention fusion in step 4.3 are as follows: Step 4.3.1 Transfer the LiDAR pseudo-image features F l and visual image features F v Mapped to the same dimension through convolutional layers; Step 4.3.2 Calculate the similarity matrix: Where d is the number of feature channels. As a normalization factor, to avoid numerical inflation due to excessive dimensionality, and to ensure the comparability of similarities at different positions, S(i,j) represents F. l The i-th spatial position of ′ and F v The similarity of the j-th spatial location is such that the higher the similarity, the greater the likelihood that the two describe the same target or scene. Step 4.3.3 performs row and column normalization on the similarity matrix S(i,j) to obtain the LiDAR attention weights A for vision. lv And visual attention weights A for LiDAR vl A lv (i,j) represents F l The i-th position of F v The level of attention at position j, A vl (j,i) represents F v The j-th position of F l The level of attention given to the i-th position; Step 4.3.4 Generate attention enhancement features, including LiDAR attention enhancement feature F. l "and visual attention enhancement features F v ": F l ″=F l +Proj(A vl ×F v ′) (8) F l "Using the portion of visual features related to LiDAR features A" lv ×F v Enhance LiDAR features to compensate for the lack of LiDAR texture information; F v ″=F v +Proj(A lv ×F l ′) (9) F v "Using the visually relevant portion A of LiDAR features" vl ×F l Enhance visual features to compensate for the lack of robustness of vision in adverse weather conditions; Step 4.3.5 will F l "and F v "The features are concatenated into 1024 channels, preserving all the details of both features. The concatenated features are then added element by element to enhance the consistent information between the two features, resulting in the final fused feature F." 9. An autonomous driving perception method based on the multi-sensor fusion processing method for LiDAR point cloud pseudo-image conversion as described in claim 1, characterized in that, Including object detection and semantic segmentation; The target detection uses the YOLOv5 model as the basic detection framework, inputting the features fused in step 4 into the neck and head networks of YOLOv5. The neck network adopts a PANet structure, achieving multi-scale feature fusion through upsampling and downsampling. The head network includes three detection heads, corresponding to large, medium, and small target sizes, respectively, to meet the detection needs of different sized targets such as vehicles, pedestrians, and cyclists in autonomous driving scenarios, achieving multi-scale target detection. The loss function adopts CIoU loss, combined with class loss and confidence loss, to improve detection accuracy. The semantic segmentation is performed using the DeepLabv3+ model. The features fused in step 4 are input into the ASPP module. The ASPP module obtains multi-scale contextual information through dilated convolutions with different dilation rates. Skip connections are used to fuse low-level and high-level features to improve the accuracy of the segmentation boundary. The loss function uses cross-entropy loss combined with Dice loss to solve the class imbalance problem.

10. A device for autonomous driving, characterized in that, The device is capable of performing the multi-sensor fusion processing method based on LiDAR point cloud pseudo-image conversion as described in claim 1, or performing the perception method as described in claim 9.

Images