Road segmentation method, system, device and medium for laser radar and camera fusion

Abstract

The road segmentation method, system, device, and medium that integrates lidar and camera technology belong to the field of computer vision technology. They address the problem that existing single-sensor methods are difficult to handle in complex scenarios, and require a road segmentation method that can combine the advantages of multiple sensors. The road RGB image and road point cloud image are preprocessed separately to obtain a road grayscale image and a denoised road point cloud image, respectively. The road grayscale image and the denoised road point cloud image are then processed synchronously. A road depth map is obtained based on the synchronously processed road grayscale image and the synchronously processed denoised road point cloud image. The road depth map is then adjusted and fused with the road RGB image to obtain a road feature image. The road feature image is then input into the optimized DeepLabv3+ for segmentation, completing the road image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a road segmentation method, system, equipment and medium that integrates lidar and camera. Background Art

[0002] Road segmentation is a key technology in autonomous driving systems, aiming to identify drivable areas. Existing road segmentation methods are primarily categorized into camera-based and lidar-based approaches. Camera-based approaches rely on image texture information but perform poorly under varying lighting conditions or occlusions. LiDAR-based approaches offer high accuracy but are costly and suffer from data sparseness.

[0003] A single sensor can only provide partial information, such as LiDAR providing depth information and camera providing color information. It cannot fully perceive the environment. In bad weather and complex scenes, the performance will be reduced to varying degrees, making it difficult to achieve high-precision target detection and segmentation. In order to overcome the shortcomings of a single sensor, multi-sensor fusion technology has become a mainstream solution. By combining the advantages of different sensors, the defects of a single sensor can be compensated and the robustness and accuracy of the system can be improved. For example, the fusion of LiDAR and camera can effectively make up for these defects: (1) The camera performs well under good lighting conditions, while LiDAR is not affected by lighting. The fusion of the two can maintain stable segmentation performance under different lighting conditions; (2) The fusion technology combines the point cloud data of LiDAR and the image information of the camera to reduce the occlusion effect between targets and improve the integrity of the segmentation; (3) The data provided by the two sensors verify each other, reducing the impact of the error of the single sensor data on the segmentation results and improving the accuracy.

[0004] In summary, the existing single sensor method is difficult to cope with complex scenes, so a road segmentation method that can combine the advantages of multiple sensors is needed. Summary of the Invention

[0005] The present invention solves the problem that the existing single sensor method is difficult to cope with complex scenes and a road segmentation method that can combine the advantages of multiple sensors is needed.

[0006] The road segmentation method of the laser radar and camera fusion of the present invention comprises the following steps:

[0007] Step S1, constructing a road image dataset, wherein the road image dataset includes a road RGB image obtained by a camera and a road point cloud image obtained by a lidar;

[0008] Step S2: pre-processing the road RGB image and the road point cloud image to obtain a road grayscale image and a denoised road point cloud image, respectively, and synchronously processing the road grayscale image and the denoised road point cloud image;

[0009] Step S3, obtaining a road depth map based on the synchronously processed road grayscale image and the synchronously processed denoised road point cloud image, adjusting the road depth map, and fusing the adjusted road depth map with the road RGB image to obtain a road feature image;

[0010] In step S4, the road feature image is input into the optimized DeepLabv3+ for segmentation, thereby completing the segmentation of the road image.

[0011] Furthermore, in one embodiment of the present invention, in step S2, the synchronization process includes spatial synchronization and temporal synchronization;

[0012] The spatial synchronization is specifically:

[0013] Synchronize the camera's coordinate system with the lidar's coordinate system to complete spatial synchronization;

[0014] The time synchronization is specifically:

[0015] The camera and lidar each record a timestamp. Based on the principle of backward compatibility, the road grayscale image and the denoised road point cloud image are synchronized through the timestamp to complete time synchronization.

[0016] Furthermore, in one embodiment of the present invention, in step S3, the road depth map is adjusted, and the adjusted road depth map and the road RGB image are fused to obtain a road feature image, specifically:

[0017] The road depth map is completed, and the completed road depth map is processed in sequence by cropping, normalization, bilinear interpolation, convolution, maximum pooling and average pooling. After normalization adjustment, it is fused with the road RGB image through dot product to obtain the road feature image.

[0018] Furthermore, in one embodiment of the present invention, the road depth map is completed by:

[0019] The high reflection intensity information of the road depth map is retained, and the low reflection intensity information of the road depth map is supplemented by interpolation.

[0020] Furthermore, in one embodiment of the present invention, in step S4, the optimized DeepLabv3+ is specifically:

[0021] CBAM is introduced in the encoder and decoder of DeepLabv3+ respectively. In the decoder of DeepLabv3+, the road feature image is processed by the convolutional neural network algorithm and convolution in sequence, and then dot-multiplied with the road depth map after adjusting the attention weight.

[0022] The road segmentation system integrating lidar and camera according to the present invention includes the following modules:

[0023] A construction module is used to construct a road image dataset, wherein the road image dataset includes RGB images of roads obtained by cameras and point cloud images of roads obtained by lidar;

[0024] A synchronization module pre-processes the road RGB image and the road point cloud image to obtain a road grayscale image and a denoised road point cloud image, and synchronizes the road grayscale image and the denoised road point cloud image;

[0025] A fusion module obtains a road depth map based on the synchronously processed road grayscale image and the synchronously processed denoised road point cloud image, adjusts the road depth map, and fuses the adjusted road depth map with the road RGB image to obtain a road feature image;

[0026] The segmentation module inputs the road feature image into the optimized DeepLabv3+ for segmentation, thus completing the segmentation of the road image.

[0027] An electronic device according to the present invention comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0028] Memory for storing computer programs;

[0029] The processor is used to implement the road segmentation method of laser radar and camera fusion as described in any of the above methods when executing the program stored in the memory.

[0030] The present invention provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the road segmentation method of the laser radar and camera fusion described in any of the above methods is implemented.

[0031] This invention solves the problem that existing single-sensor methods are difficult to handle complex scenarios and require a road segmentation method that can combine the advantages of multiple sensors. Specific benefits include:

[0032] 1. The road segmentation method that integrates lidar and camera, as described in the present invention, addresses the problem that single-sensor methods in existing technologies are difficult to handle in complex scenarios, and a road segmentation method that can combine the advantages of multiple sensors is needed. By effectively fusing the road RGB image acquired by the camera and the road point cloud image acquired by the lidar, the present invention can maintain stable performance in complex scenarios, reduce occlusion between objects, and avoid the impact of single sensor errors on the segmentation results.

[0033] 2. The LiDAR-camera fusion road segmentation method of the present invention synchronously processes the data collected by the LiDAR and the denoised camera data. This avoids waiting time for unreachable data and processes the data collected by the LiDAR and camera in parallel, reducing computational latency, fully utilizing computing resources, and improving the overall efficiency of the system. This processing method can also continue to use the data of another sensor when a sensor fails or data is interfered with, or dynamically adjust weights to adapt to these changes, thereby improving the system's fault tolerance and reliability.

[0034] 3. The LiDAR-camera fusion road segmentation method described in this invention uses road feature images as input to the segmentation model, helping the model better handle occlusions and distinguish obscured objects. The addition of LiDAR depth information can reduce the impact of illumination variations on segmentation results, improve the segmentation model's performance in low-light or backlit environments, enhance the model's ability to understand complex scenes, and improve robustness and accuracy.

[0035] 4. The road segmentation method integrating lidar and camera described in the present invention can help the model better capture the boundary information of objects by simultaneously adding CBAM to the encoder and decoder in Deeplabv3+, improve the edge accuracy of the segmentation results, reduce the calculation of irrelevant areas, and improve computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 This is the time synchronization diagram described in the second embodiment;

[0038] Figure 2 This is the DeepLabv3+ image after the optimized road feature image input described in Implementation Method 3. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe various embodiments of the present invention in conjunction with the accompanying drawings. The embodiments described with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0040] Implementation method 1: The road segmentation method using laser radar and camera fusion described in this implementation method includes the following steps:

[0041] Step S1, constructing a road image dataset, wherein the road image dataset includes a road RGB image obtained by a camera and a road point cloud image obtained by a lidar;

[0042] Step S2: pre-processing the road RGB image and the road point cloud image to obtain a road grayscale image and a denoised road point cloud image, respectively, and synchronously processing the road grayscale image and the denoised road point cloud image;

[0043] Step S3, obtaining a road depth map based on the synchronously processed road grayscale image and the synchronously processed denoised road point cloud image, adjusting the road depth map, and fusing the adjusted road depth map with the road RGB image to obtain a road feature image;

[0044] In step S4, the road feature image is input into the optimized DeepLabv3+ for segmentation, thereby completing the segmentation of the road image.

[0045] In existing technologies, single-sensor methods are difficult to cope with complex scenarios, so a road segmentation method that can combine the advantages of multiple sensors is needed.

[0046] To solve the above technical problems, this embodiment proposes a road segmentation method that integrates lidar and camera, including the following steps:

[0047] Step S1: Constructing a road image dataset, wherein the road image dataset includes RGB (red, green, and blue) road images acquired by a camera and road point cloud images acquired by a lidar. The road image dataset is a KITTI dataset (a dataset for evaluating computer vision algorithms in autonomous driving scenarios).

[0048] Step S2: Preprocess the road RGB image and the road point cloud image respectively, specifically:

[0049] The preprocessing mentioned above refers to denoising the road RGB image and the road point cloud image respectively. The denoising of the road point cloud image uses Gaussian filtering and downsampling respectively. Gaussian filtering smoothes the road point cloud image and reduces the influence of noise. Downsampling uses voxel grid filtering: the road point cloud image is divided into a regular voxel grid, and the points in each grid are averaged or randomly sampled, thereby reducing the number of road point cloud images and quickly processing the road point cloud image. The road RGB image is denoised and grayscaled. The median filtering technology is used to remove the noise in the road RGB image, and the size of the road RGB image is adjusted to adapt to the input of the subsequent network. The road RGB image is converted into a noise-free road grayscale image through the weighted averaging method. The commonly used formula for the weighted averaging method is:

[0050] ;

[0051] Where, is the road grayscale image, 、 and They are red pixels, green pixels, and blue pixels in the road RGB image, respectively.

[0052] The road grayscale image and the denoised road point cloud image are processed synchronously.

[0053] Step S3: Filter out the coordinates in the denoised road point cloud image Only the points in front of the camera are retained to prevent incorrect projection. Then the filtered and denoised road point cloud image is projected onto the image plane using the camera's intrinsic parameters to obtain the pixel position of each point in the image. .

[0054] The mathematical expression is:

[0055] ;

[0056] ;

[0057] Where, and Different focal lengths of the camera. is the coordinate of the principal point of the image, is the coordinate point after transformation (in the camera coordinate system).

[0058] Create a new two-dimensional matrix of the same size as the image, with the initial value of 0, and position each projected pixel Corresponding depth value By writing into the matrix with an initial value of 0, the road depth map can be obtained, and the road depth map is adjusted. The adjusted road depth map and the road RGB image are fused to obtain the road feature image.

[0059] In step S4, the road feature image is input into the optimized DeepLabv3+ (deep learning model) for segmentation. The segmentation result is subjected to morphological operations and noise removal to optimize edges and remove isolated areas or fill holes to obtain the final segmentation result, thus completing the segmentation of the road image.

[0060] Therefore, this embodiment can maintain stable performance under different lighting conditions by fusing the road RGB image obtained by the camera and the road point cloud image obtained by the lidar, reduce occlusion between targets, and avoid the impact of single sensor errors on the segmentation results.

[0061] Implementation 2: This implementation further limits the road segmentation method of integrating the laser radar and the camera described in Implementation 1. In step S2, the synchronization process includes spatial synchronization and temporal synchronization.

[0062] The spatial synchronization is specifically:

[0063] Synchronize the camera's coordinate system with the lidar's coordinate system to complete spatial synchronization;

[0064] The time synchronization is specifically:

[0065] The camera and lidar each record a timestamp. Based on the principle of backward compatibility, the road grayscale image and the denoised road point cloud image are synchronized through the timestamp to complete time synchronization.

[0066] In this implementation, the denoised road point cloud image and road grayscale image are generated in the LiDAR coordinate system and camera coordinate system, respectively. To achieve fusion between the LiDAR and camera data, their coordinate systems and timestamps must be unified. Therefore, in this implementation, the road grayscale image and the denoised road point cloud image are spatially and temporally synchronized, respectively.

[0067] The spatial synchronization of the road grayscale image and the denoised road point cloud image is specifically as follows:

[0068] Using the checkerboard method to calibrate the lidar and camera, we can get the intrinsic parameters of the camera and the extrinsic parameters between the lidar: that is, the rotation matrix and the translation vector. The process of converting lidar data from the lidar coordinate system to the camera coordinate system is: convert the 3D lidar point Convert the 3D laser radar coordinate system to the 3D camera coordinate system to obtain the corresponding 3D camera point , the specific conversion process is as follows:

[0069] ;

[0070] Where, is the transformation matrix between the 3D lidar coordinate system and the 3D camera coordinate system, Optimize the matrix for rotation.

[0071] According to the camera's internal parameter information, the 3D camera point Convert the 3D camera coordinate system to the 2D image coordinate system to get the corresponding 2D image point , the specific conversion process is as follows:

[0072] ;

[0073] Where, is the conversion matrix between the 3D camera coordinate system and the 2D image coordinate system. The horizontal viewing angle of the camera is about 40 to 50 degrees. Only the 2D points that fall within the camera viewing angle will be retained, that is, the 2D image points Need to meet 0< <= , and 0< <= , and are the width and height of the camera image, is the horizontal coordinate that corresponds to the width of the camera image, is the vertical coordinate corresponding to the camera image height, is the vertical coordinate in the 3D camera coordinate system.

[0074] According to the above transformation, any 3D lidar point can be Convert to the corresponding 2D image point The final conversion formula is:

[0075] ;

[0076] This completes the spatial synchronization between the lidar and the camera.

[0077] The time synchronization of the road grayscale image and the denoised road point cloud image is specifically as follows:

[0078] The collected data is synchronized using software. After the lidar and camera acquire data, the system records a timestamp, which is used to align the data from the two lidar and camera. Software synchronization is simple to implement, does not require hardware circuitry, and is easily implemented on platforms such as MATLAB (commercial mathematical software) and Python (computer programming language).

[0079] For example, if the sampling frame frequency of the laser radar used is 10 frames per second, and the frame rate of the camera video is 20 frames per second, then according to the characteristics of the laser radar and camera sampling frequencies, we select every one frame of laser radar data and every two frames of camera data as valid data and record them, that is, the time node data of 100ms, 200ms, 300ms after the start time, so we use the time alignment method as follows: Figure 1 shown.

[0080] This completes the time synchronization between the lidar and camera data.

[0081] Therefore, this embodiment helps to achieve the fusion of lidar data and camera data by spatially synchronizing and temporally synchronizing the road grayscale image and the denoised road point cloud image respectively.

[0082] Implementation 3: This implementation further limits the road segmentation method of the laser radar and camera fusion described in Implementation 1. In step S3, the road depth map is adjusted, and the adjusted road depth map and the road RGB image are fused to obtain a road feature image. Specifically,

[0083] The road depth map is completed, and the completed road depth map is processed in sequence by cropping, normalization, bilinear interpolation, convolution, maximum pooling and average pooling. After normalization adjustment, it is fused with the road RGB image through dot product to obtain the road feature image.

[0084] In this embodiment, the road depth map is completed as follows:

[0085] The high reflection intensity information of the road depth map is retained, and the low reflection intensity information of the road depth map is supplemented by interpolation.

[0086] In this implementation, in visual perception tasks, RGB road images excel at capturing surface information such as texture and color, while road depth maps contain rich spatial structure and geometric information, effectively complementing the spatial information lacking in RGB road images. The effective fusion of these two complementary information types significantly enhances the model's scene perception and object recognition accuracy.

[0087] However, if the road depth map is not fully preprocessed and guided by modeling, and is simply spliced ​​with the road RGB image or directly fed into the subsequent model for segmentation, it will not only fail to fully utilize its advantages, but may even bring additional information redundancy and misleading interference.

[0088] Furthermore, road depth maps often contain a large number of invalid or low-quality pixel values, such as those caused by sensor acquisition errors, missing data in distant areas, depth jumps in blurred edge regions, and depth holes caused by occlusion or reflection. If this invalid information is directly used in the subsequent feature extraction process without filtering, the subsequent model segmentation process will mistakenly focus on non-target areas, thereby reducing the semantic clarity and discriminability of the overall representation.

[0089] Furthermore, road RGB images and road depth maps differ in spatial resolution, scale, and receptive field. Without unified alignment, subsequent models will struggle to achieve precise pixel-level alignment during the fusion phase, leading to information mismatch and feature perturbations. For example, road depth maps typically have a wide range of values ​​and uneven distribution. Using them directly as model input without processing can easily lead to instability in subsequent model training, offset gradient updates, and even failure of the attention mechanism.

[0090] To address the above technical issues, this embodiment incorporates the unique reflection intensity of the LiDAR data while generating the road depth map. This prioritizes the depth information of high-reflection-intensity information and interpolates the low-reflection-intensity areas to obtain a higher-quality road depth map. The following processing is performed on the higher-quality road depth map:

[0091] like Figure 2 As shown in the figure, a depth threshold cropping strategy is set to remove unreasonable pixels and the higher-quality road depth map is normalized to stabilize its value range between [0, 1]. Subsequently, the higher-quality road depth map is adjusted to the same resolution as the road RGB image through interpolation (for example, bilinear interpolation) to ensure spatial alignment and lay the foundation for subsequent fusion.

[0092] Furthermore, in existing fusion methods, most subsequent models ignore the heterogeneity in information contributions between different modalities during segmentation, lacking the ability to selectively model depth information. In other words, when processing road RGB images, subsequent models fail to actively "focus" on the important geometric structures contained in depth, instead averaging or uniformly treating all depth information. This "indiscriminate fusion" strategy often results in poor performance in complex backgrounds, occluded areas, or extreme lighting conditions.

[0093] To address the aforementioned technical issues, this implementation introduces a 3×3 convolution operation on the interpolated, higher-quality road depth map to extract shallow feature representations. This operation captures local contour information, significantly enhancing object shape perception. To further enhance the subsequent model's ability to select regions of interest, maximum pooling and average pooling are used to perform spatial statistical modeling on the deep features extracted by convolution, generating a saliency attention map.

[0094] Furthermore, without an attention weight map generated based on the road depth map, the semantic guidance provided by the shallow feature map will be limited, leading to performance degradation in multiple areas. First, due to the lack of attention to deep structural information, subsequent models struggle to accurately perceive the edges and geometric details of objects in the image, which can easily lead to blurred outlines, broken edges, and other issues, affecting the final segmentation accuracy. Second, shallow features often contain a large amount of background interference information. Without the added attention weights to suppress it, this redundant information may be mistaken for foreground objects, thereby reducing the subsequent model's ability to discriminate foreground areas. Furthermore, for small objects or those with blurred boundaries, the lack of depth guidance further weakens the subsequent model's recognition ability, making small objects easily overlooked or misclassified. In summary, the lack of a deep attention-based fusion strategy limits the decoder's ability to focus on important areas and leads to significant deficiencies in the overall semantic segmentation model in terms of accuracy, edge perception, and environmental adaptability.

[0095] To address these technical issues, this implementation uses a sigmoid function (activation function) to normalize the image, generating an attention weight map between 0 and 1. This map effectively captures key image regions with dramatic depth variations and prominent structures. Finally, this attention weight map is element-wise weighted and fused with the road RGB image. This allows the subsequent model to focus more on semantically salient regions during backbone feature extraction, suppressing background interference and improving overall perception.

[0096] Therefore, this embodiment uses 3×3 convolution to extract shallow features and structural information from the interpolated higher-quality road depth map, and outputs a 1-channel feature map. Based on the extracted depth map features, an attention map is generated through maximum pooling and average pooling, and finally normalized by the Sigmoid function to obtain a weight map between [0,1]. This process can highlight areas with obvious depth changes. Finally, the generated attention map is applied to the road RGB image, and the weight of the attention weight map is re-weighted at each position of the road RGB image through dot multiplication, which suppresses background noise, strengthens the feature expression of important areas, and improves the performance of subsequent models in complex backgrounds, occluded areas or extreme lighting conditions.

[0097] In summary, this embodiment performs a series of feature extraction on a higher-quality road depth map and then fuses it with the road RGB image. This not only solves the problem of the road depth map causing a large number of invalid or low-quality pixel values ​​in the subsequent model segmentation process, but also gives full play to the advantages of the road RGB image and road depth map.

[0098] Implementation 4: This implementation further limits the road segmentation method of laser radar and camera fusion described in Implementation 1. In step S4, the optimized DeepLabv3+ is specifically:

[0099] CBAM is introduced in the encoder and decoder of DeepLabv3+ respectively. In the decoder of DeepLabv3+, the road feature image is processed by the convolutional neural network algorithm and convolution in sequence, and then dot-multiplied with the road depth map after adjusting the attention weight.

[0100] In this implementation, CBAM (Convolutional Block Attention Module) is introduced into the encoder and decoder of DeepLabv3+, respectively. Adding CBAM to the encoder enables the model to more effectively integrate features at different levels, while adding CBAM to the decoder enables the model to pay more attention to the target area.

[0101] CBAM is introduced after the results of each branch. Its main purpose is to refine the feature representation of the input through the attention mechanism. CBAM combines two mechanisms: the channel attention module and the spatial attention module. It can automatically learn the weight distribution relationship between different channels and the response degree of important areas in the spatial dimension, thereby enhancing the expression ability of key features. Specifically, the channel attention module focuses on which channels are important, while the spatial attention module pays more attention to which areas of the image are more important. This special mechanism helps the model more effectively mine the potential structural and semantic information in the input feature map, improve the model's perception of foreground targets, and reduce the interference of irrelevant background, thereby achieving more accurate feature extraction in subsequent tasks.

[0102] In order to further improve the model's perception of key target areas and suppress irrelevant or redundant information noise in the background, a spatial attention mechanism module is introduced as an important component of CBAM to perform targeted feature enhancement on the fused feature map.

[0103] The core idea of ​​the spatial attention module is to adaptively adjust the feature weights of each spatial position based on the response differences in the spatial dimension, so that the model can focus on the key areas in the image. In specific implementation, the spatial attention module first compresses the fused three-dimensional feature map along the channel dimension. This process generates two two-dimensional maps (size 200, ... ), representing different spatial statistical attributes:

[0104] (1) The average pooling feature map can capture the average response of each spatial position in the image in the overall channel dimension, reflecting the importance of the region and is suitable for preserving the overall structural information;

[0105] (2) The maximum pooling feature map focuses on the strongest activation value in the channel, which highlights the target significance and is suitable for detecting local high-response areas.

[0106] The two 2D pooling images are then concatenated in the channel dimension to form a pool of size The fusion graph is then processed through a convolution operation (usually The convolution kernel is used to learn a single-channel spatial attention weight map. After being activated by the Sigmoid function, this weight map is element-wise multiplied with the original fused feature map to re-weight the importance of features in different spatial regions.

[0107] The Sigmoid function is specifically:

[0108] ;

[0109] Where, is the output variable, is the input variable, The base of the natural logarithm The power is the exponential decay of the input.

[0110] In this way, CBAM can automatically discover and enhance discriminative regions in the image, improving spatial perception in road segmentation tasks. Compared with traditional feature fusion methods, CBAM not only achieves more fine-grained region selection, but also significantly improves the model's robustness to difficult scenes such as occlusion and complex backgrounds.

[0111] In addition, in the decoder of DeepLabv3+, the road feature image is processed by the convolutional neural network algorithm and convolution in sequence, and then dot-multiplied with the road depth map after attention weight adjustment.

[0112] The road feature image is input into DeepLabv3+ for semantic segmentation. Specifically, the pixel values ​​of the road feature image are normalized to range, and enhance the road feature image by flipping and other operations, and finally adjust the road feature image to a fixed size (such as ).

[0113] like Figure 2As shown, the adjusted road feature image enters the DeepLabv3+ backbone network, which consists of an encoder and a decoder. To obtain higher-resolution feature maps, the encoder uses Xception (a convolutional neural network algorithm) with atrous convolution as the feature extraction network. Multiple convolutions with different atrous rates are used to generate low-level and high-level feature maps, respectively. These convolutions are an extension of the original convolution module, achieving a wider visual receptive field while maintaining the same computational cost and parameter count.

[0114] The low-level feature maps obtained by Xception are directly fed into the decoder, while the high-level feature maps are processed by ASSP (Atrous Spatial Pyramid Pooling). ASSP consists of four parallel convolutions with different dilation rates and an image pooling layer, capturing multi-scale contextual information and improving segmentation accuracy by fusing multi-scale information. Each branch's results are passed through CBAM, and the CBAM outputs are concatenated and fused through a 1×1 convolution to produce a feature map containing multi-scale information. The ASSP output is upsampled by a factor of 4 and fused with the low-level features obtained through the 1×1 convolution. The attention weight map generated from the road depth map is point-multiplied with the low-level feature map. The fused result is passed through CBAM, and the CBAM output undergoes a 3×3 convolution and a factor of 4 upsampling to obtain a segmentation map with the same resolution as the input image. The segmentation result is converted to a binary image, and then subjected to opening and closing operations to remove noise while retaining the main object area, resulting in a segmentation result with complete boundaries.

[0115] In order to better process the segmentation results output by the model, the output results are then subjected to morphological operations. The main purpose is to remove some isolated misclassified pixels that may exist in the segmentation results output by the model, and to fill small holes that may exist within the target area. Specifically, the segmentation result image is converted into a binary image using the maximum probability category. Then, the opening operation in the morphological operation removes small noise while retaining the main target area. The closing operation then fills the holes within the target area, while retaining the external shape of the target and making the target boundary more complete, resulting in a more accurate segmentation result.

[0116] Regarding the evaluation of road segmentation results, the evaluation indicators mainly include AP (average precision), PRE (precision), FPR (false positive rate) and FNR (false negative rate).

[0117] ;

[0118] ;

[0119] Where FN (False Negative) is a false negative, where a positive sample is mistakenly identified as a negative sample; FP (False Positive) is a false positive, where a negative sample is mistakenly identified as a positive sample. TN (True Negative) and TP (True Positive) are both true positives.

[0120] Implementation 5: The road segmentation system integrating lidar and camera described in this implementation includes the following modules:

[0121] A construction module is used to construct a road image dataset, wherein the road image dataset includes RGB images of roads obtained by cameras and point cloud images of roads obtained by lidar;

[0122] A synchronization module pre-processes the road RGB image and the road point cloud image to obtain a road grayscale image and a denoised road point cloud image, and synchronizes the road grayscale image and the denoised road point cloud image;

[0123] A fusion module obtains a road depth map based on the synchronously processed road grayscale image and the synchronously processed denoised road point cloud image, adjusts the road depth map, and fuses the adjusted road depth map with the road RGB image to obtain a road feature image;

[0124] The segmentation module inputs the road feature image into the optimized DeepLabv3+ for segmentation, thus completing the segmentation of the road image.

[0125] Embodiment 6: An electronic device described in this embodiment includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0126] Memory for storing computer programs;

[0127] The processor is used to implement the road segmentation method of the laser radar and camera fusion described in any one of embodiments one to four when executing the program stored in the memory.

[0128] Implementation method seven, a computer-readable storage medium described in this implementation method, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the road segmentation method of the laser radar and camera fusion described in any one of implementation methods one to four is implemented.

[0129] The above is a detailed introduction to the road segmentation method, system, equipment and medium for the fusion of lidar and camera proposed in the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A road segmentation method based on the fusion of laser radar and camera, characterized in that: The following steps are involved: Step S1, constructing a road image dataset, wherein the road image dataset includes a road RGB image obtained by a camera and a road point cloud image obtained by a lidar; Step S2: pre-processing the road RGB image and the road point cloud image to obtain a road grayscale image and a denoised road point cloud image, respectively, and synchronously processing the road grayscale image and the denoised road point cloud image; Step S3, obtaining a road depth map based on the synchronously processed road grayscale image and the synchronously processed denoised road point cloud image, adjusting the road depth map, and fusing the adjusted road depth map with the road RGB image to obtain a road feature image; Step S4: Input the road feature image into the optimized DeepLabv3+ for segmentation, thereby completing the segmentation of the road image; In step S3, the road depth map is adjusted, and the adjusted road depth map and the road RGB image are fused to obtain a road feature image, specifically: The road depth map is completed, and the completed road depth map is processed in sequence by cropping, normalization, bilinear interpolation, convolution, maximum pooling and average pooling. After normalization adjustment, the road feature image is obtained by fusion with the road RGB image through dot product; In step S4, the optimized DeepLabv3+ is specifically: CBAM is introduced in the encoder and decoder of DeepLabv3+ respectively. In the decoder of DeepLabv3+, the road feature image is processed by the convolutional neural network algorithm and convolution in sequence, and then dot-multiplied with the road depth map after adjusting the attention weight.

2. The road segmentation method of laser radar and camera fusion according to claim 1, characterized in that: In the step S2, the synchronization process includes spatial synchronization and temporal synchronization; The spatial synchronization is specifically: Synchronize the camera's coordinate system with the lidar's coordinate system to complete spatial synchronization; The time synchronization is specifically: The camera and lidar each record a timestamp. Based on the principle of backward compatibility, the road grayscale image and the denoised road point cloud image are synchronized through the timestamp to complete time synchronization.

3. The road segmentation method of laser radar and camera fusion according to claim 1, characterized in that: The road depth map is completed as follows: The high reflection intensity information of the road depth map is retained, and the low reflection intensity information of the road depth map is supplemented by interpolation.

4. The road segmentation system integrating lidar and camera is characterized by: Includes the following modules: A construction module is used to construct a road image dataset, wherein the road image dataset includes RGB images of roads obtained by cameras and point cloud images of roads obtained by lidar; A synchronization module pre-processes the road RGB image and the road point cloud image to obtain a road grayscale image and a denoised road point cloud image, and synchronizes the road grayscale image and the denoised road point cloud image; A fusion module obtains a road depth map based on the synchronously processed road grayscale image and the synchronously processed denoised road point cloud image, adjusts the road depth map, and fuses the adjusted road depth map with the road RGB image to obtain a road feature image; The segmentation module inputs the road feature image into the optimized DeepLabv3+ for segmentation, thus completing the segmentation of the road image; In the fusion module, the road depth map is adjusted, and the adjusted road depth map and the road RGB image are fused to obtain a road feature image, specifically: The road depth map is completed, and the completed road depth map is processed in sequence by cropping, normalization, bilinear interpolation, convolution, maximum pooling and average pooling. After normalization adjustment, the road feature image is obtained by fusion with the road RGB image through dot product; In the segmentation module, the optimized DeepLabv3+ is specifically: CBAM is introduced in the encoder and decoder of DeepLabv3+ respectively. In the decoder of DeepLabv3+, the road feature image is processed by the convolutional neural network algorithm and convolution in sequence, and then dot-multiplied with the road depth map after adjusting the attention weight.

5. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; The processor is configured to implement the road segmentation method of laser radar and camera fusion as described in any one of claims 1 to 3 when executing the program stored in the memory.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the road segmentation method of the laser radar and camera fusion according to any one of claims 1 to 3 is implemented.

Images