A drivable area detection method and system combining laser and vision

Abstract

The application discloses a kind of laser and vision's drivable area detection method and system, wherein method, including: ground segmentation is carried out to laser point cloud under road scene, and ground point and non-ground point are obtained, road path point cloud is obtained from ground point, and non-ground point cloud is divided into left and right sides using road path point cloud, and filtering is carried out to the non-ground point cloud of left and right sides to obtain road boundary point;Laser point cloud under road scene is converted into depth map, and the surface normal information of depth map and the color image under road scene are input into drivable area detection network model, and the initial drivable area is output;Road boundary point is fused with initial drivable area, and the final drivable area is obtained.The final drivable area contains both the spatial position information of road boundary and the semantic image information of vehicle drivable area, and the application can provide efficient and accurate drivable area detection results in non-paved road and semi-natural scene road.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving and environmental perception for mobile robots, and more specifically, relates to a method and system for detecting drivable areas that combines laser and vision. Background Technology

[0002] Existing road detection algorithms are mainly focused on structured urban road scenarios, exhibiting poor adaptability in unpaved roads and semi-natural road environments such as wilderness, rural areas, farmland, forests, and mining areas. This makes it difficult for autonomous vehicles and mobile robots to navigate within accurate drivable areas.

[0003] Unpaved roads and semi-natural road scenes differ significantly from urban and rural outdoor scenes. The road surfaces are uneven, with irregular widths, and lack specific auxiliary indicators such as curbs. Mining roads are a prime example. These roads represent typical open, large scenes characterized by long target distances within the scene space, and also present challenges such as light pollution and adversarial scenarios. Vision-based solutions cannot guarantee scene adaptability and provide accurate spatial information for positioning and planning, while laser-based detection algorithms struggle to provide rich semantic information. Therefore, single-sensor solutions are insufficient.

[0004] This demonstrates that existing technologies face challenges in detecting drivable areas on unpaved roads and in semi-natural road environments. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and system for detecting drivable areas that combines laser and vision, thereby solving the technical problem that existing technologies have difficulty in detecting drivable areas on unpaved roads and semi-natural road scenes.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for detecting drivable areas combining laser and vision is provided, comprising:

[0007] Ground segmentation is performed on the laser point cloud in the road scene to obtain ground points and non-ground points. The road radial point cloud is obtained from the ground points. The non-ground point cloud is divided into left and right sides using the road radial point cloud. The non-ground point cloud on the left and right sides is filtered to obtain the road boundary points.

[0008] The laser point cloud in the road scene is converted into a depth map. The surface normal information of the depth map and the color image in the road scene are input into the drivable area detection network model to output the initial drivable area.

[0009] The road boundary points are merged with the initial drivable area to obtain the final drivable area;

[0010] The drivable area detection network model includes a decoder and two parallel encoders, and is trained as follows:

[0011] The surface normal information of the road depth image and the road color image marked with drivable areas are used as training samples. During training, two parallel encoders encode the surface normal information of the road depth image and the road color image respectively. The decoder decodes the encoded information output by the two parallel encoders to obtain the predicted drivable areas. The error between the predicted drivable areas and the marked drivable areas is used as the loss. The model parameters are updated by backpropagation and trained until convergence to obtain the trained drivable area detection network model.

[0012] Furthermore, the specific implementation method for merging the road boundary points with the initial drivable area is as follows:

[0013] The road boundary points are aligned with the image plane of the initial drivable area through projection transformation. Then, the road boundary points are fitted to form a continuous road boundary line. The area within the road boundary line is the final drivable area.

[0014] Furthermore, the specific implementation method for merging the road boundary points with the initial drivable area is as follows:

[0015] Extract the maximum and minimum corner coordinates from the bounding box of the point cloud containing road boundary points to form a point cloud rectangle; extract the maximum and minimum corner coordinates from the bounding box of the image containing the initial drivable area to form an image rectangle.

[0016] Calculate the intersection-union ratio (IUR) of the point cloud bounding box and the image bounding box. When the IUR is greater than or equal to a set threshold, the overlapping area of ​​the point cloud bounding box and the image bounding box is taken as the final drivable area. When the IUR is less than the set threshold, the area within the road boundary line fitted by the road boundary points or the initial drivable area is taken as the final drivable area.

[0017] Furthermore, the two parallel encoders include a surface normal encoder and an RGB encoder. The surface normal encoder includes multiple units for encoding geometric features extracted from surface normal information, and the RGB encoder includes multiple units for encoding color texture features extracted from road color images. The output of each unit in the surface normal encoder is fused with the output of the corresponding unit in the RGB encoder to obtain a fused feature.

[0018] Furthermore, the decoder includes multiple upsampling modules and multiple feature extraction modules. Skip connections are used between the upsampling modules and the feature extraction modules, as well as between the upsampling modules and the units in the RGB encoder. The upsampling modules upsample the fused features and then input them into the feature extraction modules, which then decode the input data.

[0019] Furthermore, the fused features are fed into the decoder after passing through the hybrid attention module. The hybrid attention module, which combines channel attention and spatial attention, receives the fused features and generates two sets of channel feature descriptors by performing global max pooling and global average pooling operations, respectively. The two sets of channel feature descriptors are then input into the same shared fully connected layer, generating two different intermediate vectors. The two sets of intermediate vectors are directly merged, and the Sigmoid activation function is applied to generate channel weights. Finally, the weighted fused features are obtained by multiplying the fused features by the channel weights.

[0020] Furthermore, the method also includes:

[0021] During detection, the laser point cloud and color image in the road scene are time-aligned, and then the laser point cloud in the road scene is converted into a depth map through spatial transformation to complete the spatiotemporal matching of the laser point cloud and the color image.

[0022] During training, the road depth images and road color images in the training samples have already been spatiotemporally matched.

[0023] Furthermore, after distortion removal of the laser point cloud in the road scene, ground segmentation is achieved through plane fitting, normal vector method, planar grid method, or non-uniform grid division.

[0024] Furthermore, the specific method of non-uniform grid division is as follows:

[0025] The region containing the laser point cloud in the road scene after distortion correction is taken as the target space region. The target space region is first divided into multiple coarse annular regions according to the polar radius. Each coarse annular region is further divided into multiple basic grid units according to the polar radius and polar angle.

[0026] For all laser point clouds in a basic grid cell, select the points with the lowest height values ​​as initial ground points and add them to the ground point set. Calculate the eigenvalues ​​and corresponding eigenvectors of the initial ground points. Use the eigenvector corresponding to the minimum eigenvalue as the ground plane normal vector. Calculate the transformation range of the initial ground points in the z-direction as the ground plane coefficient. Project the other laser point clouds in a basic grid cell, excluding the initial ground points, onto the ground plane normal vector in sequence. When the error between the distance projected onto the ground plane normal vector and the ground plane coefficient is less than a preset empirical value, add the projected laser point cloud to the ground point set.

[0027] By combining the ground point sets divided by all basic grid units, the ground points of the laser point cloud in the road scene are obtained, thereby achieving ground segmentation.

[0028] According to another aspect of the present invention, a drivable area detection system combining laser and vision is provided, comprising:

[0029] LiDAR is used to collect laser point clouds in road scenes;

[0030] A camera used to capture color images of road scenes;

[0031] The road boundary point detection module is used to perform ground segmentation on the laser point cloud in the road scene to obtain ground points and non-ground points. It obtains the road radial point cloud from the ground points, uses the road radial point cloud to divide the non-ground point cloud into left and right sides, and filters the non-ground point clouds on the left and right sides to obtain the road boundary points.

[0032] The training module uses the surface normal information of the road depth image and the road color image marked with drivable areas as training samples. When training with training samples, two parallel encoders encode the surface normal information of the road depth image and the road color image respectively. The decoder decodes the encoded information output by the two parallel encoders to obtain the predicted drivable areas. The error between the predicted drivable areas and the marked drivable areas is used as the loss. The model parameters are updated by backpropagation and trained until convergence to obtain the trained drivable area detection network model.

[0033] The drivable area detection module is used to convert the laser point cloud in the road scene into a depth map, input the surface normal information of the depth map and the color image in the road scene into the drivable area detection network model, and output the initial drivable area.

[0034] The decision fusion module is used to merge road boundary points with the initial drivable area to obtain the final drivable area.

[0035] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0036] (1) After ground segmentation of the point cloud, ground points and non-ground points are obtained. Unpaved roads and semi-natural scene roads have uneven surfaces and irregular widths, and there are no specific auxiliary indicators such as curbs. At this time, the point cloud is only classified according to whether it is a ground point. In order to obtain the road boundary, further classification and filtering operations are required to obtain the road boundary points. In order to obtain an intuitive drivable area, rich semantic information is provided through the network model. Surface normal information and images are used as input to the network model. By utilizing the spatial geometric structure information of the point cloud and the texture and color information of the image, the segmentation accuracy and performance are improved, thereby solving the problem of visual backlighting and improving the detection accuracy of the drivable area. The road boundary points are fused with the initial drivable area so that the final drivable area contains both the spatial position information of the road boundary and the semantic image information of the vehicle drivable area. This invention can provide efficient and accurate drivable area detection results in unpaved roads and semi-natural scene roads. This enables autonomous vehicles using the method of this invention to drive in the accurate drivable area, and mobile robots using the method of this invention to move in the accurate drivable area.

[0037] (2) This invention first aligns the point cloud data of the road boundary with the image plane of the drivable area through projection transformation, ensuring that the point cloud data matches the area in the image. Then, these road boundary points are fitted to form continuous road boundary lines, which makes up for the lack of spatial distance information in the drivable area, integrates the distance information of the road boundary, increases the dimension of information, and thus improves the accuracy of detection.

[0038] (3) This invention also provides a second fusion method, which transforms the road boundary points and drivable areas into a multi-boundary box fusion problem. The Hungarian algorithm is then used for accurate matching. Finally, the intersection-union ratio (IUU) is used as the evaluation criterion to improve the overall detection performance and enhance the fault tolerance of the sensor data. This method optimizes the fusion results, making the detection more accurate and reliable.

[0039] (4) The RGB encoder of this invention is responsible for capturing visual features from color images, mainly including color and texture information, while the surface normal encoder extracts geometric features from surface normal information. The output of the surface normal encoder is fused with the output of the RGB encoder. The fused features contain rich visual and geometric information, adapting to unpaved roads and semi-natural road scenes. RGB images lose some details under changes in lighting, while surface normal information provides a supplement.

[0040] (5) The decoder module of this invention employs skip connections to progressively perform upsampling and feature extraction, restoring high resolution to generate the final drivable region. Skip connections help establish richer information flow between the encoder and decoder to improve network performance.

[0041] (6) Attention mechanisms are mainly divided into channel attention mechanisms, spatial attention mechanisms, and hybrid domain attention mechanisms. Channel attention mechanisms evaluate the correlation between numerous channel features and key information, and assign corresponding weights to each channel, thereby guiding the network to pay more attention to channel features carrying important information while ignoring less relevant channel features, thus enhancing the model's performance. In addition to applying attention mechanisms in the channel dimension, operations can also be performed in the spatial dimension. Spatial attention mechanisms identify key regions in an image that contain target semantics, ensuring that the network focuses its attention on key regions containing these target semantics, guiding the network to learn these regions in depth. These two methods enhance the attention to features from different perspectives, thereby effectively improving the network's prediction accuracy. To further optimize model performance, this invention integrates these two attention mechanisms to create an innovative hybrid attention mechanism. This invention also optimizes the fused features by incorporating the hybrid attention mechanism. The network is optimized in both channel attention and spatial attention dimensions to focus more on key information while ignoring unnecessary details. This dual-dimensional optimization not only consolidates feature integration but also significantly improves the model's accuracy in identifying drivable area pixels on unpaved roads and semi-natural road scenes.

[0042] (7) To ensure the effectiveness of multi-sensor data fusion, it is necessary to ensure and achieve the unification of multi-sensor data under the same spatiotemporal reference. This implementation process is the spatiotemporal matching of multi-sensor data, also known as data alignment. This invention removes distortion from laser point clouds, which can improve the efficiency and accuracy of ground segmentation. This invention provides a variety of ground segmentation methods, among which non-uniform grid partitioning is most suitable for unpaved roads and semi-natural road scenes. A single-plane model is insufficient to represent the real unpaved roads and semi-natural road scenes, and in fact, most methods are based on the general assumption that the ground is not completely flat. Therefore, based on a multi-plane model, assuming that a small area of ​​the ground is relatively flat, it can be approximated as a plane, and the space is partitioned by a non-uniform grid to distribute appropriate cloud point density between different areas. Attached Figure Description

[0043] Figure 1 This is a flowchart of a drivable area detection method combining laser and vision provided by an embodiment of the present invention;

[0044] Figure 2 This is a flowchart of the drivable area detection provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0046] like Figure 1 As shown, a method for detecting drivable areas combining laser and vision includes:

[0047] Ground segmentation is performed on the laser point cloud in the road scene to obtain ground points and non-ground points. The road radial point cloud is obtained from the ground points. The non-ground point cloud is divided into left and right sides using the road radial point cloud. The non-ground point cloud on the left and right sides is filtered to obtain the road boundary points.

[0048] The laser point cloud in the road scene is converted into a depth map. The surface normal information of the depth map and the color image in the road scene are input into the drivable area detection network model to output the initial drivable area.

[0049] The road boundary points are merged with the initial drivable area to obtain the final drivable area;

[0050] The drivable area detection network model includes a decoder and two parallel encoders, and is trained as follows:

[0051] The surface normal information of the road depth image and the road color image marked with drivable areas are used as training samples. During training, two parallel encoders encode the surface normal information of the road depth image and the road color image respectively. The decoder decodes the encoded information output by the two parallel encoders to obtain the predicted drivable areas. The error between the predicted drivable areas and the marked drivable areas is used as the loss. The model parameters are updated by backpropagation and trained until convergence to obtain the trained drivable area detection network model.

[0052] Example 1

[0053] A method for detecting drivable areas combining laser and vision, comprising:

[0054] Ground segmentation is performed on the laser point cloud in the road scene to obtain ground points and non-ground points. The road radial point cloud is obtained from the ground points. The non-ground point cloud is divided into left and right sides using the road radial point cloud. The non-ground point cloud on the left and right sides is filtered to obtain the road boundary points.

[0055] The laser point cloud in the road scene is converted into a depth map. The surface normal information of the depth map and the color image in the road scene are input into the drivable area detection network model to output the initial drivable area.

[0056] The road boundary points are merged with the initial drivable area to obtain the final drivable area;

[0057] The drivable area detection network model includes a decoder and two parallel encoders, and is trained as follows:

[0058] The surface normal information of the road depth image and the road color image marked with drivable areas are used as training samples. During training, two parallel encoders encode the surface normal information of the road depth image and the road color image respectively. The decoder decodes the encoded information output by the two parallel encoders to obtain the predicted drivable areas. The error between the predicted drivable areas and the marked drivable areas is used as the loss. The model parameters are updated by backpropagation and trained until convergence to obtain the trained drivable area detection network model.

[0059] Road boundary points are obtained in the following way:

[0060] The region containing the laser point cloud in the road scene after distortion correction is taken as the target space region. The target space region is first divided into multiple coarse annular regions according to the polar radius. Each coarse annular region is further divided into multiple basic grid units according to the polar radius and polar angle.

[0061] For all laser point clouds in a basic grid cell, select the points with the lowest height values ​​as initial ground points and add them to the ground point set. Calculate the eigenvalues ​​and corresponding eigenvectors of the initial ground points. Use the eigenvector corresponding to the minimum eigenvalue as the ground plane normal vector. Calculate the transformation range of the initial ground points in the z-direction as the ground plane coefficient. Project the other laser point clouds in a basic grid cell, excluding the initial ground points, onto the ground plane normal vector in sequence. When the error between the distance projected onto the ground plane normal vector and the ground plane coefficient is less than a preset empirical value, add the projected laser point cloud to the ground point set.

[0062] By combining the ground point sets divided by all basic grid units, the ground points of the laser point cloud in the road scene are obtained, thereby achieving ground segmentation.

[0063] After ground segmentation of the point cloud, ground point clouds and non-ground point clouds are obtained. Road boundary points are distributed on the left and right sides of the non-ground points, which are either above or below the ground. Therefore, road boundary point extraction mainly focuses on this spatial range. Generally, the road boundary is divided into left and right sides. By using an improved beam model to analyze the road radial direction from the ground points, the non-ground point cloud is classified into candidate points for the left and right road boundaries. Then, distance filtering and RANSAC filtering are applied to the candidate points to remove noise points in and outside the boundaries. Gaussian iterative tracking is then used to obtain the road boundary points.

[0064] like Figure 2 As shown, the initial drivable area is obtained in the following way:

[0065] First, the laser point cloud and RGB image data are time-aligned using a time synchronization method. Then, the point cloud data is converted into a depth image through spatial transformation to achieve spatiotemporal matching of multi-sensor data. Next, the depth image is processed by a surface normal calculation module to obtain surface normal information, which, together with the synchronized RGB image data, serves as input to the drivable area detection network model. Finally, the detection model encodes and extracts features from the RGB image and surface normal information. The decoder module incorporates an improved feature extraction design with an attention mechanism. The decoder module progressively performs upsampling and feature extraction to restore high resolution and generate the drivable area.

[0066] The encoder consists of two parallel branch channels: an RGB encoder and a surface normal encoder. They are responsible for extracting features from the input RGB image and surface normal information, respectively, and their outputs are fused at a certain stage.

[0067] The surface normal encoder, as a component of the encoder module, is used to extract geometric features from surface normal information to help the network understand the 3D structure of the road scene. It is built based on the ResNet-50 residual network and captures the spatial structural features in the surface normal through a series of convolutional layers, batch normalization layers and activation functions, as shown in the figure.

[0068] The surface normal encoder built on the ResNet-50 residual network consists of five stages: a convolutional layer conv1 and residual block stages conv2_x, conv3_x, conv4_x, and conv5_x. The module composed of convolutional layers, batch normal (BN) layers, and ReLU activations is called standard convolution Conv2D.

[0069] Stage 1: The feature map input first passes through a 7x7 biaoz hybrid convolutional layer conv1 with a stride of 2 for downsampling, changing the size to 1 / 2x1 / 2 and increasing the number of channels to 64.

[0070] Stage 2: Next, a 3x3 max pooling layer with a stride of 2 is used to further reduce the size of the feature map, which is then used as the input to conv2_x.

[0071] Stage 3, stage 4, stage 5, and so on, with each residual block stage including a varying number of residual blocks.

[0072] ResNet-50 is a deep network with 50 layers, enabling it to learn more complex feature representations. Deep networks typically capture richer image content, which is highly helpful for understanding complex driving scenarios. By introducing residual connections, it addresses the vanishing and exploding gradient problems that arise with increasing network depth, improving the model's accuracy and generalization ability. Simultaneously, it preserves some detailed information, allowing decoding to access features from previous layers. It also boasts excellent feature extraction capabilities. These features are highly useful for downstream tasks, such as free space detection, as they provide rich spatial and semantic information.

[0073] The RGB encoder, as another component of the encoder module, is used to extract color texture features from RGB images to help the network understand scene content. It is similar in structure to the surface normal encoder and is also built based on the residual network ResNet-50. However, after each stage, the surface normal feature map and the RGB feature map are added together.

[0074] The decoder employs densely-connected skip connections, which link intermediate feature maps from the encoder to corresponding feature maps at the corresponding levels in the decoder. Densely-connected skip connections are a connection strategy used in deep neural networks, particularly in encoder-decoder architectures such as U-Net and RoadSeg. This approach aims to improve network performance by establishing richer information flow between the encoder and decoder. In traditional encoder-decoder architectures, the decoder typically connects only to the encoder's last output (i.e., the highest-level feature map). This connection, called a skip connection, helps recover spatial information lost during encoding. Densely-connected skip connections further develop this concept, connecting not only the decoder's output to the encoder's last output but also each level of the decoder to the corresponding feature map at the corresponding level in the encoder.

[0075] Attention mechanisms are primarily categorized into channel attention, spatial attention, and hybrid domain attention. Channel attention evaluates the relevance of numerous channel features to key information and assigns corresponding weights to each channel, guiding the network to focus more on channel features carrying important information while ignoring less relevant features, thus enhancing model performance. Besides applying attention mechanisms along the channel dimension, they can also be applied along the spatial dimension. Spatial attention identifies key regions in an image that contain target semantics, ensuring the network focuses its attention on these key regions and guiding the network to learn more deeply about them. These two methods enhance feature attention from different perspectives, effectively improving the network's prediction accuracy. To further optimize model performance, this invention integrates these two attention mechanisms to create an innovative hybrid attention mechanism. The fused features are fed into the decoder after passing through the hybrid attention module. The hybrid attention module, which combines channel attention and spatial attention, receives the fused features and generates two sets of channel feature descriptors through global max pooling and global average pooling operations, respectively. The two sets of channel feature descriptors are then input into the same shared fully connected layer, generating two different intermediate vectors. The two sets of intermediate vectors are directly merged, and the Sigmoid activation function is applied to generate channel weights. Finally, the weighted fused features are obtained by multiplying the fused features by the channel weights.

[0076] By utilizing the CBAM (Hybrid Attention) module, the network is optimized in both channel and spatial attention dimensions to better focus on key information while ignoring unnecessary details. This dual-dimensional optimization not only strengthens feature integration but also significantly improves the model's accuracy in identifying pixels in drivable areas of mine roads. Furthermore, fusing RGB image data and surface normal information helps the network better perform semantic segmentation tasks; therefore, these two data streams are merged during the feature integration stage. To ensure the model's real-time detection capabilities in open-pit mine environments and reduce computational burden, the feature integration stage is designed to be concise. Feature fusion is performed element-wise, and the CBAM module refines the fused features, allowing the network to focus more on learning important features while ignoring unimportant information. This further enhances the feature integration effect, thereby improving the model's accuracy in pixel classification.

[0077] The specific implementation method for merging the road boundary points with the initial drivable area is as follows:

[0078] The road boundary points are aligned with the image plane of the initial drivable area through projection transformation. Then, the road boundary points are fitted to form a continuous road boundary line. The area within the road boundary line is the final drivable area.

[0079] Example 2

[0080] Mine road scenarios are typical open outdoor environments characterized by long target distances. Conventional road detection algorithms lack spatial distance information or have significant errors, failing to meet the requirements of autonomous mining trucks. Mine roads are unstructured, with poor surface smoothness, irregular shapes, and blurred boundaries. Because they are unpaved, there are no route guidance indicators like those found on urban roads, and the road color is similar to the background, making them difficult to distinguish. When applied to autonomous mining trucks, this invention's system is installed on a two-layer platform to ensure data consistency and stability. A LiDAR is positioned in the center of the second layer, while a camera is mounted at the front of the first layer to prevent field-of-view obstruction. This provides a wide field of view, covering the vehicle's surrounding environment and capturing detailed point cloud and image data. The system includes:

[0081] LiDAR is used to collect laser point clouds in road scenes;

[0082] A camera used to capture color images of road scenes;

[0083] The road boundary point detection module is used to perform ground segmentation on the laser point cloud in the road scene to obtain ground points and non-ground points. It obtains the road radial point cloud from the ground points, uses the road radial point cloud to divide the non-ground point cloud into left and right sides, and filters the non-ground point clouds on the left and right sides to obtain the road boundary points.

[0084] The training module uses the surface normal information of the road depth image and the road color image marked with drivable areas as training samples. When training with training samples, two parallel encoders encode the surface normal information of the road depth image and the road color image respectively. The decoder decodes the encoded information output by the two parallel encoders to obtain the predicted drivable areas. The error between the predicted drivable areas and the marked drivable areas is used as the loss. The model parameters are updated by backpropagation and trained until convergence to obtain the trained drivable area detection network model.

[0085] The drivable area detection module is used to convert the laser point cloud in the road scene into a depth map, input the surface normal information of the depth map and the color image in the road scene into the drivable area detection network model, and output the initial drivable area.

[0086] The decision fusion module is used to merge road boundary points with the initial drivable area to obtain the final drivable area.

[0087] When the system of this invention is applied to autonomous mining trucks, it can accurately detect the drivable area of ​​the mine road, and the autonomous mining truck can drive in the accurate drivable area.

[0088] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting drivable areas combining laser and vision, characterized in that, include: Ground segmentation is performed on the laser point cloud in the road scene to obtain ground points and non-ground points. The road radial point cloud is obtained from the ground points. The non-ground point cloud is divided into left and right sides using the road radial point cloud. The non-ground point cloud on the left and right sides is filtered to obtain the road boundary points. The laser point cloud in the road scene is converted into a depth map. The surface normal information of the depth map and the color image in the road scene are input into the drivable area detection network model to output the initial drivable area. The road boundary points are merged with the initial drivable area to obtain the final drivable area; The drivable area detection network model includes a decoder and two parallel encoders, and is trained as follows: The surface normal information of the road depth image and the road color image marked with drivable areas are used as training samples. During training, two parallel encoders encode the surface normal information of the road depth image and the road color image respectively. The decoder decodes the encoded information output by the two parallel encoders to obtain the predicted drivable areas. The error between the predicted drivable areas and the marked drivable areas is used as the loss. The model parameters are updated by backpropagation and trained until convergence to obtain the trained drivable area detection network model.

2. The method for detecting drivable areas combining laser and vision as described in claim 1, characterized in that, The specific implementation method for merging the road boundary points with the initial drivable area is as follows: The road boundary points are aligned with the image plane of the initial drivable area through projection transformation. Then, the road boundary points are fitted to form a continuous road boundary line. The area within the road boundary line is the final drivable area.

3. The method for detecting drivable areas combining laser and vision as described in claim 1, characterized in that, The specific implementation method for merging the road boundary points with the initial drivable area is as follows: Extract the maximum and minimum corner coordinates from the bounding box of the point cloud containing road boundary points to form a point cloud rectangle; extract the maximum and minimum corner coordinates from the bounding box of the image containing the initial drivable area to form an image rectangle. Calculate the intersection-union ratio (IUR) of the point cloud bounding box and the image bounding box. When the IUR is greater than or equal to a set threshold, the overlapping area of ​​the point cloud bounding box and the image bounding box is taken as the final drivable area. When the IUR is less than the set threshold, the area within the road boundary line fitted by the road boundary points or the initial drivable area is taken as the final drivable area.

4. A method for detecting drivable areas combining laser and vision as described in any one of claims 1-3, characterized in that, The two parallel encoders include a surface normal encoder and an RGB encoder. The surface normal encoder includes multiple units for encoding geometric features extracted from surface normal information. The RGB encoder includes multiple units for encoding color texture features extracted from road color images. The output of each unit in the surface normal encoder is fused with the output of the corresponding unit in the RGB encoder to obtain a fused feature.

5. The drivable area detection method combining laser and vision as described in claim 4, characterized in that, The decoder includes multiple upsampling modules and multiple feature extraction modules. Skip connections are used between the upsampling modules and the feature extraction modules, as well as between the upsampling modules and the units in the RGB encoder. The upsampling modules upsample the fused features and then input them into the feature extraction modules, which then decode the input data.

6. The method for detecting drivable areas combining laser and vision as described in claim 5, characterized in that, The fused features are fed into the decoder after passing through the hybrid attention module. The hybrid attention module, which combines channel attention and spatial attention, receives the fused features and generates two sets of channel feature descriptors by performing global max pooling and global average pooling operations respectively. The two sets of channel feature descriptors are then input into the same shared fully connected layer, generating two different intermediate vectors. The two sets of intermediate vectors are directly merged, and the Sigmoid activation function is applied to generate channel weights. Finally, the weighted fused features are obtained by multiplying the fused features by the channel weights.

7. A method for detecting drivable areas combining laser and vision as described in any one of claims 1-3, characterized in that, The method further includes: During detection, the laser point cloud and color image in the road scene are time-aligned, and then the laser point cloud in the road scene is converted into a depth map through spatial transformation to complete the spatiotemporal matching of the laser point cloud and the color image. During training, the road depth images and road color images in the training samples have already been spatiotemporally matched.

8. A method for detecting drivable areas combining laser and vision as described in any one of claims 1-3, characterized in that, After distortion correction of the laser point cloud in the road scene, ground segmentation is achieved through plane fitting, normal vector method, planar grid method or non-uniform grid division.

9. The method for detecting drivable areas combining laser and vision as described in claim 8, characterized in that, The specific method for non-uniform grid division is as follows: The region containing the laser point cloud in the road scene after distortion correction is taken as the target space region. The target space region is first divided into multiple coarse annular regions according to the polar radius. Each coarse annular region is further divided into multiple basic grid units according to the polar radius and polar angle. For all laser point clouds in a basic grid cell, select the points with the lowest height values ​​as initial ground points and add them to the ground point set. Calculate the eigenvalues ​​and corresponding eigenvectors of the initial ground points. Use the eigenvector corresponding to the minimum eigenvalue as the ground plane normal vector. Calculate the transformation range of the initial ground points in the z-direction as the ground plane coefficient. Project the other laser point clouds in a basic grid cell, excluding the initial ground points, onto the ground plane normal vector in sequence. When the error between the distance projected onto the ground plane normal vector and the ground plane coefficient is less than a preset empirical value, add the projected laser point cloud to the ground point set. By combining the ground point sets divided by all basic grid units, the ground points of the laser point cloud in the road scene are obtained, thereby achieving ground segmentation.

10. A drivable area detection system combining laser and vision, characterized in that, include: LiDAR is used to collect laser point clouds in road scenes; A camera used to capture color images of road scenes; The road boundary point detection module is used to perform ground segmentation on the laser point cloud in the road scene to obtain ground points and non-ground points. It obtains the road radial point cloud from the ground points, uses the road radial point cloud to divide the non-ground point cloud into left and right sides, and filters the non-ground point clouds on the left and right sides to obtain the road boundary points. The training module uses the surface normal information of the road depth image and the road color image marked with drivable areas as training samples. When training with training samples, two parallel encoders encode the surface normal information of the road depth image and the road color image respectively. The decoder decodes the encoded information output by the two parallel encoders to obtain the predicted drivable areas. The error between the predicted drivable areas and the marked drivable areas is used as the loss. The model parameters are updated by backpropagation and trained until convergence to obtain the trained drivable area detection network model. The drivable area detection module is used to convert the laser point cloud in the road scene into a depth map, input the surface normal information of the depth map and the color image in the road scene into the drivable area detection network model, and output the initial drivable area. The decision fusion module is used to merge road boundary points with the initial drivable area to obtain the final drivable area.

Images