Gravel terrain drivable region segmentation method

Abstract

The application discloses a gravel terrain passable area segmentation method. The original RGB image and the surface normal vector picture are subjected to down-sampling coding and feature fusion and further coding, and finally, the multi-scale feature map is up-sampled to obtain a result with the same resolution size as the original image. The original RGB image and the surface normal vector image are taken as network input and subjected to convolution down-sampling, up-sampling and other operations to complete segmentation of the passable area, so that the training process is more efficient, and the segmentation result is more accurate.

Description

Technical Field

[0001] This invention belongs to the field of robot navigation and positioning. Specifically, it is a method for fusing three-dimensional data with two-dimensional data to analyze traversable areas. Background Technology

[0002] Traditional environmental perception methods rely on single sensors, such as LiDAR or cameras. LiDAR is widely used in map building and obstacle detection due to its high accuracy and anti-interference capabilities. However, LiDAR also has limitations, such as insensitivity to objects with low reflectivity and significant susceptibility to environmental factors. Alternatively, using visual methods to analyze passable areas is another existing approach, segmenting the surrounding passable area through large-scale model calculations. However, this method is easily affected by factors such as weather and lighting conditions.

[0003] Due to the limitations of single sensors, the fusion of LiDAR and cameras has received widespread attention in recent years. On one hand, LiDAR point clouds provide rich spatial information, including distance and angle. On the other hand, images provide rich semantic information. Therefore, the advantages of both can be combined to achieve accessibility detection of an area. Outdoor environments often have more complex terrain features, such as uneven ground, rocks, and vegetation. LiDAR can provide high-resolution 3D point cloud data, helping autonomous vehicles accurately perceive surrounding objects, terrain, and obstacles in such environments. In outdoor environments, weather conditions such as rain, snow, and fog can affect the performance of visual sensors, while LiDAR's laser sensing technology can operate under a wider range of weather conditions, ensuring all-weather operation. By fusing LiDAR and cameras, autonomous driving systems can utilize the precise ranging of LiDAR and the rich visual information of cameras to achieve more comprehensive and reliable environmental perception. Summary of the Invention

[0004] The purpose of this invention is to provide a method for segmenting traversable areas in gravel terrain. This method involves projecting a 3D radar point cloud onto a front view to generate a normal vector map, and then fusing the original RGB image with the normal vector map to obtain the traversable area segmentation result.

[0005] The technical solution to achieve the purpose of this invention is: a method for segmenting traversable areas in gravel terrain, comprising the following steps:

[0006] The first step is to input the original RGB image of size H*W and its corresponding surface normal vector image into the network;

[0007] The second step involves feeding both images into the Patch Embedding layer to divide them into multiple small blocks. Each block is then embedded into a low-dimensional feature space and preliminarily encoded to extract their respective features. By combining the features of the RGB image and the normal vector map, a weight learning module is used. The two features are then fused through an attention mechanism to learn the correlation between the RGB image and the normal vector map, generating a feature map of H / 4*W / 4*C1.

[0008] The third step is to merge the images output from the second step, and then process the merged feature map through two VSS blocks to complete the image encoding task. The image features are further extracted through image encoding. The input feature map is processed through two VSS blocks three times to generate multi-scale feature maps.

[0009] The fourth step involves upsampling the multi-scale feature maps obtained in the previous steps through Patch Expanding four times to gradually restore the resolution of the original RGB image. After each upsampling round, a 1*1 convolution operation is performed to adjust the number of channels. Through the deep supervision mechanism, features are extracted from different stages for supervision to ensure that each stage contributes to the final output. After these steps, a segmentation result with the same size as the input image is generated.

[0010] Furthermore, in the second step, the specific implementation steps of the input image encoding process are as follows:

[0011] 1) The input RGB image and surface normal vector image are divided into multiple small blocks, and each block is then embedded into a fixed-dimensional vector representation;

[0012] 2) The vectors obtained in the previous step are fed into the VSS encoder twice. For the original RGB image, texture and edge features are extracted. For the surface normal vector image, three-dimensional spatial features are extracted.

[0013] 3) After the initial feature extraction, the RGB image and surface normal vector features interact through a Cross-Attention module, and the two modalities are weighted and fused to dynamically assign weights to the RGB and normal vector features; the resulting dimension is H / 4*W / 4*C1.

[0014] Furthermore, in the third step, the feature map is further encoded, and the specific implementation steps are as follows:

[0015] 1) Merge the H / 4*W / 4*C1 feature images obtained in the previous stage to reduce the image resolution and increase the channel dimension. After passing through two VSS blocks, further extract the internal features of the fused image. The feature map of the input dimension H / 4*W / 4*C1 is output as the feature map of H / 8*W / 8*C2.

[0016] 2) The feature map of H / 8*W / 8*C2 is downsampled again to halve the spatial size of the image features; after processing by two VSS blocks, multi-scale features are further extracted, and the feature map with the input dimension of H / 8*W / 8*C2 is output as a feature map of H / 16*W / 16*C3.

[0017] 3) Perform a final downsampling on the above output feature map to further reduce the spatial size. Then, process it through two VSS blocks to output a feature map with an input dimension of H / 16*W / 16*C3 as H / 32*W / 32*C4.

[0018] Furthermore, in the fourth step, the multi-scale feature map is upsampled and decoded, and the specific implementation steps are as follows:

[0019] 1) The obtained multi-scale feature map is upsampled by the Patch Expanding module to restore the feature dimension map of H / 32*W / 32*C4 to H / 16*W / 16*C3. After passing through two layers of VSS blocks, the number of feature map channels is controlled by 1*1Conv.

[0020] 2) The feature dimension map of H / 16*W / 16*C3 obtained above is restored to H / 8*W / 8*C2 dimension by passing it through the Patch Expanding module, two VSS blocks, 1*1Conv and Deep Supervision respectively;

[0021] 3) The H / 8*W / 8*C2 dimension feature map obtained from the previous layer is processed through two VSS blocks, 1*1Conv and DeepSupervision, to obtain an H / 4*W / 4*C1 dimension feature map;

[0022] 4) Decode the obtained H / 4*W / 4*C1 dimension feature map, use Patch Expanding to upsample the feature map, and then gradually restore it to the original resolution of the input image through 1*1 Conv.

[0023] Compared with the prior art, the present invention has the following significant advantages: (1) High accuracy: The combination of Patch Merging and VSSBlock modules can obtain higher-dimensional information, and the segmentation results will be more accurate. (2) High efficiency: The upsampling module is replaced by patch expanding and two VSS Blocks, which can effectively reduce the number of network parameters and computational overhead. Attached Figure Description

[0024] Figure 1 This is the original RGB image.

[0025] Figure 2 This is a surface normal vector image.

[0026] Figure 3 This refers to the image encoding process.

[0027] Figure 4 This is the process of decoding a multi-scale dimensional graph.

[0028] Figure 5 The results show the passable area.

[0029] Figure 6 Flowchart of a method for segmenting passable areas in gravel terrain. Detailed Implementation

[0030] This invention proposes a method for segmenting passable areas in gravel terrain. By downsampling and encoding the original RGB image and the surface normal vector image, and then further encoding them, the multi-scale feature map is upsampled to obtain a result with the same resolution as the original image. This step can efficiently detect passable areas with high real-time performance and good segmentation effect.

[0031] The present invention will now be further described with reference to the accompanying drawings.

[0032] The present invention provides a method for segmenting traversable areas in gravel terrain, the specific implementation of which includes the following steps:

[0033] The first step is to input the original RGB image of size H*W and its corresponding surface normal vector image into the network.

[0034] The second step involves feeding both images into a Patch Embedding layer, dividing them into multiple small blocks. Each block is then embedded into a low-dimensional feature space and preliminarily encoded to extract its respective features. These features are then combined with those of the RGB image and the normal vector map. Through a weight learning module, the two features are fused using an attention mechanism to learn the correlation between the RGB image and the normal vector map. This generates a feature map of H / 4*W / 4*C1.

[0035] See also Figure 1 Original RGB image and see Figure 2 The downsampling and feature fusion process of the surface normal vector image is as follows:

[0036] 1) The input RGB image and surface normal vector image are divided into multiple small blocks, and each block is then embedded into a vector representation of a fixed dimension.

[0037] 2) The vectors obtained in the previous step are fed into two VSS encoders. The function of this encoder is to extract texture and edge features from the original RGB image and to extract 3D spatial features from the surface normal vector image. This captures both local and global features. For segmentation tasks, the extracted edge information is very important for segmentation, helping the model find a balance between fine-grained and global context.

[0038] 3) After initial feature extraction, the RGB image and surface normal vector features interact through a Cross-Attention module. This step uses an attention mechanism to allow the data from the two modalities to learn from each other, fully utilizing their complementary information, and weightedly fusing the two modalities, dynamically assigning weights to the RGB and normal vector features. The resulting dimension is H / 4*W / 4*C1.

[0039] The third step involves merging the images output from the second step to reduce the image resolution while increasing the channel dimension. These images are then processed through two VSS blocks to further extract image features. This process is repeated three times, applying the input feature map through two VSS blocks to generate multi-scale feature maps.

[0040] The process of further encoding the fusion result into higher-dimensional feature maps can be found in [link to documentation]. Figure 3 Specifically:

[0041] 1) Merge the output images obtained in the previous stage to reduce the image resolution and increase the channel dimension. After passing through two VSS blocks, further extract the image features. The feature map with the input dimension H / 4*W / 4*C1 is output as the feature map H / 8*W / 8*C2.

[0042] 2) The H / 8*W / 8*C2 feature map is downsampled again to halve the spatial size of the image features. After processing by two VSS blocks, multi-scale features are further extracted, and the feature map with the input dimension of H / 8*W / 8*C2 is output as a feature map with the dimension of H / 16*W / 16*C3.

[0043] 3) The last layer is downsampled to further reduce the spatial size. After processing by two VSS blocks, the feature map with the input dimension of H / 16*W / 16*C3 is output as a feature map of H / 32*W / 32*C4.

[0044] The fourth step involves upsampling the downsampled feature maps obtained in the previous steps using Patch Expanding to gradually restore the resolution to the original RGB image. After each round of upsampling, a 1x1 convolution operation is performed to adjust the number of channels. Deep supervision is then used to extract features from different stages for oversight, ensuring that each stage contributes to the final output. After these steps, a segmentation result with the same size as the input image is generated.

[0045] For multi-scale feature maps, the upsampling process restores them to the original image size; see [link to process details]. Figure 4 Specifically:

[0046] 1) The obtained multi-scale feature maps are upsampled by the Patch Expanding module to restore the feature dimension map of H / 32*W / 32*C4 to H / 16*W / 16*C3. After passing through two layers of VSS blocks, the expression of contextual information is preserved and enhanced during the restoration of high resolution. The number of feature map channels can be controlled by 1*1Conv, which reduces redundant information when the model restores high resolution. Through the deep supervision mechanism, the model can be guided to optimize feature learning at different depth levels during model training.

[0047] 2) Based on the feature dimension map of H / 16*W / 16*C3 obtained above, it is restored to H / 8*W / 8*C2 dimension by passing through the Patch Expanding module, two VSS blocks, 1*1Conv and Deep Supervision respectively.

[0048] 3) The H / 8*W / 8*C2 dimension feature map obtained from the previous layer is processed through two VSS blocks, 1*1Conv and DeepSupervision, to obtain an H / 4*W / 4*C1 dimension feature map.

[0049] 4) Decode the obtained H / 4*W / 4*C1 dimensional feature map, upsample the feature map using Patch Expanding, and then gradually restore it to the original resolution of the input image using 1*1 Conv. See [link to documentation]. Figure 5 This is the segmentation result.

[0050] The present invention will be further described below with reference to the embodiments.

[0051] This embodiment is based on the following hardware platform: ① Data Acquisition Platform: CPU is Intel Core i5-8259U; Memory is DDR4 2400MHz 16GB; Storage is Kioxia EXCERIA G2 SSD 500GB; Camera is Hikvision USB Camera DS-2CS54U0B-SD; LiDAR is Livox HAP LiDAR; The USB camera, LiDAR, and IMU are connected to the Smart Box shown in the figure. The Smart Box is connected to the mobile robot control chassis via USB. The power required by the device is provided by the lithium battery carried by the mobile robot itself. ② Experimental Platform: CPU is Intel Core i5-8259U; Memory is DDR4 2400MHz 16GB; Storage is Kioxia EXCERIA G2 SSD 500GB.

[0052] This embodiment is based on the following software environment: Computer operating system: Ubuntu 18.04, paired with the ROS Melodic robot operating system.

[0053] Combination Figure 6 This invention provides a method for segmenting traversable areas in gravel terrain, and the specific implementation steps are as follows:

[0054] The first step is to input the original RGB image of size H*W and its corresponding surface normal vector image into the network.

[0055] The second step involves feeding both images into a Patch Embedding layer, dividing them into multiple small blocks. Each block is then embedded into a low-dimensional feature space and preliminarily encoded to extract its respective features. These features are then combined with those of the RGB image and the normal vector map. Through a weight learning module, the two features are fused using an attention mechanism to learn the correlation between the RGB image and the normal vector map. This generates a feature map of H / 4*W / 4*C1.

[0056] The third step involves merging the images output from the second step to reduce the image resolution while increasing the channel dimension. These images are then processed through two VSS blocks to further extract image features. This process is repeated three times, applying the input feature map through two VSS blocks to generate multi-scale feature maps.

[0057] The fourth step involves performing four rounds of upsampling using Patch Expanding on the downsampled feature maps obtained in the previous steps, gradually restoring the resolution to the original RGB image. After each round of upsampling, a 1x1 convolution operation is performed to adjust the number of channels. Deep supervision is then used to extract features from different stages for oversight, ensuring that each stage contributes to the final output. After these steps, a segmentation result with the same size as the input image is generated.

Claims

1. A gravel terrain passable area segmentation method characterized by, The method comprises the following steps: First, the original RGB image with a size of H*W and the surface normal vector image corresponding thereto are input into the network as inputs; Second, the two are respectively sent into a Patch Embedding layer to be divided into multiple small blocks, and then each block is embedded into a low-dimensional feature space and respectively encoded to extract respective features. The features of the RGB image and the normal vector image are combined, and the two features are fused through an attention mechanism to learn the correlation between the RGB image and the normal vector image; A feature map with a size of H / 4*W / 4*C1 is generated; Third, the image output in the second step is merged, and the merged feature map is subjected to calculation and processing of two VSS blocks to complete the image encoding task and further extract image features. The input feature map is subjected to calculation and processing of two VSS blocks for three times to generate multiple scale feature maps respectively; Fourth, the multiple scale feature maps obtained in the above steps are subjected to four rounds of upsampling through Patch Expanding, and are gradually restored to the resolution of the original RGB image. After each round of upsampling, a 1*1 convolution operation is performed to adjust the number of channels. Deep supervision is performed on the features extracted from different stages to ensure that each stage contributes to the final output. After these steps, a segmentation result with the same size as the input image is generated.

2. The gravel-terrain passable area segmentation method according to claim 1, characterized by, In the second step, the encoding process of the image is specifically implemented as follows: 1) The input RGB image and the surface normal vector image are divided into multiple small blocks, and then each block is embedded into a fixed-dimensional vector representation; 2) The vectors obtained in the above step are input into the two VSS encoders. The texture and edge features in the original RGB image are extracted, and the three-dimensional space features of the surface normal vector image are extracted; 3) After preliminary feature extraction, the RGB image and the surface normal vector feature are interacted through a Cross-Attention module, and the two modalities are weighted and fused to dynamically assign weights to the RGB and normal vector features. The obtained dimension is H / 4*W / 4*C1.

3. The method of claim 1, wherein, In the third step, the specific steps of further encoding the image are as follows: 1) The H / 4*W / 4*C1 feature image output in the previous stage is merged to reduce the image resolution and increase the channel dimension. The internal features of the fused image are further extracted through two VSS blocks. The input dimension of the feature map is H / 4*W / 4*C1, and the output is a feature map with a dimension of H / 8*W / 8*C2; 2) The H / 8*W / 8*C2 feature map is downsampled again to halve the spatial size of the image features. The multiple scale features are further extracted through two VSS block calculation and processing. The input dimension of the feature map is H / 8*W / 8*C2, and the output is a feature map with a dimension of H / 16*W / 16*C3; 3) The last down-sampling is performed on the output feature map, further reducing the spatial size, and then two VSS blocks are used to process the feature map with the input dimension of H / 16*W / 16*C3 to output a feature map with the dimension of H / 32*W / 32*C4.

4. The method of claim 1, wherein, The specific steps of the fourth step are: 1) The obtained multi-scale feature map is up-sampled by the Patch Expanding module to restore the feature map with the dimension of H / 32*W / 32*C4 to the dimension of H / 16*W / 16*C3, and then two VSS blocks and a 1*1Conv are used to control the number of feature map channels; 2) The obtained feature map with the dimension of H / 16*W / 16*C3 is respectively processed by the Patch Expanding module, twice VSS blocks, a 1*1Conv and Deep Supervision to restore the dimension to H / 8*W / 8*C2; 3) The obtained feature map with the dimension of H / 8*W / 8*C2 is processed by twice VSS blocks, a 1*1Conv and Deep Supervision to obtain a feature map with the dimension of H / 4*W / 4*C1; 4) The obtained feature map with the dimension of H / 4*W / 4*C1 is decoded, the feature map is up-sampled by the Patch Expanding, and then a 1*1Conv is used to gradually restore the original resolution of the input image.

Images