Gravel terrain passable area segmentation method

Abstract

The invention discloses a gravel terrain passable area segmentation method. Down-sampling coding and feature fusion are carried out on an original RGB image and a surface normal vector picture, then coding is further carried out, and finally, a result with the same resolution as that of the original image is obtained after up-sampling is carried out on a multi-scale feature map. According to the method, the original RGB image and the surface normal vector image serve as network input, the passable area is segmented through operations such as convolution down-sampling and up-sampling, the training process can be ensured to be more efficient, and the segmentation result is more accurate.

Description

Technical Field

[0001] The invention belongs to the field of robot navigation and positioning, and in particular, is a method for fusing three-dimensional data with two-dimensional data to analyze a traversable area. Background Art

[0002] Traditional environmental perception methods rely on a single sensor, such as LiDAR or a camera. LiDAR is widely used in map construction, obstacle detection, etc. due to its high accuracy and anti-interference ability. However, LiDAR also has some limitations, such as insensitivity to objects with low reflectivity and being greatly affected by environmental factors. In addition, using visual methods to analyze the passable area is also one of the existing methods. The surrounding passable area can be segmented through the calculation of a large model. However, this method is easily affected by factors such as weather and lighting.

[0003] Due to the limitations of a single sensor, the method of fusing LiDAR with a camera has also received widespread attention in recent years. On the one hand, radar point clouds can provide rich spatial information, including distance, angle, etc. On the other hand, images can provide rich semantic information. Therefore, the advantages of the two can be combined to achieve the passability detection of the area. The wild environment often has more complex terrain features, such as uneven ground, rocks, vegetation, etc. LiDAR can provide high-resolution three-dimensional point cloud data to help autonomous vehicles accurately perceive surrounding objects, terrain and obstacles in such an environment. In the wild environment, weather conditions such as rain, snow, fog and haze can affect the performance of visual sensors, while LiDAR's laser sensing technology can work under a wider range of meteorological conditions, ensuring that the system can operate around the clock. By fusing LiDAR and cameras, autonomous driving systems can use LiDAR's precise ranging and camera's rich visual information to achieve more comprehensive and reliable environmental perception. Summary of the invention

[0004] The purpose of the invention is to provide a method for segmenting the traversable area on gravel terrain, by projecting a three-dimensional radar point cloud onto a front view and generating a normal vector map, and then fusing the original RGB image with the normal vector map to finally obtain a traversable area segmentation result.

[0005] The technical solution to achieve the purpose of the present invention is: a method for segmenting a traversable area on gravel terrain, comprising the following steps:

[0006] In the first step, the original RGB image of size H*W and its corresponding surface normal vector image are input into the network;

[0007] In the second step, the two are sent to the Patch Embedding layer to be divided into multiple small blocks. Each block is then embedded into a low-dimensional feature space and preliminarily encoded to extract their respective features. The features of the RGB image and the normal vector map are combined. Through the weight learning module, the two features are fused through the attention mechanism to learn the correlation between the RGB image and the normal vector map. The feature map of H / 4*W / 4*C1 is generated.

[0008] In the third step, the images output in the second step are merged, and the merged feature map is processed by two VSS blocks to complete the image coding task, and the image features are further extracted by encoding the image; the input feature map is processed by two VSS blocks three times to generate multi-scale feature maps respectively;

[0009] In the fourth step, the multi-scale feature map obtained in the above steps is upsampled four times through Patch Expanding to gradually restore the resolution of the original RGB image. After each round of upsampling, a 1*1 convolution operation is performed to adjust the number of channels. Through the deep supervision mechanism Deep Supervision, features are extracted from different stages for supervision to ensure that each stage contributes to the final output. After these steps, a segmentation result consistent with the size of the input image is generated.

[0010] Furthermore, in the second step, the encoding process of the input image is specifically implemented as follows:

[0011] 1) The input RGB image and surface normal vector image are divided into multiple small blocks, and then each block is embedded into a vector representation of fixed dimension;

[0012] 2) The vector obtained in the previous step is input into the VSS encoder twice to extract the texture and edge features in the original RGB image and the three-dimensional spatial features in the surface normal vector image;

[0013] 3) After preliminary feature extraction, the RGB image and surface normal features interact through a Cross-Attention module, and the two modalities are weighted fused to dynamically assign weights to the RGB and normal features; the resulting dimension is H / 4*W / 4*C1.

[0014] Furthermore, in the third step, the feature map is further encoded, and the specific implementation steps are:

[0015] 1) The feature images of H / 4*W / 4*C1 obtained in the previous stage are merged to reduce the image resolution and increase the channel dimension. The internal features of the fused image are further extracted after passing through two VSS blocks. The feature map of the above input dimension H / 4*W / 4*C1 is output as a feature map of H / 8*W / 8*C2;

[0016] 2) Downsample the feature map of H / 8*W / 8*C2 again to halve the spatial size of the image features; then, after two VSS blocks are calculated and processed, multi-scale features are further extracted, and the feature map with the input dimension of H / 8*W / 8*C2 is output as a feature map of H / 16*W / 16*C3;

[0017] 3) The feature map outputted above is downsampled for the last time to further reduce the spatial size, and then processed by two VSS blocks, so that the feature map with the input dimension of H / 16*W / 16*C3 is output as a feature map of H / 32*W / 32*C4.

[0018] Furthermore, in the fourth step, the multi-scale feature map is upsampled and decoded, and the specific implementation steps are as follows:

[0019] 1) The obtained multi-scale feature map is upsampled to the low-resolution feature map through the Patch Expanding module, and the feature dimension map of H / 32*W / 32*C4 is restored to H / 16*W / 16*C3, passing through two layers of VSS blocks and controlling the number of feature map channels through 1*1Conv;

[0020] 2) The feature dimension map of H / 16*W / 16*C3 obtained above is restored to H / 8*W / 8*C2 dimension through Patch Expanding module, two VSS blocks, 1*1Conv and Deep Supervision respectively;

[0021] 3) The feature map of the H / 8*W / 8*C2 dimension obtained in the previous layer is passed through two VSS blocks, 1*1Conv and DeepSupervision to obtain a feature map of the H / 4*W / 4*C1 dimension;

[0022] 4) Decode the obtained H / 4*W / 4*C1 dimensional feature map, upsample the feature map using Patch Expanding, and then gradually restore it to the original resolution of the input image through 1*1Conv.

[0023] Compared with the prior art, the present invention has the following significant advantages: (1) High accuracy: The combination of Patch Merging and VSSBlock modules can obtain higher-dimensional information, and the segmentation result will be more accurate. (2) High efficiency: The upsampling module is replaced by patch expanding and 2 VSS Blocks, which can effectively reduce the network parameters and computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is the original RGB image.

[0025] Figure 2 is the surface normal vector image.

[0026] Figure 3 The picture encoding process.

[0027] Figure 4 It is the multi-scale dimension graph decoding process.

[0028] Figure 5 This is the detection result of the passable area.

[0029] Figure 6 Flowchart of the method for segmenting traversable areas on gravel terrain. DETAILED DESCRIPTION

[0030] The present invention proposes a method for segmenting traversable areas in gravel terrain. The original RGB image and the surface normal vector image are downsampled and encoded, and feature fusion is further encoded. Finally, the multi-scale feature map is upsampled to obtain a result with the same resolution as the original image. This step can efficiently complete the detection of traversable areas with high real-time performance and good segmentation effect.

[0031] The present invention will be further described below in conjunction with the accompanying drawings.

[0032] The present invention provides a method for segmenting a traversable area on gravel terrain, and its specific implementation method comprises the following steps:

[0033] In the first step, the original RGB image of size H*W and its corresponding surface normal vector image are input into the network.

[0034] In the second step, the two are sent to the Patch Embedding layer to be divided into multiple small blocks, and then each block is embedded into a low-dimensional feature space and preliminarily encoded to extract their respective features. The features of the RGB image and the normal vector map are combined, and the two features are fused through the attention mechanism through the weight learning module to learn the correlation between the RGB image and the normal vector map. Generate a feature map of H / 4*W / 4*C1.

[0035] See also Figure 1 Original RGB image and see Figure 2 The downsampling and feature fusion process of the surface normal vector image is as follows:

[0036] 1) The input RGB image and surface normal vector image are divided into multiple small blocks, and then each block is embedded into a vector representation of fixed dimension.

[0037] 2) The vector obtained in the previous step enters the VSS encoder twice. The encoder extracts the texture and edge features in the original RGB image and extracts the three-dimensional spatial features from the surface normal vector image. This captures local and global features. For segmentation tasks, the extracted edge information is very important for segmentation and can help the model find a balance between fine-grained and global context.

[0038] 3) After the initial feature extraction, the RGB image and surface normal features interact through a Cross-Attention module. This step allows the two modal data to learn from each other through the attention mechanism, making full use of their complementary information, and performing weighted fusion on the two modalities, dynamically assigning weights to the RGB and normal features. The resulting dimension is H / 4*W / 4*C1.

[0039] In the third step, the images output from the second step are merged to reduce the image resolution and increase the channel dimension, and the image features are further extracted after being processed by two VSS blocks. The input feature map is processed by two VSS blocks three times to generate multi-scale feature maps.

[0040] The fusion result is further encoded to generate a higher-dimensional feature map. The process is shown in Figure 3 Specifically:

[0041] 1) The output images obtained in the previous stage are merged to reduce the image resolution and increase the channel dimension. After passing through two VSS blocks, the image features are further extracted. The feature map of the above input dimension H / 4*W / 4*C1 is output as a feature map of H / 8*W / 8*C2.

[0042] 2) Downsample the feature map of H / 8*W / 8*C2 again to halve the spatial size of the image features. After two VSS blocks are processed, multi-scale features are further extracted, and the feature map with the input dimension of H / 8*W / 8*C2 is output as a feature map of H / 16*W / 16*C3.

[0043] 3) The last layer is downsampled to further reduce the spatial size, and then processed by two VSS blocks. The feature map with the above input dimension of H / 16*W / 16*C3 is output as a feature map of H / 32*W / 32*C4.

[0044] In the fourth step, the downsampled feature map obtained in the above steps is upsampled through Patch Expanding, and gradually restored to the resolution of the original RGB image. After each round of upsampling, a 1*1 convolution operation is performed to adjust the number of channels. Through the deep supervision mechanism (Deep Supervision), features are extracted from different stages for supervision to ensure that each stage contributes to the final output. After these steps, a segmentation result consistent with the size of the input image is generated.

[0045] For the multi-scale feature map, the upsampling is restored to the original image size. The process is shown in Figure 4 Specifically:

[0046] 1) The obtained multi-scale feature map is upsampled from the low-resolution feature map through the Patch Expanding module, and the feature dimension map of H / 32*W / 32*C4 is restored to H / 16*W / 16*C3. It passes through two layers of VSS blocks to ensure that the expression of contextual information is retained and enhanced in the process of restoring high resolution. The number of feature map channels can be controlled by 1*1Conv, so that the model can reduce redundant information when restoring high resolution. The deep supervision mechanism can guide the model to optimize feature learning at different depth levels during model training.

[0047] 2) The feature dimension map of H / 16*W / 16*C3 obtained above is restored to H / 8*W / 8*C2 dimension through Patch Expanding module, two VSS blocks, 1*1Conv and Deep Supervision respectively.

[0048] 3) The feature map of dimension H / 8*W / 8*C2 obtained in the previous layer is passed through two VSS blocks, 1*1Conv and DeepSupervision to obtain a feature map of dimension H / 4*W / 4*C1.

[0049] 4) Decode the obtained H / 4*W / 4*C1 dimensional feature map, upsample the feature map using Patch Expanding, and then gradually restore it to the original resolution of the input image through 1*1Conv. Figure 5 is the segmentation result.

[0050] The present invention will be further described below in conjunction with the embodiments.

[0051] This embodiment is based on the following hardware platform. ① Acquisition platform: CPU is Intel Core i5-8259U; memory is DDR4 2400MHz 16GB; storage is Kioxia EXCERIA G2 SSD 500GB; camera is Hikvision USB camera DS-2CS54U0B-SD; laser radar is Livox HAP laser radar; USB camera, laser radar, IMU are connected to the smart box shown in the figure, and the smart box is connected to the mobile robot control chassis via USB. The power required by the device is provided by the lithium battery carried by the mobile robot itself. ② Experimental platform: CPU is Intel Core i5-8259U; memory is DDR4 2400MHz 16GB, and storage is Kioxia EXCERIA G2 SSD 500GB.

[0052] This embodiment is based on the following software environment: Computer operating system: Ubuntu 18.04, with ROS Melodic robot operating system.

[0053] Combination Figure 6 The present invention is a method for segmenting traversable areas on gravel terrain, and the specific implementation steps are as follows:

[0054] In the first step, the original RGB image of size H*W and its corresponding surface normal vector image are input into the network.

[0055] In the second step, the two are sent to the Patch Embedding layer to be divided into multiple small blocks, and then each block is embedded into a low-dimensional feature space and preliminarily encoded to extract their respective features. The features of the RGB image and the normal vector map are combined, and the two features are fused through the attention mechanism through the weight learning module to learn the correlation between the RGB image and the normal vector map. Generate a feature map of H / 4*W / 4*C1.

[0056] In the third step, the images output from the second step are merged to reduce the image resolution and increase the channel dimension, and the image features are further extracted after being processed by two VSS blocks. The input feature map is processed by two VSS blocks three times to generate multi-scale feature maps.

[0057] In the fourth step, the downsampled feature map obtained in the above steps is upsampled four times through Patch Expanding to gradually restore the resolution of the original RGB image. After each upsampling, a 1*1 convolution operation is performed to adjust the number of channels. Through the deep supervision mechanism (Deep Supervision), features are extracted from different stages for supervision to ensure that each stage contributes to the final output. After these steps, a segmentation result consistent with the size of the input image is generated.

Claims

1. A method for segmenting traversable areas on gravel terrain, characterized in that: The following steps are involved: In the first step, the original RGB image of size H*W and its corresponding surface normal vector image are input into the network; In the second step, the two are sent to the Patch Embedding layer to be divided into multiple small blocks. Each block is then embedded into a low-dimensional feature space and preliminarily encoded to extract their respective features. The features of the RGB image and the normal map are combined through the weight learning module. The two features are fused through the attention mechanism to learn the correlation between the RGB image and the normal map. Generate a feature map of H / 4*W / 4*C1; In the third step, the images output in the second step are merged, and the merged feature map is processed by two VSS blocks to complete the image coding task, and the image features are further extracted by encoding the image; the input feature map is processed by two VSS blocks three times to generate multi-scale feature maps respectively; In the fourth step, the multi-scale feature map obtained in the above steps is upsampled four times through Patch Expanding to gradually restore the resolution of the original RGB image. After each round of upsampling, a 1*1 convolution operation is performed to adjust the number of channels. Through the deep supervision mechanism Deep Supervision, features are extracted from different stages for supervision to ensure that each stage contributes to the final output. After these steps, a segmentation result with the same size as the input image is generated.

2. The gravel terrain passable area segmentation method according to claim 1, characterized in that: In the second step, the specific implementation process of picture encoding is as follows: 1) The input RGB image and surface normal vector image are divided into multiple small blocks, and then each block is embedded into a vector representation of fixed dimension; 2) The vector obtained in the previous step is input into the VSS encoder twice to extract the texture and edge features in the original RGB image and the three-dimensional spatial features in the surface normal vector image; 3) After preliminary feature extraction, the RGB image and surface normal features interact through a Cross-Attention module, and the two modalities are weighted fused to dynamically assign weights to the RGB and normal features; the resulting dimension is H / 4*W / 4*C1.

3. The method for segmenting traversable areas on gravel terrain according to claim 1, characterized in that: In the third step, the specific steps of further encoding the image are: 1) The feature images of H / 4*W / 4*C1 obtained in the previous stage are merged to reduce the image resolution and increase the channel dimension. The internal features of the fused image are further extracted after passing through two VSS blocks. The feature map of the above input dimension H / 4*W / 4*C1 is output as a feature map of H / 8*W / 8*C2; 2) Downsample the feature map of H / 8*W / 8*C2 again to halve the spatial size of the image features; then, after two VSS blocks are calculated and processed, multi-scale features are further extracted, and the feature map with the input dimension of H / 8*W / 8*C2 is output as a feature map of H / 16*W / 16*C3; 3) The feature map outputted above is downsampled for the last time to further reduce the spatial size, and then processed by two VSS blocks, so that the feature map with the input dimension of H / 16*W / 16*C3 is output as a feature map of H / 32*W / 32*C4.

4. The method for segmenting traversable areas on gravel terrain according to claim 1, characterized in that: The specific steps of the fourth step are: 1) The obtained multi-scale feature map is upsampled to the low-resolution feature map through the Patch Expanding module, and the feature dimension map of H / 32*W / 32*C4 is restored to H / 16*W / 16*C3, passing through two layers of VSS blocks and controlling the number of feature map channels through 1*1Conv; 2) The feature dimension map of H / 16*W / 16*C3 obtained above is restored to H / 8*W / 8*C2 dimension through Patch Expanding module, two VSS blocks, 1*1Conv and Deep Supervision respectively; 3) The feature map of H / 8*W / 8*C2 dimension obtained in the previous layer is passed through two VSS blocks, 1*1Conv and DeepSupervision to obtain a feature map of H / 4*W / 4*C1 dimension; 4) Decode the obtained H / 4*W / 4*C1 dimensional feature map, upsample the feature map using Patch Expanding, and then gradually restore it to the original resolution of the input image through 1*1Conv.

Images