Cross-country feasible region detection method based on Transform and multi-modal feature fusion

Abstract

The invention provides a cross-country feasible region detection method based on Transform and multi-modal feature fusion, which is used for carrying out traversal region detection in a cross-country environment and aims to solve the problems of insufficient modal feature extraction, unclear segmentation in automatic driving, poor long-distance road scene performance and the like when an RGB image and a depth image are fused. A CM-FRM module is introduced to replace a cross attention mechanism in a reference model so as to meet more complex feature extraction requirements in a cross-country scene. In the feature fusion part, an FFX module is provided, information from a camera and a LiDAR sensor is aggregated, and therefore the road detection task effect in the cross-country environment is improved; and finally, the decoder part is redesigned, and a layered up-sampling method is adopted, so that the strength of the feature information is further improved, and meanwhile, the parameter quantity and the calculation complexity are reduced.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology. Specifically, it realizes the deep integration of artificial intelligence and human-centered autonomous driving under Industry 5.0, and proposes a model training method for cross-domain learning in the system structure, from basic model to detection of drivable areas on off-road roads. Background Technology

[0002] Autonomous driving, as a component of Industry 5.0, plays a crucial role in accelerating human-machine collaboration and promoting a human-centered industrial world. Humans, through long-term social interaction, learning, and work, have learned how to make various decisions, giving them advantages in standard selection, pattern recognition, decision-making, and group perception. However, humans are easily influenced by emotions, further affecting rational decision-making during manual driving. For autonomous vehicles, the driving environment is not limited to urban roads but also sometimes involves unstructured roads such as mines and off-road terrain. In urban environments, traversable space is generally defined as free space, i.e., paved open areas without obstacles. Meanwhile, in unstructured roads, the concept of traversable areas is relatively vague, often filled with obstacles such as weeds and tree stumps, and affected by unknown weather conditions such as rain, snow, fog, or low visibility, posing significant challenges to autonomous driving. Furthermore, when vehicles are operating in unstructured environments such as mines and off-road terrain, coupled with the interference of complex and highly dynamic scenarios, the vehicle's perception system cannot guarantee robustness, and the decision-making system is easily affected. However, current research mainly focuses on structured urban road environments, with relatively little research on other types of road environments, such as rural roads and mountain roads. Considering the high complexity and diversity of off-road environments, directly using existing structured road segmentation models may lead to poor prediction results or even performance degradation compared to structured roads. Although existing research has been dedicated to improving perception capabilities in off-road environments, there is still no optimal solution for the challenges of off-road environments and adverse scenarios. Previous work has mostly utilized CNNs. While CNN-based networks perform well in structured urban navigation with clear roads and structures, their performance degrades in off-road environments with limited receptive fields and in areas with unclear boundaries, overlaps, and complex interweaving. Furthermore, perception is an indispensable module in the drivable area detection task for autonomous vehicles, and multimodal fusion is currently recognized as the most effective method to enhance vehicle perception capabilities. LiDAR point cloud data contains spatial geometric information but lacks semantic information, while monocular RGB images contain higher-level environmental semantic information. By fusing LiDAR and camera data, superior perception performance can be achieved, making it more suitable for dealing with the challenges of off-road environments and adverse scenarios. Cross-attention is widely used for feature extraction and fusion, but its performance is not ideal in off-road environments due to challenges such as harsh weather and complex scenes, especially in predicting edge regions and distant view acquisition. Finally, in free space detection tasks, surface normal information is easier to identify than depth information because surface normals are consistent on the same road plane. To improve segmentation accuracy, current work has explored inferring surface normal information from dense depth images and fusing it with image information to enhance free space detection performance. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide an off-road feasible region detection method based on Transformer and multimodal feature fusion to solve the problems of decreased segmentation accuracy in off-road region detection and insufficient feature extraction in autonomous driving.

[0004] The off-road feasible area detection method adopts an off-road network, which includes an encoding module and a decoding module;

[0005] The encoding module includes a path embedding module, a first encoder module, a CM-FRM module, an FFX module, a second encoder module, a third encoder module, and a fourth encoder module.

[0006] During the feature extraction process in the encoding module, an RGB image with a resolution of H×W×3 and a surface normal image derived from the LiDAR point cloud are first input into the path embedding module. The input image is then segmented into image blocks of a set size. Subsequently, the first encoder processes the feature representation output by the path embedding module and outputs the encoded RGB features. and point cloud features ,in This indicates the number of channels, with H and W representing the height and width of the RGB image, respectively; then it is input into the CM-FR module;

[0007] In the CM-FRM module, the modal features of the parallel stream are refined at each stage of feature extraction, which is divided into two parts: channel dimension feature correction and spatial dimension feature correction. Specifically:

[0008] In the channel dimension feature correction section, features are processed in two dimensions: RGB features of the input. and point cloud features For each modality, global max pooling is performed along the channel dimension, followed by global average pooling to obtain two feature vectors. A total of four feature vectors are generated from both modalities. These four feature vectors are then concatenated to obtain the feature vectors. Next, the features were calculated using MLP and the Sigmoid function. weight And separate them according to their respective modes into and ;

[0009] The features are corrected based on the weights, and finally the two modality-corrected features are calculated. and ;

[0010]

[0011] In spatial dimension feature correction, the input RGB features are... and point cloud features Features of splicing Where S=C; features are calculated using MLP and Sigmoid functions respectively. weight And separate them into weights according to their respective modes. and ;

[0012] Calculate the spatial correction characteristics across modes:

[0013]

[0014] Where * denotes spatial multiplication; combining channel and spatial correction features, the final correction feature is obtained:

[0015]

[0016]

[0017] in, and These are two hyperparameters that need to be trained;

[0018] In the feature fusion stage of the encoding module, the FFX module is used to perform feature fusion. The process is as follows:

[0019] First, the input and The channel embedding module performs channel embedding, transforming the dimensions to a form suitable for subsequent processing. Then, these embedded features are normalized. Next, the normalized features of the two modalities are concatenated along the channel dimensions to form a fusion feature containing RGB and surface normal information, with dimensions of H / 4×W / 4×2C.

[0020] Next, the concatenated feature maps are processed by 1x1 convolution to perform feature fusion, outputting a new feature map with dimensions H / 4×W / 4×C. Subsequently, the new feature map is further processed by depthwise convolution, activation function, and convolutional layer to extract and enhance features, resulting in an enhanced feature map. The enhanced feature map is then subjected to residual connections and normalized again before being fused to obtain a feature map X with feature dimensions H / 4×W / 4×C.

[0021] In the efficient attention module, the input feature map X, with dimensions H / 4×W / 4×C, is first flattened into an n×C matrix as the initial input to the efficient attention module. After processing by the efficient attention module, the obtained features are added to the flattened matrix features corresponding to the initial input feature map X to form the residual structure output X1, with dimensions H / 4×W / 4×C1.

[0022] The second encoder receives the input feature map X1 and performs encoding processing to obtain a feature map X2 of size H / 8×W / 8×C2, which is then output to the MLP layer.

[0023] The third encoder receives the input feature map X2 and performs encoding processing to obtain a feature map X3 of size H / 16×W / 16×C3, which is then output to the MLP layer.

[0024] The fourth encoder receives the input feature map X3 and performs encoding processing to obtain a feature map X4 of size H / 16×W / 16×C3, which is then output to the MLP layer.

[0025] in The parameters are set;

[0026] The MLP layer receives multi-level feature maps X1, X2, X3, and X4, aggregates channel information of the features, and outputs processed feature maps. The output feature maps of each layer are upsampled to restore the size to H / 4 × W / 4 × C. All upsampled feature maps are convolved. Finally, all convolved feature maps enter the Fusion Module for fusion. In this module, the tensors corresponding to these feature maps are first stacked into a new tensor according to the index order of the original tensor list. Then, a summation operation is performed along the first dimension of the new tensor to generate a new feature map. Finally, the final prediction result is output through another MLP layer.

[0027] Finally, the Off-road network is trained: During the network training phase, the input data consists of LiDAR point clouds and image pairs, which include three categories: passable areas, impassable areas, and inaccessible areas.

[0028] Finally, an Off-road network is applied for identification: the LiDAR point cloud and image data to be identified are input into the trained Off-road network, which outputs feasible regions.

[0029] Preferably, when training the off-road network, SGDM is used as the optimizer, with the number of epochs set to 30, the initial learning rate set to 0.00095, and the batch size set to 2.

[0030] Preferably, when training an Off-road network, the image size used for training and testing is set to 1280×704.

[0031] Ideally, when evaluating the performance of off-road networks, five metrics should be used: Accuracy, Precision, Recall, F-score, and IOU.

[0032] Preferably, the normalization method includes the Scaling function and the Softmax function.

[0033] Preferably, in the efficient attention module, the flattened matrix X is separated into Q, K, and V, which are then normalized to obtain a dimension of n× The result; then, this result was reshaped into H / 4×W / 4× ;like Apply a 1x1 convolution to restore the dimension to C; finally, output X1.

[0034] Preferably, the normalization method includes Softmax followed by... The calculation yields a dimension of n× The result; among which, and These are the normalization functions for the query and key features, respectively.

[0035] The present invention has the following beneficial effects:

[0036] (1) This invention proposes an Off-road model framework based on the multimodal perception and decision-making system basic model of off-road roads, which is used to detect traversable areas in off-road environments. This aims to solve problems such as insufficient modal feature extraction when fusing RGB images and depth images, unclear segmentation in autonomous driving, and poor performance in long-distance road scenes.

[0037] (2) This invention uses the CM-FRM module to replace the cross-attention mechanism in the baseline model to meet the more complex feature extraction requirements in off-road scenarios. In the feature fusion part, this invention proposes the FFX module, which aggregates information from the camera and LiDAR sensor to improve the performance of road detection tasks in off-road environments. Finally, this invention redesigns the decoder part and adopts a hierarchical upsampling method to further enhance the intensity of feature information while reducing the number of parameters and computational complexity. Attached Figure Description

[0038] Figure 1 This is the algorithm architecture for the encoder.

[0039] Figure 2 The CM-FRM algorithm architecture is used.

[0040] Figure 3 It is based on the FFX algorithm architecture.

[0041] Figure 4 This is the algorithm structure for the decoder.

[0042] Figure 5 This is a visualization of the segmentation results of the model proposed in this invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] This invention proposes an off-road network for combining camera and LiDAR information (i.e., surface normal information obtained from LiDAR point cloud computing). The off-road network of this invention includes an encoding module and a decoding module; wherein the encoding module further includes a path embedding module, a first encoder module, a CM-FRM module, an FFX module, a second encoder module, a third encoder module, and a fourth encoder module.

[0045] In free space detection tasks, surface normals are consistent across the same road plane. Therefore, surface normal information is easier to identify than depth information, which is beneficial for feature extraction and fusion with RGB images. This invention chooses surface normal information as the network input. Since drivable area detection tasks require a large receptive field, and the receptive field of CNNs is limited, using a raw CNN would lead to a decline in image processing performance. The Transformer framework performs very well in capturing both local and global information; therefore, this invention also chooses the Transformer framework for model design. Based on the specific problems encountered, this invention uses the CM-FRM module for feature extraction and proposes a feature fusion module, FFX. Simultaneously, the decoder employs a hierarchical upsampling method to optimize feature enhancement and reduce the number of parameters and computational complexity. This model performs excellently in experimental results. After transferring the large model to off-road environments, the vehicle system will have greater generalization capabilities, making correct perceptions and decisions even in complex and unknown scenarios. This enables safer and more intelligent decision-making in applications such as route planning and cooperative driving.

[0046] In the encoder stage, this invention employs the Transformer module and the CM-FRM module for feature extraction, and introduces an innovative FFX module to dynamically fuse multimodal features, as follows: Figure 1 As shown. Furthermore, this invention redesigns the encoder using a convolutional layer output method, as detailed below.

[0047] (1) In the feature extraction process of the encoding module, the RGB image with a resolution of H×W×3 and the surface normal image derived from the lidar point cloud are first input into the path embedding module to divide the input image into image blocks of a set size. Then the first encoder, Transformer Encoder 1, processes the feature representation output by the path embedding module and outputs the RGB features after encoding. and point cloud features ,in , representing the number of channels, H and W represent the height and width of the RGB image, respectively; then it is input into the CM-FR module.

[0048] (2) In the CM-FRM module, the modal features of the parallel stream are refined at each stage of feature extraction, which is divided into two parts: channel dimension feature correction and spatial dimension feature correction, such as Figure 2 As shown.

[0049] In the channel-dimensional feature correction section, the CM-FRM module processes features in two dimensions to address noise and uncertainty in different modes. Specifically, the two input modal features are RGB features. and point cloud features For each modality, global max pooling is performed along the channel dimension, followed by global average pooling to obtain two feature vectors. A total of four feature vectors are generated from both modalities. These four feature vectors are then concatenated to obtain the feature vectors. Next, the features were calculated using MLP and the sigmoid function, respectively. weight And separate them according to their respective modes into and .

[0050]

[0051] The features are corrected based on the weights, and finally the two modality-corrected features are calculated. and ;

[0052]

[0053] In spatial dimension feature correction, the input RGB features are... and point cloud features Features of splicing Where S=C, similarly, the features are calculated using MLP and the Sigmoid function in turn. weight And separate them into weights according to their respective modes. and .

[0054]

[0055] Calculate the spatial correction characteristics across modes:

[0056]

[0057] Where * denotes spatial multiplication. Combining channel and spatial correction features, the final correction feature is obtained:

[0058]

[0059]

[0060] in, and These are two hyperparameters. They are both set to 0.5 as the default value. and It is the rectified feature after comprehensive calibration, which is then sent to the feature fusion stage.

[0061] (3) such as Figure 3 As shown, in the feature fusion stage of the encoding module, this invention proposes an FFX module for efficient feature fusion. The data processing procedure for each module of the FFX (Feature Fusion eXtractor) network is as follows:

[0062] 1) First, the input and The data is processed through the Channel Embedding module, which embeds the features and transforms their dimensions to a form suitable for subsequent processing. Then, these embedded features are normalized by the Add&Norm module to ensure data uniformity and stability. Next, the normalized features of the two modalities are concatenated along the channel dimensions by the Concatenate and merge modules to form a fused feature containing RGB and surface normal information, with dimensions of H / 4×W / 4×2C.

[0063] Next, the concatenated feature map is processed by a 1x1 convolution to fuse features, outputting a new feature map with dimensions H / 4×W / 4×C. Subsequently, the new feature map is further processed by depthwise convolution (DWConv 3x3), an activation function (ReLU), and a convolutional layer (Conv 1) to extract and enhance features, resulting in an enhanced feature map. After this module, the enhanced feature map is passed to the second Add&Norm module for residual connections and further normalization. Finally, it is fused using the F merged module to obtain feature X with dimensions H / 4×W / 4×C.

[0064] 2) High-efficiency attention module. For example... Figure 4 As shown, in the efficient attention module, firstly, the input feature map X, with dimensions H / 4×W / 4×C, is flattened into a matrix of dimension n×C as the initial input to the efficient attention module. Next, the matrix was separated into Q, K, and V, and then... and After normalization (normalization methods include scaling and softmax; this invention uses softmax), through... The calculation yields a dimension of n× The result. Then, this result was reshaped to H / 4×W / 4× .like Then, a 1x1 convolution is applied to restore the dimension to C. Finally, the obtained features are added to the flattened matrix features corresponding to the original input feature map X to form the residual structure output X1, with dimensions H / 4×W / 4×C1.

[0065] in, and These are the normalization functions for the query and key features, respectively. The same two normalization methods used in point attention are employed:

[0066]

[0067] in, Indicates along the matrix Apply the softmax function to each row or column.

[0068] (4) In order to obtain multi-level features, we designed three Transformer encoders. The input and output of each Transformer module are as follows:

[0069] Transformer Encoder 2 receives the input feature map X1, encodes it, and outputs the resulting feature map X2 of size H / 8×W / 8×C2 to the MLP layer.

[0070] Transformer Encoder 3 receives the input feature map X2, encodes it, and outputs the resulting feature map X3 of size H / 16×W / 16×C3 to the MLP layer.

[0071] Transformer Encoder 4 receives the input feature map X3, encodes it, and outputs the resulting feature map X4 of size H / 16×W / 16×C3 to the MLP layer.

[0072] in .

[0073] In the decoding module, this invention redesigns the decoder, such as... Figure 4 As shown, this effectively integrates local and global information.

[0074] First, the decoder module includes an MLP layer that receives multi-level feature maps X1, X2, X3, and X4 from the encoder. The MLP layer aggregates channel information from the features and outputs processed feature maps. Each layer's output feature map is upsampled by an Upsample layer to restore it to a size of H / 4 × W / 4 × C. All upsampled feature maps are then fed into the Convolutional Module for convolution. Finally, all convolutionally processed feature maps enter the Fusion Module for fusion. In this module, the tensors corresponding to these feature maps are first stacked into a new tensor according to the index order of the original tensor list. Then, a summation operation is performed along the first dimension of the new tensor to generate a new feature map. Finally, the final prediction result is output through the MLP layer.

[0075] (4) Finally, the Off-road network is trained. During the network training phase, the ORFD dataset is used, which contains 12,198 frames of LiDAR point cloud and image pairs, including three categories: passable areas, impassable areas, and inaccessible areas (such as the sky). SGDM is used as the optimizer, with the number of epochs set to 30, the initial learning rate set to 0.00095, and the batch size set to 2. The image size used for training and testing is set to 1280×704. In evaluating model performance, this embodiment uses five metrics: Accuracy, Precision, Recall, F-score, and IOU. Among these, hyperparameters also need to be learned. and .

[0076] (5) Finally, the trained Off-road network is used for recognition: the input data is a 1280×704 image, and the output is the feasible region.

[0077] The results of this experiment are as follows Figure 5 As shown, the model includes RGB images, surface normals, segmented images, and labeled images. This invention cites research results from recent years and provides a detailed analysis of the model results, as shown in Table 2. Compared to FuseNet and FtFoot models that use depth and RGB image information as input, and SNE-RoadSeg, FSN_Swin, and Off_Net models that use RGB images and surface normals as input, this model achieves better evaluation metrics.

[0078]

[0079] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting off-road feasible areas, characterized in that, An off-road network is adopted, which includes an encoding module and a decoding module. In the encoder module, the Transformer module and the CM-FRM module are used for feature extraction, and the FFX module is introduced to dynamically fuse multimodal features. The decoder module includes an MLP layer that receives multi-level feature maps X1, X2, X3, and X4 from the encoder. The MLP layer aggregates channel information of the features and outputs the processed feature map.

2. The off-road feasible area detection method as described in claim 1, characterized in that, The encoding module includes a path embedding module, a first encoder module, a CM-FRM module, an FFX module, a second encoder module, a third encoder module, and a fourth encoder module; During the feature extraction process in the encoding module, an RGB image with a resolution of H×W×3 and a surface normal image derived from the LiDAR point cloud are first input into the path embedding module. The input image is then segmented into image blocks of a set size. Subsequently, the first encoder processes the feature representation output by the path embedding module and outputs the encoded RGB features. and point cloud features ,in This indicates the number of channels, with H and W representing the height and width of the RGB image, respectively; it is then input into the CM-FR module. In the CM-FRM module, the modal features of the parallel stream are refined at each stage of feature extraction, which is divided into two parts: channel dimension feature correction and spatial dimension feature correction. Specifically: In the channel dimension feature correction section, features are processed in two dimensions: RGB features of the input. and point cloud features For each modality, global max pooling is performed along the channel dimension, followed by global average pooling to obtain two feature vectors. A total of four feature vectors are generated from both modalities. These four feature vectors are then concatenated to obtain the feature vectors. Next, the features were calculated using MLP and the Sigmoid function. weight And separate them according to their respective modes into and ; The features are corrected based on the weights, and finally the two modality-corrected features are calculated. and ; ； In spatial dimension feature correction, the input RGB features are... and point cloud features Features of splicing Where S=C; features are calculated using MLP and Sigmoid functions respectively. weight And separate them into weights according to their respective modes. and ; Calculate the spatial correction characteristics across modes: ； Where * denotes spatial multiplication; combining channel and spatial correction features, the final correction feature is obtained: ； ； in, and These are two hyperparameters that need to be trained; In the feature fusion stage of the encoding module, the FFX module is used to perform feature fusion. The process is as follows: First, the input and The channel embedding module performs channel embedding, transforming the dimensions to a form suitable for subsequent processing. Then, these embedded features are normalized. Next, the normalized features of the two modalities are concatenated along the channel dimensions to form a fusion feature containing RGB and surface normal information, with dimensions of H / 4×W / 4×2C. Next, the concatenated feature maps are processed by 1x1 convolution to perform feature fusion, outputting a new feature map with dimensions H / 4×W / 4×C. Subsequently, the new feature map is further processed by depthwise convolution, activation function, and convolutional layer to extract and enhance features, resulting in an enhanced feature map. The enhanced feature map is then subjected to residual connections and normalized again before being fused to obtain a feature map X with feature dimensions H / 4×W / 4×C. In the efficient attention module, the input feature map X, with dimensions H / 4×W / 4×C, is first flattened into an n×C matrix as the initial input to the efficient attention module. After processing by the efficient attention module, the obtained features are added to the flattened matrix features corresponding to the initial input feature map X to form the residual structure output X1, with dimensions H / 4×W / 4×C1. The second encoder receives the input feature map X1 and performs encoding processing to obtain a feature map X2 of size H / 8×W / 8×C2, which is then output to the MLP layer. The third encoder receives the input feature map X2 and performs encoding processing to obtain a feature map X3 of size H / 16×W / 16×C3, which is then output to the MLP layer. The fourth encoder receives the input feature map X3 and performs encoding processing to obtain a feature map X4 of size H / 16×W / 16×C3, which is then output to the MLP layer. in The parameters are set; The MLP layer receives multi-level feature maps X1, X2, X3, and X4, aggregates channel information of the features, and outputs processed feature maps. The output feature maps of each layer are upsampled to restore the size to H / 4 × W / 4 × C. All upsampled feature maps are convolved. Finally, all convolved feature maps enter the Fusion Module for fusion. In this module, the tensors corresponding to these feature maps are first stacked into a new tensor according to the index order of the original tensor list. Then, a summation operation is performed along the first dimension of the new tensor to generate a new feature map. Finally, the final prediction result is output through another MLP layer. Finally, the Off-road network is trained: During the network training phase, the input data consists of LiDAR point clouds and image pairs, which include three categories: passable areas, impassable areas, and inaccessible areas. Finally, an Off-road network is applied for identification: the LiDAR point cloud and image data to be identified are input into the trained Off-road network, which outputs feasible regions.

3. The off-road feasible area detection method as described in claim 2, characterized in that, When training the Off-road network, SGDM was used as the optimizer, with the number of epochs set to 30, the initial learning rate set to 0.00095, and the batch size set to 2.

4. The off-road feasible area detection method as described in claim 2, characterized in that, When training the Off-road network, the image size used for training and testing is set to 1280×704.

5. The off-road feasible area detection method as described in claim 2, characterized in that, When evaluating the performance of off-road networks, five metrics are used: Accuracy, Precision, Recall, F-score, and IOU.

6. The off-road feasible area detection method as described in claim 2, characterized in that, The normalization methods include the Scaling function and the Softmax function.

7. The off-road feasible area detection method as described in claim 2, characterized in that, In the efficient attention module, the flattened matrix X is separated into Q, K, and V, which are then normalized to obtain n× The result; then, this result was reshaped into H / 4×W / 4× ;like Apply a 1x1 convolution to restore the dimension to C; finally, output X1.

8. The off-road feasible area detection method as described in claim 7, characterized in that, The normalization method includes Softmax, and then... The calculation yields a dimension of n× The result; among which, and These are the normalization functions for the query and key features, respectively.

9. The off-road feasible area detection method as described in claim 2, characterized in that, and The value is 0.5.

Images