Road segmentation method based on multi-modal feature adaptive progressive fusion

Abstract

The invention discloses a road segmentation method based on multi-modal feature adaptive progressive fusion, and the method comprises the following steps: constructing a road image segmentation data set which comprises a plurality of road RGB images obtained through photographing and a plurality of road depth images obtained through laser radar; constructing a road segmentation model, wherein the road segmentation model is composed of an RGB information coding module, a depth information coding module, a feature conversion module, a feature progressive adaptive fusion module and a multi-scale feature decoding module; inputting the road RGB image and the road depth image into a road segmentation model for fusion segmentation to obtain a final road segmentation result; according to the invention, through the feature conversion module and the feature progressive adaptive fusion module which are uniquely designed, the depth information can dynamically supplement RGB information, so that the precision and robustness of the road segmentation model in a complex scene are improved, and the road segmentation model especially has more excellent performance in low-illumination, shielding and dynamic environments.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a road segmentation method based on multi-modal feature adaptive progressive fusion. Background Art

[0002] In the road segmentation task, traditional image segmentation methods based on RGB images have achieved remarkable results. However, relying solely on the features of RGB images usually has certain limitations, especially in the case of lighting changes, weather changes or complex scenes, the performance of RGB images may not be stable and reliable enough. Therefore, how to effectively integrate other types of perception data to improve segmentation accuracy and robustness has become a key research direction in recent years.

[0003] The depth map obtained by LiDAR can provide more accurate depth information than traditional RGB images due to its high-precision spatial measurement capability, greatly improving the accuracy of scene understanding. In the road segmentation task, depth information can effectively help the model understand the geometric shape of objects and distinguish the spatial relationship between roads and other objects, thereby reducing the occurrence of segmentation errors in complex backgrounds. However, the existing technology still has some defects and deficiencies in the fusion process of RGB images and LiDAR depth maps. First, the depth map usually has a low resolution and large noise, which may introduce unnecessary errors when it is directly fused with a high-resolution RGB image. Although the existing fusion methods use mechanisms such as channel attention and spatial attention to enhance the synergy between different modalities, due to the huge differences in the information expression between RGB images and depth maps, it is still difficult to achieve the desired effect by relying solely on traditional fusion methods. Secondly, most of the current fusion methods ignore the contextual relationship between different modal data, such as the joint use of texture features with semantic information in RGB images and geometric features with structural information in depth maps. Existing methods often fail to perform effective multi-level and multi-scale fusion on the basis of fully considering the complementarity of these different types of information.

[0004] Therefore, the existing RGB and depth map fusion technology still has a lot of room for improvement in terms of accuracy, robustness and efficiency. It is urgent to develop a new fusion mechanism that can better fuse the data of these two modalities, make full use of the rich semantic information of RGB images and the spatial geometric characteristics of lidar depth maps, and further improve the performance of road segmentation. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a road segmentation method based on adaptive progressive fusion of multimodal features, which aims to solve the problems mentioned in the background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a road segmentation method based on multi-modal feature adaptive progressive fusion, comprising the following steps: Step S1: constructing a road image segmentation dataset, wherein the road image segmentation dataset includes a number of road RGB images acquired by photographing and road depth images acquired by laser radar; Step S2: constructing a road segmentation model, the road segmentation model consists of an RGB information encoding module, a depth information encoding module, a feature conversion module, a feature progressive adaptive fusion module and a multi-scale feature decoding module; the RGB information encoding module is composed of a P2T network combined with an Agent attention mechanism, referred to as an Agent P2T network; the Agent P2T network consists of four layers with the same structure; the depth information encoding module adopts a ResNet network, the ResNet network consists of five layers connected in sequence, and the five layers of the ResNet network have the same structure; the multi-scale feature decoding module consists of four layers with the same structure; Step S3: first extracting the surface normal of the road depth image to obtain the surface normal features, and then inputting the surface normal features into the depth information encoding module for extraction layer by layer to obtain the road depth feature information of each layer; Step S4: inputting the road RGB image into the first layer of the RGB information encoding module for feature extraction to obtain the first layer of road RGB feature information; Step S5: inputting the road depth feature information extracted by the second layer of the depth information encoding module and the road RGB feature information extracted by the first layer of the RGB information encoding module into the feature conversion module to obtain first improved depth feature information, inputting the first improved depth feature information and the road RGB feature information extracted by the first layer of the RGB information encoding module into the feature progressive adaptive fusion module to obtain first fused feature information, and using the first fused feature information as the input of the second layer of the RGB information encoding module, and so on, to obtain second fused feature information, third fused feature information and fourth fused feature information; Step S6: input the fourth fusion feature information and the third feature fusion information into the first layer of the multi-scale feature decoding module to obtain the first-level up-sampling feature, input the first-level up-sampling feature and the second feature fusion information into the second layer of the multi-scale feature decoding module to obtain the second-level up-sampling feature, input the second-level up-sampling feature and the first feature fusion information into the third layer of the multi-scale feature decoding module to obtain the third-level up-sampling feature, input the third-level up-sampling feature and the road depth feature information D1 extracted by the first layer of the depth information encoding module into the fourth layer of the multi-scale feature decoding module to obtain the fourth-level up-sampling feature, and obtain the final output of the road segmentation model, i.e., the final road segmentation result, after the fourth-level up-sampling feature is activated.

[0007] Further, the specific process of step S5 is: inputting the road depth feature information extracted by the second layer of the depth information encoding module and the road RGB feature information extracted by the first layer of the RGB information encoding module into the feature conversion module to obtain the first improved depth feature information, inputting the first improved depth feature information and the road RGB feature information of the first layer into the feature progressive adaptive fusion module to obtain the first fused feature information, inputting the first fused feature information into the second layer of the RGB information encoding module for feature extraction to obtain the road RGB feature information of the second layer, inputting the road RGB feature information of the second layer and the road depth feature information extracted by the third layer of the depth information encoding module into the feature conversion module to obtain the second improved depth feature information, inputting the second improved depth feature information and the road RGB feature information of the second layer into the feature progressive adaptive fusion module to obtain the second fused feature information, and inputting the second fused feature information into the feature progressive adaptive fusion module to obtain the second fused feature information. The signal is input into the third layer of the RGB information encoding module for feature extraction to obtain the road RGB feature information of the third layer, the road RGB feature information of the third layer and the road depth feature information extracted by the fourth layer of the depth information encoding module are input into the feature conversion module to obtain the third improved depth feature information, the third improved depth feature information and the road RGB feature information of the third layer are input into the feature progressive adaptive fusion module to obtain the third fused feature information, the third fused feature signal is input into the fourth layer of the RGB information encoding module for feature extraction to obtain the road RGB feature information of the fourth layer, the road RGB feature information of the fourth layer and the road depth feature information extracted by the fifth layer of the depth information encoding module are input into the feature conversion module to obtain the fourth improved depth feature information, the fourth improved depth feature information and the road RGB feature information of the fourth layer are input into the feature progressive adaptive fusion module to obtain the fourth fused feature information.

[0008] Furthermore, in step S3, the road depth image is first subjected to surface normal extraction to obtain surface normal features, which are expressed as: ; ; ; In the formula, represents the obtained normal direction; Indicates normal normalization; represents the modulus of the normal line; Represents the final surface normal feature; represents a road depth image; , Respectively represent the pixel coordinates in the horizontal and vertical directions in the road depth image; Represents the horizontal gradient of the road depth image; Represents the vertical gradient of the road depth image; Represents the vertical component of the normal.

[0009] Furthermore, the surface normal features are input into the depth information encoding module for extraction layer by layer, and the specific process of obtaining the road depth feature information of each layer is as follows: The surface normal features obtained after surface normal extraction are input into the first layer of the depth information encoding module, i.e., the first layer of the ResNet network, to obtain the road depth feature information extracted by the first layer. , expressed as: ; In the formula, It is a 7×7 convolutional layer; is the convolution kernel weight; is the batch normalization layer; for Activation function; It is the maximum pooling layer of 3×3 convolution; Input D1 into the second layer of the ResNet network to obtain the road depth feature information extracted by the second layer ,Will Input to the third layer of the ResNet network to obtain the road depth feature information extracted by the third layer ,Will Input to the 4th layer of the ResNet network to obtain the road depth feature information extracted by the 4th layer ,Will Input to the 5th layer of the ResNet network to obtain the road depth feature information extracted by the 5th layer .

[0010] Furthermore, each layer of the Agent P2T network includes an image segmentation vectorization layer, a Transformer encoding layer, a fusion layer, and a linear transformation layer. The Transformer encoding layer is composed of an Agent attention mechanism and a feedforward network FFN. The specific process of step S4 is: convert the road RGB image The input is sent to the first layer of the RGB information encoding module, that is, the first layer of the Agent P2T network for feature extraction. First, the road RGB image is embedded through the image segmentation and vectorization layer. The output after embedding is Enter the Transformer encoding layer and extract it through the Agent attention mechanism After the multi-scale features Input the fusion layer for fusion to obtain multi-scale fusion features , the multi-scale fusion features are enhanced through the feed-forward network FFN, and the enhanced output is fused with the multi-scale fusion feature input fusion layer to obtain the multi-scale enhanced features Finally, the multi-scale enhanced features are transformed through the linear transformation layer Mapped to the target dimension, the road RGB feature information extracted by the first layer of the RGB information encoding module is obtained : ; ; ; ; In the formula, is the input road RGB image; It is a two-dimensional convolutional layer; Represents the output of the image segmentation vectorization layer; Represents the Agent attention mechanism; It is the layer normalization operation; Represents the feed-forward network FFN; is the linear transformation layer; is the bias term.

[0011] Furthermore, the road depth feature information extracted by the second layer of the depth information encoding module The road RGB feature information extracted by the first layer of the RGB information encoding module The specific process of inputting the corresponding information into the feature conversion module to obtain the first improved depth feature information is as follows: the feature conversion module first converts and Perform channel stacking and output the channel stacking After a convolution operation, the first conversion feature map is obtained by the ReLU activation function. and the second conversion feature map , expressed as: ; ; In the formula, express Convolutional layers; express and The output after channel stacking; Will After a convolution operation, The features of each pixel are linearly transformed on the channel, and the result of the linear transformation is then passed through a ReLu activation function to obtain the preliminary transformed depth features. and Scaling and biasing the initial transformed deep features to obtain the first improved deep feature information , expressed as: .

[0012] Furthermore, the first improved depth feature information and the RGB feature information of the road at layer 1 The specific process of inputting the first fusion feature information into the feature progressive adaptive fusion module is as follows: the feature progressive adaptive fusion module first adds the features to the first fusion feature information. and Perform a splicing operation and input the output after the splicing operation into the gated attention mechanism AG module. The gated attention mechanism AG module consists of channel attention and spatial attention. The output after the fusion operation is generated into an attention map through channel attention and spatial attention. , expressed as: ; ; ; ; In the formula, For splicing operation; for and Output after splicing operation; and They are Output after channel attention and spatial attention; It is a 3×3 convolution; for Activation function; It is an element-by-element multiplication operation; The generated attention map is and Weighted, the weighted and Add and get the first fusion feature information , expressed as: ; ; ; In the formula, express The output weighted by the attention map; express Output weighted by the attention map.

[0013] Furthermore, the fourth fusion feature information and the third feature fusion information are input into the first layer of the multi-scale feature decoding module to obtain the first-level up-sampling feature The specific process is as follows: First, the multi-scale feature decoding module converts the fourth fusion feature information Use bilinear interpolation to perform double upsampling, and then Fusion feature information with the third Splicing is performed through jump connections to obtain a preliminary splicing feature map , expressed as: ; ; In the formula, Express Perform a two-fold upsampling operation; Then will After a depth-wise separable convolution, the output of the depth-wise separable convolution is A convolution is used to compress the number of channels, and then the output after the compression of the number of channels is The ReLU activation function is used to increase nonlinearity, and batch normalization is used to accelerate training; the output after batch normalization is After the first dilated convolutional layer, the output of the first dilated convolutional layer After the second dilated convolutional layer, the segmentation feature map is obtained , expressed as: ; ; ; In the formula, represents depthwise separable convolution; and They represent the first dilated convolutional layer and the second dilated convolutional layer, respectively. The convolution kernel sizes of the first dilated convolutional layer and the second dilated convolutional layer are and ; Will Input SE attention module to get weight feature map , the weight feature map and segmentation feature map Multiply them together to get the segmentation weight feature map , expressed as: ; ; In the formula, represents the global average pooling operation; and Represent the first fully connected layer and the second fully connected layer respectively; Will , and Splice by channel to form a fused segmentation weight feature map , and then By reducing the number of channels through a convolution, the Perform regularization operation to obtain the first-level upsampling feature , expressed as: ; ; ; In the formula, represents the regularization operation.

[0014] Compared with the existing technology, the present invention has the following beneficial effects:

[0015] (1) The present invention combines the road RGB image and the road depth image in a multimodal manner, and can simultaneously utilize color features and spatial depth information, thereby improving the accuracy and robustness of road segmentation in complex scenes, especially in low-light, occluded and dynamic environments.

[0016] (2) By using a P2T network combined with an agent attention mechanism, the present invention can dynamically adjust the processing strategy of the road segmentation model in different areas. Compared with the traditional CNN network, it has stronger adaptability and flexibility, can more accurately handle complex backgrounds, dynamic changes and occlusion situations, and improves the accuracy and robustness of road segmentation.

[0017] (3) The present invention uses a uniquely designed feature conversion module and a feature progressive adaptive fusion module to enable depth information to dynamically supplement RGB information, thereby improving the performance of the road segmentation model; the feature conversion module effectively transforms the depth information to better adapt to the fusion with RGB information; and the feature progressive adaptive fusion module dynamically adjusts the information weights of the two modalities according to task requirements, ensuring that the complementary advantages of depth information and RGB information can be fully utilized in different scenarios.

[0018] (4) The present invention improves the efficiency of multimodal data fusion by adopting a multi-scale feature decoding module consisting of depthwise separable convolution, dilated convolution and skip connection, especially when processing RGB information and depth information, which can be better integrated. The SE attention mechanism further strengthens the adaptive selection of key channels, effectively enhancing the accuracy and generalization ability in road segmentation tasks, especially in complex and dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the road segmentation model structure of the present invention. DETAILED DESCRIPTION

[0020] The present invention provides a technical solution: a road segmentation method based on multi-modal feature adaptive progressive fusion, comprising the following steps:

[0021] Step S1: construct a road image segmentation dataset, which includes a number of road RGB images acquired by photographing and road depth images acquired by laser radar.

[0022] The road image segmentation dataset uses the KITTI dataset, which contains road RGB images collected by taking photos of scenes such as urban areas, rural areas, and highways, and road depth images obtained by lidar.

[0023] Among them, road RGB images and road depth images contain a variety of weather conditions and traffic conditions for better application in real life.

[0024] There are three main types of roads in the KITTI dataset: UU (urban unmarked), UM (urban marked), and UMM (urban multiply marked).

[0025] Step S2: Construct a road segmentation model, such as Figure 1 As shown, the road segmentation model consists of an RGB information encoding module, a depth information encoding module, a feature conversion module, a feature progressive adaptive fusion module and a multi-scale feature decoding module; wherein the RGB information encoding module and the depth information encoding module are in a parallel structure, the feature conversion module and the feature progressive adaptive fusion module are interspersed between the layers of the RGB information encoding module and the depth information encoding module, and the multi-scale feature decoding module is connected in series after the RGB information encoding module; Figure 1 middle Represents the feature progressive adaptive fusion module.

[0026] The RGB information encoding module is composed of a Pyramid Pooling Transformer (P2T) network combined with an Agent attention mechanism, referred to as an Agent P2T network. The Agent P2T network consists of four layers with the same structure.

[0027] The depth information encoding module adopts a ResNet network, which is composed of five layers connected in sequence, and the five layers of the ResNet network have the same structure.

[0028] The multi-scale feature decoding module consists of four layers with the same structure.

[0029] Step S3: first extract the surface normal of the road depth image to obtain the surface normal features, and then input the surface normal features into the depth information encoding module for extraction layer by layer to obtain the road depth feature information of each layer.

[0030] Among them, each layer of the depth information encoding module includes a convolution layer, a batch normalization layer, a Relu activation function and a maximum pooling layer.

[0031] Step S3.1: Surface normal extraction is first performed on the road depth image. By calculating the normal of each point, the local geometric information of the object surface is captured. Since the points in the road area are almost on a ground plane, the points on the ground plane have similar surface normals. After the surface normal extraction layer extracts features from the depth map, effective geometric information can be provided, which is expressed as: (1); (2); (3); In the formula, represents the obtained normal direction; Indicates normal normalization; represents the modulus of the normal line; Represents the final surface normal feature; represents a road depth image; , Respectively represent the pixel coordinates in the horizontal and vertical directions in the road depth image; Represents the horizontal gradient of the road depth image; Represents the vertical gradient of the road depth image; Represents the vertical component of the normal.

[0032] Step S3.2: The surface normal features obtained after surface normal extraction are input into the first layer of the depth information encoding module, i.e., the first layer of the ResNet network. First, the surface normal features pass through a convolution layer with a convolution kernel size of 7×7 and a step size of 2, and then pass through a batch normalization layer to accelerate training and improve generalization ability. Then, the Relu activation function is used to help the ResNet network introduce nonlinear characteristics to avoid possible gradient vanishing problems and improve computational efficiency. Finally, a convolution pooling operation is performed through a maximum pooling layer to further reduce the spatial dimension of the features, thereby effectively reducing the number of parameters and the amount of computation in the network. This not only helps to improve computational efficiency, but also enhances the robustness of the ResNet network to rotation and displacement changes, and finally obtains the road depth feature information extracted by the first layer for subsequent processing. , expressed as: (4); In the formula, It is a 7×7 convolutional layer; is the convolution kernel weight; is the batch normalization layer; for Activation function; It is a maximum pooling layer of 3×3 convolution.

[0033] Step S3.2: Input D1 into the second layer of the ResNet network to obtain the road depth feature information extracted by the second layer ,Will Input to the third layer of the ResNet network to obtain the road depth feature information extracted by the third layer ,Will Input to the 4th layer of the ResNet network to obtain the road depth feature information extracted by the 4th layer ,Will Input to the 5th layer of the ResNet network to obtain the road depth feature information extracted by the 5th layer .

[0034] Step S4: Input the road RGB image into the first layer of the RGB information encoding module for feature extraction to obtain the first layer of road RGB feature information.

[0035] Among them, each layer of the Agent P2T network includes an image segmentation vectorization layer (Patch Embedding), a Transformer encoding layer (including the Agent attention mechanism and the feedforward network FFN), a fusion layer, and a linear transformation layer.

[0036] The specific process of step S4 is: convert the road RGB image The input is sent to the first layer of the RGB information encoding module, that is, the first layer of the Agent P2T network for feature extraction. First, the road RGB image is embedded through the image segmentation vectorization layer (PatchEmbedding). The output after embedding is Enter the Transformer encoding layer and extract it through the Agent attention mechanism After the multi-scale features Input the fusion layer for fusion to obtain multi-scale fusion features , the multi-scale fusion features are enhanced through the feed-forward network FFN, and the enhanced output is fused with the multi-scale fusion feature input fusion layer to obtain the multi-scale enhanced features Finally, the multi-scale enhanced features are transformed through the linear transformation layer Mapped to the target dimension, the road RGB feature information extracted by the first layer of the RGB information encoding module is obtained : (5) (6); (7); (8); In the formula, is the input road RGB image; It is a two-dimensional convolutional layer; Represents the output of the image segmentation and vectorization layer (Patch Embedding); Represents the Agent attention mechanism; It is the layer normalization operation; Represents the feed-forward network FFN, which is used for feature projection and feature enhancement; is the linear transformation layer; is the bias term.

[0037] Step S5: input the road depth feature information extracted by the second layer of the depth information encoding module and the road RGB feature information extracted by the first layer of the RGB information encoding module into the feature conversion module accordingly to obtain the first improved depth feature information, input the first improved depth feature information and the road RGB feature information extracted by the first layer of the RGB information encoding module into the feature progressive adaptive fusion module to obtain the first fused feature information, use the first fused feature information as the input of the second layer of the RGB information encoding module, and so on to obtain the second fused feature information, the third fused feature information and the fourth fused feature information.

[0038] The specific process of step S5 is as follows: the road depth feature information extracted by the second layer of the depth information encoding module and the road RGB feature information extracted by the first layer of the RGB information encoding module are input into the feature conversion module accordingly; the feature conversion module learns the conversion parameters and transforms the road depth feature information through multiple convolution operations, so that it can better supplement and improve the road RGB feature information, and obtain the first improved depth feature information; the first improved depth feature information and the road RGB feature information of the first layer are input into the feature progressive adaptive fusion module to obtain the first fused feature information; the first fused feature information is input into the second layer of the RGB information encoding module for feature extraction to obtain the road RGB feature information of the second layer; the road RGB feature information of the second layer and the road depth feature information extracted by the third layer of the depth information encoding module are input into the feature conversion module to obtain the second improved depth feature information; the second improved depth feature information and the road RGB feature information of the second layer are input into the feature progressive adaptive fusion module to obtain the first fused feature information; The fusion module is input to obtain the second fused feature information, the second fused feature signal is input to the third layer of the RGB information encoding module for feature extraction, and the third layer of road RGB feature information is obtained, the third layer of road RGB feature information and the road depth feature information extracted by the fourth layer of the depth information encoding module are input to the feature conversion module to obtain the third improved depth feature information, the third improved depth feature information and the third layer of road RGB feature information are input to the feature progressive adaptive fusion module to obtain the third fused feature information, the third fused feature signal is input to the fourth layer of the RGB information encoding module for feature extraction, and the fourth layer of road RGB feature information is obtained, the fourth layer of road RGB feature information and the road depth feature information extracted by the fifth layer of the depth information encoding module are input to the feature conversion module to obtain the fourth improved depth feature information, the fourth improved depth feature information and the fourth layer of road RGB feature information are input to the feature progressive adaptive fusion module to obtain the fourth fused feature information.

[0039] Among them, the road depth feature information extracted by the second layer of the depth information encoding module The road RGB feature information extracted by the first layer of the RGB information encoding module The specific process of inputting the corresponding information into the feature conversion module to obtain the first improved depth feature information is as follows: the feature conversion module first converts and Perform channel stacking and output the channel stacking After a convolution operation with a convolution kernel size of 1×1, it maps the features of each pixel on the channel (without changing the spatial size), and then passes through the ReLU activation function to obtain the first conversion feature map and the second conversion feature map , expressed as: (9); (10); In the formula, express Convolutional layers; express and Output after channel stacking.

[0040] Will After a convolution operation, The features of each pixel are linearly transformed on the channel, and the result of the linear transformation is then passed through a ReLu activation function to obtain the preliminary transformed depth features. and Scaling and biasing the initial transformed deep features to obtain the first improved deep feature information , expressed as: (11).

[0041] Similarly, the specific process of obtaining the second improved depth feature information, the third improved depth feature information and the fourth improved depth feature information is the same as the specific process of obtaining the first improved depth feature information, which will not be described in detail here.

[0042] Among them, the first improved depth feature information and the RGB feature information of the road at layer 1 The specific process of inputting the first fusion feature information into the feature progressive adaptive fusion module is as follows: the feature progressive adaptive fusion module first adds the features to the first fusion feature information. and Perform a splicing operation and input the output after the splicing operation into the gated attention mechanism AG module. The gated attention mechanism AG module consists of channel attention and spatial attention. The output after the fusion operation is generated into an attention map through channel attention and spatial attention. , expressed as: (12); (13); (14); (15); In the formula, For splicing operation; for and Output after splicing operation; and They are Output after channel attention and spatial attention; It is a 3×3 convolution with padding of 1; for Activation function; It is an element-wise multiplication operation.

[0043] The attention map represents the and The importance of determines which parts should be emphasized and which parts should be suppressed in the subsequent steps, making the fusion process an adaptive state instead of specifying the fusion weights of the two modalities.

[0044] The generated attention map is and Weighted, the weighted and Add and get the first fusion feature information , expressed as: (16); (17); (18); In the formula, express The output weighted by the attention map; express Output weighted by the attention map.

[0045] Similarly, the specific process of obtaining the second fusion feature information, the third fusion feature information and the fourth fusion feature information is the same as the specific process of obtaining the first fusion feature information, which will not be described in detail here.

[0046] Step S6: Input the fourth fusion feature information and the third feature fusion information into the first layer of the multi-scale feature decoding module to obtain the first-level up-sampling feature. The second feature fusion information is input into the second layer of the multi-scale feature decoding module to obtain the second-level upsampling feature, the second-level upsampling feature and the first feature fusion information are input into the third layer of the multi-scale feature decoding module to obtain the third-level upsampling feature, the third-level upsampling feature and the road depth feature information D1 extracted by the first layer of the depth information encoding module are input into the fourth layer of the multi-scale feature decoding module to obtain the fourth-level upsampling feature, and the fourth-level upsampling feature is passed through the activation function to obtain the final output of the road segmentation model, that is, the final road segmentation result.

[0047] The fourth fusion feature information and the third feature fusion information are input into the first layer of the multi-scale feature decoding module to obtain the first-level upsampling feature The specific process is: First, the multi-scale feature decoding module converts the fourth fusion feature information Use bilinear interpolation to perform double upsampling, and then Fusion feature information with the third Splicing is performed through skip connection to obtain a preliminary splicing feature map , expressed as: (19); (20); In the formula, Express Perform a two-fold upsampling operation.

[0048] Then will After a depth-wise separable convolution, Each input channel of the convolution operation is independently convolved, which reduces the amount of calculation and converts the output of the depth-separable convolution into The number of channels is compressed through a 3×3 convolution, and the output after the compression of the number of channels is The ReLU activation function is used to increase nonlinearity, and batch normalization is used to accelerate training; the output after batch normalization is After the first dilated convolutional layer, the output of the first dilated convolutional layer After the second dilated convolutional layer, the segmentation feature map is obtained ; Among them, the first dilated convolution layer and the second dilated convolution layer are connected to a batch normalization operation and a ReLU function. The convolution kernel size and dilation factor of the first dilated convolution layer and the second dilated convolution layer are (3, 2) and (2, 2) respectively. The purpose of using dilated convolution is to improve the segmentation accuracy by increasing the receptive field of the convolution kernel without increasing the amount of calculation; it is expressed as: (twenty one); (twenty two); (twenty three); In the formula, represents depthwise separable convolution; and They represent the first dilated convolutional layer and the second dilated convolutional layer, respectively. The convolution kernel sizes of the first dilated convolutional layer and the second dilated convolutional layer are and .

[0049] Will Input the SE attention module, adjust the weight of each channel through the SE attention module, highlight the important channels and suppress the unimportant channels, and obtain the weight feature map , the weight feature map and segmentation feature map Multiply them together to get the segmentation weight feature map , expressed as: (twenty four); (25); In the formula, represents the global average pooling operation; and Represent the first fully connected layer and the second fully connected layer respectively.

[0050] Will , and Splice by channel to form a fused segmentation weight feature map , and then Through a 1 1 convolution reduces the number of channels, which will reduce the number of channels Perform regularization operation to obtain the first-level upsampling feature , expressed as: (26); (27); (28); In the formula, represents the regularization operation.

[0051] Similarly, the specific process of obtaining the second-level upsampling features, the third-level upsampling features, and the fourth-level upsampling features is the same as the specific process of the first-level upsampling features, and will not be repeated here.

[0052] Most road segmentation tasks are optimized using the cross entropy loss function. However, this method may encounter difficulties when dealing with unbalanced samples. In particular, in road segmentation tasks, the number of pixels in the road area is usually much less than that in the non-road area, resulting in the non-road area having an excessive impact on the model during model training. This imbalance may cause the model to tend to identify pixels as non-road areas when classifying them, thereby reducing the accuracy of segmentation. In order to solve this problem, the present invention combines the sampling boundary loss and the cross entropy loss as the total loss of the road segmentation model. , expressed as: (29); In the formula, and They are the cross entropy loss function and the boundary loss function respectively.

[0053] This loss function design enables the model to not only perform well in global pixel classification, but also optimize local details, which is particularly suitable for tasks that require fine boundary division.

[0054] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A road segmentation method based on adaptive progressive fusion of multimodal features, characterized in that: The steps include: Step S1: constructing a road image segmentation dataset, wherein the road image segmentation dataset includes a number of road RGB images acquired by photographing and road depth images acquired by laser radar; Step S2: constructing a road segmentation model, the road segmentation model consists of an RGB information encoding module, a depth information encoding module, a feature conversion module, a feature progressive adaptive fusion module and a multi-scale feature decoding module; the RGB information encoding module is composed of a P2T network combined with an Agent attention mechanism, referred to as an Agent P2T network; the Agent P2T network consists of four layers with the same structure; the depth information encoding module adopts a ResNet network, the ResNet network consists of five layers connected in sequence, and the five layers of the ResNet network have the same structure; the multi-scale feature decoding module consists of four layers with the same structure; Step S3: first extracting the surface normal of the road depth image to obtain the surface normal features, and then inputting the surface normal features into the depth information encoding module for extraction layer by layer to obtain the road depth feature information of each layer; Step S4: inputting the road RGB image into the first layer of the RGB information encoding module for feature extraction to obtain the first layer of road RGB feature information; Step S5: inputting the road depth feature information extracted by the second layer of the depth information encoding module and the road RGB feature information extracted by the first layer of the RGB information encoding module into the feature conversion module to obtain first improved depth feature information, inputting the first improved depth feature information and the road RGB feature information extracted by the first layer of the RGB information encoding module into the feature progressive adaptive fusion module to obtain first fused feature information, and using the first fused feature information as the input of the second layer of the RGB information encoding module, and so on, to obtain second fused feature information, third fused feature information and fourth fused feature information; Step S6: input the fourth fusion feature information and the third feature fusion information into the first layer of the multi-scale feature decoding module to obtain the first-level up-sampling feature, input the first-level up-sampling feature and the second feature fusion information into the second layer of the multi-scale feature decoding module to obtain the second-level up-sampling feature, input the second-level up-sampling feature and the first feature fusion information into the third layer of the multi-scale feature decoding module to obtain the third-level up-sampling feature, input the third-level up-sampling feature and the road depth feature information D1 extracted by the first layer of the depth information encoding module into the fourth layer of the multi-scale feature decoding module to obtain the fourth-level up-sampling feature, and obtain the final output of the road segmentation model, i.e., the final road segmentation result, after the fourth-level up-sampling feature is activated.

2. The road segmentation method based on multi-modal feature adaptive progressive fusion according to claim 1, characterized in that: The specific process of step S5 is: inputting the road depth feature information extracted by the second layer of the depth information encoding module and the road RGB feature information extracted by the first layer of the RGB information encoding module into the feature conversion module to obtain the first improved depth feature information; inputting the first improved depth feature information and the road RGB feature information of the first layer into the feature progressive adaptive fusion module to obtain the first fused feature information; inputting the first fused feature information into the second layer of the RGB information encoding module for feature extraction to obtain the road RGB feature information of the second layer; inputting the road RGB feature information of the second layer and the road depth feature information extracted by the third layer of the depth information encoding module into the feature conversion module to obtain the second improved depth feature information; inputting the second improved depth feature information and the road RGB feature information of the second layer into the feature progressive adaptive fusion module to obtain the second fused feature information; and inputting the second fused feature signal into the feature progressive adaptive fusion module to obtain the second fused feature information. The road RGB feature information of the third layer is input into the third layer of the RGB information encoding module for feature extraction to obtain the road RGB feature information of the third layer; the road RGB feature information of the third layer and the road depth feature information extracted by the fourth layer of the depth information encoding module are input into the feature conversion module to obtain the third improved depth feature information; the third improved depth feature information and the road RGB feature information of the third layer are input into the feature progressive adaptive fusion module to obtain the third fused feature information; the third fused feature signal is input into the fourth layer of the RGB information encoding module for feature extraction to obtain the road RGB feature information of the fourth layer; the road RGB feature information of the fourth layer and the road depth feature information extracted by the fifth layer of the depth information encoding module are input into the feature conversion module to obtain the fourth improved depth feature information; the fourth improved depth feature information and the road RGB feature information of the fourth layer are input into the feature progressive adaptive fusion module to obtain the fourth fused feature information.

3. The road segmentation method based on multi-modal feature adaptive progressive fusion according to claim 2 is characterized by: In step S3, the road depth image is first subjected to surface normal extraction to obtain surface normal features, which are expressed as: ； ； ； In the formula, represents the obtained normal direction; Indicates normal normalization; represents the modulus of the normal line; Represents the final surface normal feature; represents a road depth image; , Respectively represent the pixel coordinates in the horizontal and vertical directions in the road depth image; Represents the horizontal gradient of the road depth image; Represents the vertical gradient of the road depth image; Represents the vertical component of the normal.

4. The road segmentation method based on multi-modal feature adaptive progressive fusion according to claim 3 is characterized by: The surface normal features are input into the depth information encoding module for extraction layer by layer. The specific process of obtaining the road depth feature information of each layer is as follows: The surface normal features obtained after surface normal extraction are input into the first layer of the depth information encoding module, i.e., the first layer of the ResNet network, to obtain the road depth feature information extracted by the first layer. , expressed as: ； In the formula, It is a 7×7 convolutional layer; is the convolution kernel weight; is the batch normalization layer; for Activation function; It is the maximum pooling layer of 3×3 convolution; Input D1 into the second layer of the ResNet network to obtain the road depth feature information extracted by the second layer ,Will Input to the third layer of the ResNet network to obtain the road depth feature information extracted by the third layer ,Will Input to the 4th layer of the ResNet network to obtain the road depth feature information extracted by the 4th layer ,Will Input to the 5th layer of the ResNet network to obtain the road depth feature information extracted by the 5th layer .

5. The road segmentation method based on multi-modal feature adaptive progressive fusion according to claim 4, characterized in that: Each layer of the Agent P2T network includes a picture segmentation vectorization layer, a Transformer encoding layer, a fusion layer, and a linear transformation layer. The Transformer encoding layer is composed of an Agent attention mechanism and a feedforward network FFN. The specific process of step S4 is: convert the road RGB image The input is sent to the first layer of the RGB information encoding module, that is, the first layer of the Agent P2T network for feature extraction. First, the road RGB image is embedded through the image segmentation and vectorization layer. The output after embedding is Enter the Transformer encoding layer and extract it through the Agent attention mechanism After the multi-scale features Input the fusion layer for fusion to obtain multi-scale fusion features , the multi-scale fusion feature is enhanced through the feed-forward network FFN, and the enhanced output is fused with the multi-scale fusion feature input fusion layer to obtain the multi-scale enhanced feature Finally, the multi-scale enhanced features are transformed through the linear transformation layer Mapped to the target dimension, the road RGB feature information extracted by the first layer of the RGB information encoding module is obtained : ； ； ； ； In the formula, is the input road RGB image; It is a two-dimensional convolutional layer; Represents the output of the image segmentation vectorization layer; Represents the Agent attention mechanism; It is the layer normalization operation; Represents the feed-forward network FFN; is the linear transformation layer; is the bias term.

6. The road segmentation method based on multi-modal feature adaptive progressive fusion according to claim 5, characterized in that: The road depth feature information extracted by the second layer of the depth information encoding module The road RGB feature information extracted by the first layer of the RGB information encoding module The specific process of inputting the corresponding information into the feature conversion module to obtain the first improved depth feature information is as follows: the feature conversion module first converts and Perform channel stacking and output the channel stacking After a convolution operation, the first conversion feature map is obtained by the ReLU activation function. and the second conversion feature map , expressed as: ； ； In the formula, express Convolutional layers; express and The output after channel stacking; Will After a convolution operation, The features of each pixel are linearly transformed on the channel, and the result of the linear transformation is then passed through a ReLu activation function to obtain the preliminary transformed depth features. and Scaling and biasing the initial transformed deep features to obtain the first improved deep feature information , expressed as: 。 7. The road segmentation method based on multi-modal feature adaptive progressive fusion according to claim 6, characterized in that: First improve the depth feature information and the RGB feature information of the road at layer 1 The specific process of inputting the first fusion feature information into the feature progressive adaptive fusion module is as follows: the feature progressive adaptive fusion module first adds the features to the first fusion feature information. and Perform a splicing operation and input the output after the splicing operation into the gated attention mechanism AG module. The gated attention mechanism AG module consists of channel attention and spatial attention. The output after the fusion operation is generated into an attention map through channel attention and spatial attention. , expressed as: ； ； ； ； In the formula, For splicing operation; for and Output after splicing operation; and They are Output after channel attention and spatial attention; It is a 3×3 convolution; for Activation function; It is an element-by-element multiplication operation; The generated attention map is and Weighted, the weighted and Add and get the first fusion feature information , expressed as: ； ； ； In the formula, express The output weighted by the attention map; express Output weighted by the attention map.

8. The road segmentation method based on multi-modal feature adaptive progressive fusion according to claim 7, characterized in that: The fourth fusion feature information and the third feature fusion information are input into the first layer of the multi-scale feature decoding module to obtain the first-level up-sampled feature The specific process is as follows: First, the multi-scale feature decoding module converts the fourth fusion feature information Use bilinear interpolation to perform double upsampling, and then Fusion feature information with the third Splicing is performed through jump connections to obtain a preliminary splicing feature map , expressed as: ； ； In the formula, Express Perform a two-fold upsampling operation; Then will After a depth-wise separable convolution, the output of the depth-wise separable convolution is A convolution is used to compress the number of channels, and then the output after the compression of the number of channels is The ReLU activation function is used to increase nonlinearity, and batch normalization is used to accelerate training; the output after batch normalization is After the first dilated convolutional layer, the output of the first dilated convolutional layer After the second dilated convolutional layer, the segmentation feature map is obtained , expressed as: ； ； ； In the formula, represents depthwise separable convolution; and They represent the first dilated convolutional layer and the second dilated convolutional layer, respectively. The convolution kernel sizes of the first dilated convolutional layer and the second dilated convolutional layer are and ; Will Input SE attention module to get weight feature map , the weight feature map and segmentation feature map Multiply them together to get the segmentation weight feature map , expressed as: ； ； In the formula, represents the global average pooling operation; and Represent the first fully connected layer and the second fully connected layer respectively; Will , and Splice by channel to form a fused segmentation weight feature map , and then By reducing the number of channels through a convolution, the Perform regularization operation to obtain the first-level upsampling feature , expressed as: ； ； ； In the formula, represents the regularization operation.

Images