A road segmentation method based on local-global dual-stream information collaborative perception

Abstract

The present invention discloses a road segmentation method based on local-global dual-stream information collaborative perception, comprising the following steps: collecting high-resolution road images; constructing a road segmentation model; inputting the image into the road segmentation model, first introducing shadows into the high-resolution road image to enhance it in the road segmentation model, and then reducing the enhanced image; obtaining a global semantic feature map of the reduced image through a global perception branch; obtaining a local detail feature map of the enhanced image through a local collaborative perception branch; aligning the features of the global semantic feature map and then adding the local detail feature map element by element to obtain a preliminary fused feature map, and inputting the preliminary fused feature map into a dynamic dual-path fusion module to obtain a road segmentation map; the present invention realizes a full-link upgrade from data generation, feature extraction, feature fusion to training optimization, significantly improving the model's recognition accuracy and robustness of road features and its ability to detect small targets in complex environments.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a road segmentation method based on local-global dual-stream information collaborative perception. Background Art

[0002] Deep learning-based image segmentation methods have made significant progress in road segmentation tasks. However, existing methods typically rely on a single feature extraction approach, making it difficult to effectively integrate local details with global semantic information. Traditional methods often sacrifice spatial resolution to expand the receptive field during feature extraction, resulting in reduced perception of small-scale structures (such as road markings and cracks). Alternatively, they may ignore long-range contextual dependencies while emphasizing local details, affecting segmentation consistency in complex scenes.

[0003] While existing technologies such as FPN and U-Net can alleviate this problem to some extent, they still suffer from key drawbacks. The local-global feature split occurs: Most methods employ a serially stacked convolutional architecture, which gradually dilutes fine local features as they propagate through the deep network, failing to preserve key microstructural information in the final prediction. Inadequate feature interaction mechanisms exist: Existing two-stream architectures (such as high- and low-resolution parallel networks) typically rely on simple feature concatenation or additive fusion, failing to establish effective cross-level dynamic interaction mechanisms. This results in insufficient coordinated optimization of local details and global semantics. Poor adaptability to complex scenarios: In challenging scenarios such as extreme lighting, occlusion, or long-distance roads, existing methods are prone to disjointed missegmentation (e.g., blurred road edges and missed small objects) due to local feature ambiguity or lack of global context. Of particular note, while some studies have attempted to incorporate attention mechanisms (such as the non-local module) to enhance long-range dependency modeling, these approaches suffer from high computational complexity and struggle to accurately align global semantics while maintaining high-resolution local features. In addition, existing methods lack the ability to explicitly model road topology and are unable to adaptively adjust the contribution weights of local and global features, resulting in limited generalization performance of the algorithm in complex urban scenarios.

[0004] Therefore, there is an urgent need for a new local-global dual-stream information collaborative perception mechanism that can achieve a dynamic balance between high-resolution local feature extraction and efficient global context modeling, and improve the accuracy and robustness of road segmentation through structured cross-level interaction and adaptive feature calibration. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention provides a road segmentation method based on local-global dual-stream information collaborative perception, which aims to solve the problems in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a road segmentation method based on local-global dual-stream information collaborative perception, comprising the following steps:

[0007] Step S1: Construct a data set, which includes several high-resolution road images, and the road areas in the high-resolution road images are all marked with labels;

[0008] Step S2: Constructing a road segmentation model; the road segmentation model consists of a dynamic shadow synthesis data enhancement module, a local collaborative perception branch, a global perception branch, and a dynamic dual-path fusion module;

[0009] Step S3: Inputting the high-resolution road image into the dynamic shadow synthesis data enhancement module, first introducing elliptical or tree-shaped shadows into the high-resolution road image to obtain a high-resolution road enhanced image, and then reducing the high-resolution road enhanced image to obtain a low-resolution road enhanced image;

[0010] Step S4: obtaining a global semantic feature map of the low-resolution road enhanced image through the global perception branch, and obtaining a local detail feature map of the high-resolution road enhanced image through the local collaborative perception branch;

[0011] Step S5: After feature alignment, the global semantic feature map is added element by element to the local detail feature map to obtain a preliminary fusion feature map, which is input into the dynamic two-way fusion module to obtain a road segmentation map.

[0012] Furthermore, the dynamic two-way fusion module is composed of a spatial attention module, a channel attention module, an edge auxiliary branch, a pixel attention module and a segmentation head; the preliminary fusion feature map Input into the dynamic dual-path fusion module to obtain the road segmentation map The specific process is:

[0013] Initially fusion feature map Input into the spatial attention module to obtain the spatial attention map , expressed as:

[0014] ;

[0015] Where, express Convolutional layers; Represents a splicing operation; represents the average pooling layer; represents the maximum pooling layer;

[0016] The spatial attention map And the preliminary fusion feature map Input the edge auxiliary branch to obtain the edge enhanced spatial attention map , expressed as:

[0017] ;

[0018] Where, express function; express Convolutional layers;

[0019] Initially fusion feature map Input to the channel attention module to obtain the channel attention map ;

[0020] Edge-enhanced spatial attention map With channel attention map Add by broadcasting operation to get the spatial-channel attention map ;

[0021] The spatial-channel attention map With the preliminary fusion feature map At the same time, it is input into the pixel attention module to obtain the space-channel-pixel attention map , expressed as:

[0022] ;

[0023] Where, express Grouped convolutional layer; Represents batch normalization operation;

[0024] Through spatial-channel-pixel attention map Global semantic feature map and local detail feature maps Perform dynamic adaptive fusion and combine the output of dynamic adaptive fusion with the preliminary fusion feature map After adding, input into the segmentation head to obtain the road segmentation map , expressed as:

[0025] ;

[0026] Where, Indicates a split header.

[0027] Furthermore, the specific process of introducing elliptical or tree-shaped shadows into the high-resolution road image to obtain a high-resolution road enhanced image is as follows:

[0028] Step S3.11: randomly selecting different types of shadows, including elliptical shadows and tree shadows;

[0029] Step S3.12: After determining the shadow type, dynamically generate enhancement parameters for the corresponding shadow type based on the label information of the road area in the high-resolution road image;

[0030] Step S3.13: Based on the generated enhancement parameters of the corresponding type of shadow, call the ellipse shadow mask generation function or the tree shadow mask generation function to generate a shadow mask of the corresponding shape , expressed as:

[0031] ;

[0032] Where, represents the ellipse shadow mask generation function; represents the tree shadow mask generation function; Represents the coordinates of any pixel in a high-resolution road image; Indicates the type of shadow; Indicates the enhancement parameters of the generated elliptical shadow; represents the enhancement parameters of the generated tree shadows; Represents the label information of the road area in the high-resolution road image; Represents an elliptical shadow;

[0033] Step S3.14: Combine the shadow mask with the high-resolution road image through a weighted fusion formula Overlay according to transparency parameters to output high-resolution road enhanced images .

[0034] Furthermore, the specific process of step S3.12 is as follows:

[0035] For the elliptical shadow, the center coordinates of the elliptical shadow are first randomly sampled within the road area in the high-resolution road image. ; Generate random size parameters for elliptical shadows , Indicates the horizontal semi-axis length of the elliptical shadow, Indicates the vertical semi-axis length of the elliptical shadow; generates the transparency parameter used to control the depth of the elliptical shadow ;

[0036] For tree shadows, we first set a fixed template of a preset size in the road area of ​​the high-resolution road image, and then superimpose three predefined elliptical shadows on the fixed template to generate tree shadows. Then, we calculate the scaling factor based on the direction of the road in the high-resolution road image. To control the size of the fixed template, and finally generate the transparency parameter used to control the depth of the tree shadow And the horizontal flip flag used to control the horizontal flip of the tree shadow ;

[0037] Among them, the three predefined ellipse shadows include a center ellipse shadow, a left ellipse shadow, and a right ellipse shadow;

[0038] The process of dynamically generating enhanced parameters for the corresponding type of shadow is expressed as:

[0039] ;

[0040] ;

[0041] ;

[0042] Where, Indicates the enhancement parameters of the corresponding type of shadow generated; Indicates a fixed template Position coordinates within the road area in the high-resolution road image;

[0043] The ellipse shadow mask generation function is expressed as:

[0044] ;

[0045] Tree shadow mask generation function Expressed as:

[0046] ;

[0047] Where, Represents a resize operation, used to To adjust the fixed template size; Indicates a flip operation, used according to Flip the fixed template; Indicates a fixed template; Represents the coordinates of any pixel in the fixed template.

[0048] Furthermore, the local collaborative perception branch has a three-layer structure, the first layer of the local collaborative perception branch adopts the initialization layer of the ResNet18 network, the second layer of the local collaborative perception branch adopts the residual layer, and the third layer of the local collaborative perception branch adopts the multi-directional local feature enhancement module; the global perception branch has a five-layer structure, the first layer of the global perception branch adopts the PatchEmbed module, the second, third and fourth layers of the global perception branch adopt the first, second and third layers of the MambaVision network respectively, and the fifth layer of the global perception branch adopts the adaptive channel interaction attention module.

[0049] Furthermore, the specific process of the global perception branch to obtain the global semantic feature map of the low-resolution road enhanced image is as follows:

[0050] Step S4.11: Enhance the low-resolution road image Input PatchEmbed module to get embedded features ; Among them, the PatchEmbed module consists of two convolutional layers with a stride of 2; After two convolutional layers with a step size of 2, the embedded features are obtained ;

[0051] Step S4.12: Embedding features Input the first layer of the MambaVision network to get the first feature map ; Among them, the first layer of the MambaVision network consists of a convolution block, which contains two convolution layers with a stride of 1; the embedded features After passing through two convolutional layers with a step size of 1, the intermediate feature map is obtained , for the intermediate feature map The random path drop operation is used for regularization, and the regularized output is combined with the embedded features. Add and get the first feature map ;

[0052] Step S4.13: For the first feature map Perform downsampling operation to obtain the first downsampling feature map ;

[0053] Step S4.14: Subtract the first downsampled feature map Input the second layer of the MambaVision network to obtain the second feature map ; Among them, the second layer of the MambaVision network consists of three convolutional blocks connected in sequence. The convolutional blocks in the second layer of the MambaVision network have the same structure as the convolutional blocks in the first layer of the MambaVision network; the first down-sampled feature map After passing through the three convolution blocks in the second layer of the MambaVision network, the second feature map is obtained. ;

[0054] Step S4.14: For the second feature map Perform downsampling operation to obtain the second downsampling feature map ;

[0055] Step S4.15: Subtract the second downsampled feature map Input the third layer of the MambaVision network to obtain the third feature map ; Among them, the third layer of the MambaVision network consists of the first hybrid module, the second hybrid module and the third hybrid module connected in sequence; first, the second down-sampled feature map Flatten it, and then After the first mixing module, the second mixing module and the third mixing module in sequence, the third characteristic map is obtained ;

[0056] Step S4.16: The third feature map Input the adaptive channel interaction attention module to obtain the global semantic feature map .

[0057] Furthermore, the specific process of step S4.15 is as follows:

[0058] For the second downsampled feature map Flatten in the spatial dimension to obtain the feature sequence ;

[0059] The feature sequence Input the first mixing module and get the output of the first mixing module ; Wherein, the first hybrid module is composed of a first MLP branch and a Mamba branch connected in parallel;

[0060] The first MLP branch consists of two fully connected layers; the feature sequence After passing through two fully connected layers in sequence, the first feature enhancement sequence is obtained , which is the output of the first MLP branch;

[0061] In the Mamba branch, the characteristic sequence The dimension is increased through a linear transformation layer; the feature sequence after dimension increase After a one-dimensional convolution layer, the second feature enhancement sequence is obtained ; Enhance the second feature sequence Generate a gating signal through a linear transformation layer ; Through the gate signal To control the second feature enhancement sequence The information flow is obtained to obtain the third feature enhancement sequence ;

[0062] Enhance the sequence for the third feature Perform a selective sweep operation with a learnable time step parameter ranging from 0 to 1 , to dynamically adjust the fusion ratio of the third feature enhancement sequence input at the current moment and the third feature enhancement sequence input at the previous moment; restore the original dimension of the third feature enhancement sequence after adjusting the fusion ratio through a linear transformation layer and combine it with the second feature enhancement sequence Splice to get the fourth enhanced feature sequence , which is the output of the Mamba branch;

[0063] The fourth enhanced feature sequence After passing through a linear transformation layer and the first feature enhancement sequence Splicing is performed on the latitude to obtain the fifth enhanced feature sequence , the fifth enhanced feature sequence After the first MLP branch and random path discarding operation, the feature sequence Perform element-by-element addition to obtain the output of the first mixing module ;

[0064] The output of the first mixing module Input the second hybrid module; the second hybrid module consists of a parallel self-attention branch and a second MLP branch; the second MLP branch of the second hybrid module has the same structure as the first MLP branch of the first hybrid module; the output of the first hybrid module is converted into After the self-attention branch and the second MLP branch respectively, the output of the self-attention branch and the output of the second MLP branch are obtained respectively. The output of the self-attention branch and the output of the second MLP branch are spliced ​​to obtain the sixth feature enhancement sequence , enhance the sixth feature sequence After passing through the second MLP branch and performing random path discarding operation, the output of the first mixing module Perform element-by-element addition to obtain the output of the second mixing module ;

[0065] The structure of the third mixing module is the same as that of the second mixing module; the output of the second mixing module Input the third mixing module and get the output of the third mixing module ;

[0066] The output of the third mixing module Convert the feature sequence into a feature map to get the third feature map .

[0067] Furthermore, high-resolution road enhancement images are obtained through local collaborative perception branches Local detail feature map The specific process is:

[0068] Step S4.21: High-resolution road enhanced image Input the initialization layer of the ResNet18 network to obtain the fourth feature map The initialization layer of the ResNet18 network consists of a convolutional layer with a stride of 2 and a padding of 3, and a maximum pooling layer. The high-resolution road enhancement image is sequentially passed through the convolutional layer and the maximum pooling layer in the initialization layer of the ResNet18 network to obtain the fourth feature map. ;

[0069] Step S4.22: The fourth feature map Input the residual layer and get the output of the residual layer ; Among them, the residual layer consists of the first residual block and the second residual block. The first residual block and the second residual block have the same structure, both consisting of two convolutional layers with a step size of 1; the fourth feature map The output of the first residual block is obtained after two convolutional layers with a step size of 1 in the first residual block. The output of the first residual block is added to the output of the first residual block after two convolutional layers with a step size of 1 in the second residual block to obtain the output of the residual layer. ;

[0070] Step S4.23: The output of the residual layer Input the multi-directional local feature enhancement module to obtain the local detail feature map .

[0071] Furthermore, the multi-directional local feature enhancement module includes a depth-separable convolutional layer, a convolutional layer, and a local attention mechanism; the output of the residual layer is Input the multi-directional local feature enhancement module to obtain the local detail feature map The specific process is:

[0072] The output of the residual layer After passing through the horizontal depth separable convolution layer and one convolution layer in sequence, the horizontal features are obtained ; The output of the residual layer After passing through the vertical depth separable convolution layer and one convolution layer in sequence, the vertical features are obtained ; The horizontal feature , vertical features and the output of the residual layer Add together to get multi-directional fusion features ;

[0073] Generate multi-directional fusion features through local attention mechanism The channel weight , the channel weight and the output of the residual layer After multiplication, fusion features are added to multiple directions Add together to get the local detail feature map .

[0074] Compared with the existing technology, the present invention has the following beneficial effects:

[0075] (1) The present invention organically combines dynamic shadow synthesis data enhancement, parallel global and local collaborative perception branches, and dynamic two-way fusion to form a complete and efficient technical system. By generating diverse difficult samples through data enhancement, the global and local collaborative perception branches are provided with rich training materials, which enable them to fully utilize the advantages of complementary feature extraction under complex lighting and occlusion conditions; comprehensive features are extracted through the global and local collaborative perception branches, and then the dynamic two-way fusion module achieves deep fusion and optimization of features through adaptive interaction based on the triple attention mechanism; the present invention realizes a full-link upgrade from data generation, feature extraction, feature fusion to training optimization, significantly improving the model's recognition accuracy, robustness of road features and detection ability of small targets in complex environments, showing obvious advantages in road segmentation accuracy, detail integrity and processing of unbalanced data, effectively overcoming the problems of information loss, low fusion efficiency, training bias, etc. in the existing technology.

[0076] (2) The dynamic shadow synthesis data enhancement module designed in this invention can simulate complex lighting and occlusion scenes, so that the model can be exposed to a variety of difficult samples during the training phase, thereby improving the model's robust perception of road features in actual complex environments such as shadow coverage and partial occlusion.

[0077] (3) The present invention realizes the complementary extraction of low-resolution global semantic information and high-resolution local details through the parallel design of global perception branch and local collaborative perception branch; the global perception branch captures the semantic context through the hybrid layer of the MambaVision network and the adaptive channel interaction attention module, and the local collaborative perception branch uses the ResNet18 network and the multi-directional local feature enhancement module. The combination of the two improves the comprehensiveness of feature expression. Compared with traditional neural networks, it can more accurately balance global scene understanding and local detail depiction, avoiding semantic ambiguity or detail loss caused by relying only on single-scale features.

[0078] (4) The dynamic two-way fusion module designed in this invention dynamically adjusts the weights of the outputs of the global perception branch and the local collaborative perception branch through a triple attention mechanism of space, channel, and pixel, thus achieving adaptive interaction of features. Compared with fixed-weight fusion, this module can intelligently allocate fusion strategies based on the complexity of features in different regions, enhancing the segmentation accuracy of the model and avoiding the problem of dilution of key information or interference of secondary information caused by the inability to perceive differences in feature importance in traditional fixed fusion methods. This significantly improves the accuracy and detail richness of the segmentation results.

[0079] (5) The present invention introduces a loss function based on difficult sample mining. The loss is calculated by dynamically screening difficult samples with a predicted probability lower than a threshold. Combined with the difference in category weight balance, it avoids simple samples dominating the training, effectively alleviates the segmentation bias caused by imbalance, and improves the small target road recognition ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flowchart of the road segmentation model processing of the present invention. DETAILED DESCRIPTION

[0081] The present invention provides a technical solution: a road segmentation method based on local-global dual-stream information collaborative perception, comprising:

[0082] Step S1: Construct a data set, which includes several high-resolution road images, and the road areas in the high-resolution road images are all marked with labels.

[0083] The KITTI dataset was selected as the core data source. The KITTI dataset is collected from a rich variety of real-world scenarios, covering typical traffic environments such as urban streets, rural roads, and highways. It not only fully records real-world road images but also simultaneously collects corresponding radar information, fully recreating road scene details. Road types in the KITTI dataset can be categorized into three main types: urban unmarked (UU), urban marked (UM), and urban multiply marked (UMM). This detailed classification provides accurate data support for the subsequent training and testing of road segmentation models in different road scenarios.

[0084] Step S2: Construct a road segmentation model, such as Figure 1 As shown in FIG, the road segmentation model consists of a dynamic shadow synthesis data enhancement module, a local collaborative perception branch, a global perception branch, and a dynamic dual-path fusion module.

[0085] Step S3: The high-resolution road image is input into the dynamic shadow synthesis data enhancement module. Elliptical or tree-shaped shadows are first introduced into the high-resolution road image to obtain a high-resolution road enhanced image. The high-resolution road enhanced image is then reduced to obtain a low-resolution road enhanced image.

[0086] Among them, the dynamic shadow synthesis data enhancement module aims to simulate the complex situation of roads covered by shadows in real scenes, and enhance the adaptability and robustness of the road segmentation model to lighting changes through a probability-driven dual-mode shadow addition strategy.

[0087] The specific process of introducing elliptical or tree-shaped shadows into a high-resolution road image to obtain a high-resolution road enhanced image is as follows:

[0088] Step S3.11: Randomly select different types of shadows (elliptical shadows or tree shadows) with a probability of 50%, expressed as:

[0089] ;

[0090] Where, Indicates the type of shadow; Represents an elliptical shadow; Indicates tree shadows; Represents a random number between 0 and 1.

[0091] Step S3.12: After determining the shadow type, dynamically generate enhancement parameters for the corresponding shadow type based on the label information of the road area in the high-resolution road image; specifically:

[0092] For the elliptical shadow, the center coordinates of the elliptical shadow are first randomly sampled within the road area in the high-resolution road image. , and generate random size parameters for elliptical shadows , Represents the horizontal semi-axis length of the elliptical shadow, Indicates the vertical semi-axis length of the ellipse shadow; These two parameters together determine the size and shape of the elliptical shadow; in addition, a transparency parameter is generated to control the depth of the elliptical shadow. .

[0093] For tree shadows, we first set a fixed template of 400×200 in the road area of ​​the high-resolution road image, and superimpose three predefined elliptical shadows on the fixed template to generate tree shadows; then calculate the scaling factor according to the direction of the road in the high-resolution road image. To control the size of the fixed template, and finally generate the transparency parameter used to control the depth of the tree shadow And the horizontal flip flag used to control the horizontal flip of the tree shadow .

[0094] Among them, the three predefined elliptical shadows include the central elliptical shadow, the left elliptical shadow and the right elliptical shadow; the center coordinates of the central elliptical shadow are (200, 67), its horizontal semi-axis length is 133, and its vertical semi-axis length is 50; the center coordinates of the left elliptical shadow are (100, 100), its horizontal semi-axis length is 80, and its vertical semi-axis length is 67; the center coordinates of the right elliptical shadow are (300, 100), its horizontal semi-axis length is 80, and its vertical semi-axis length is 67.

[0095] The process of dynamically generating the enhancement parameters of the corresponding type of shadow can be expressed as:

[0096] ;

[0097] ;

[0098] ;

[0099] Where, Indicates the enhancement parameters of the corresponding type of shadow generated; Indicates the enhancement parameters of the generated elliptical shadow; represents the enhancement parameters of the generated tree shadows; Represents the label information of the road area in the high-resolution road image; Represents the position coordinates of the fixed template within the road area in the high-resolution road image.

[0100] Step S3.13: Based on the generated enhancement parameters of the corresponding type of shadow, call the ellipse shadow mask generation function or the tree shadow mask generation function to generate a shadow mask of the corresponding shape , which can be expressed as:

[0101] ;

[0102] Where, represents the ellipse shadow mask generation function; represents the tree shadow mask generation function; Represents the coordinates of any pixel in a high-resolution road image.

[0103] Among them, the ellipse shadow mask generation function can be expressed as:

[0104] ;

[0105] Among them, the tree shadow mask generation function It can be expressed as:

[0106] ;

[0107] Where, Represents a resize operation, used to To adjust the fixed template size; Indicates a flip operation, used according to Flip the fixed template; Indicates a fixed template; Represents the coordinates of any pixel in the fixed template.

[0108] in, Expressed as:

[0109] .

[0110] Step S3.14: Combine the shadow mask with the high-resolution road image through a weighted fusion formula Overlay according to transparency parameters to output high-resolution road enhanced images .

[0111] This process not only simulates the morphological changes of real shadows, but also ensures the diversity of each enhancement through dynamic parameter adjustment, thereby effectively improving the segmentation performance of the model in complex lighting scenes.

[0112] Among them, high-resolution road enhancement images Zoom out to get a low-resolution road enhanced image The specific process is: high-resolution road enhanced image The size of the image is 1240 × 375, and the high-resolution road enhancement image is transformed into The size is adjusted to 310×188 to obtain a low-resolution road enhanced image .

[0113] Step S4: obtaining a global semantic feature map of the low-resolution road enhanced image through the global perception branch, and obtaining a local detail feature map of the high-resolution road enhanced image through the local collaborative perception branch.

[0114] Among them, the local collaborative perception branch has a three-layer structure. The first layer of the local collaborative perception branch adopts the initialization layer of the ResNet18 network, the second layer of the local collaborative perception branch adopts the residual layer, and the third layer of the local collaborative perception branch adopts a multi-directional local feature enhancement module.

[0115] Among them, the global perception branch has a five-layer structure. The first layer of the global perception branch adopts the PatchEmbed module, the second, third and fourth layers of the global perception branch adopt the first, second and third layers of the MambaVision network respectively, and the fifth layer of the global perception branch adopts the adaptive channel interaction attention module.

[0116] The specific process of the global perception branch obtaining the global semantic feature map of the low-resolution road enhanced image is as follows:

[0117] Step S4.11: Enhance the low-resolution road image Input PatchEmbed module to get embedded features ; Among them, the PatchEmbed module consists of two convolutional layers (3×3) with a stride of 2; After two convolutional layers (3×3) with a stride of 2, the embedded features are obtained , expressed as:

[0118] ;

[0119] ;

[0120] Where, express The output after the first convolutional layer in the PatchEmbed module; express Activation function; Represents batch normalization operation; Represents a 3×3 convolutional layer.

[0121] Step S4.12: Embedding features Input the first layer of the MambaVision network to get the first feature map ; Among them, the first layer of the MambaVision network consists of a convolution block, which contains two convolution layers (3×3) with a stride of 1; the embedded features After passing through two convolutional layers (3×3) with a step size of 1, the intermediate feature map is obtained , for the intermediate feature map The random path drop operation is used for regularization, and the regularized output is combined with the embedded features. Add and get the first feature map , expressed as:

[0122] ;

[0123] ;

[0124] ;

[0125] Where, Represents embedded features Output after the first convolutional layer (3×3) with a stride of 1 in the convolutional block of the MambaVision network; express Activation function; Indicates a random path drop operation.

[0126] Step S4.13: For the first feature map Perform downsampling operation to obtain the first downsampling feature map , expressed as:

[0127] .

[0128] Step S4.14: Subtract the first downsampled feature map Input the second layer of the MambaVision network to obtain the second feature map ; Among them, the second layer of the MambaVision network consists of three convolutional blocks connected in sequence. The convolutional blocks in the second layer of the MambaVision network have the same structure as the convolutional blocks in the first layer of the MambaVision network; the first down-sampled feature map After passing through the three convolution blocks in the second layer of the MambaVision network, the second feature map is obtained. .

[0129] Step S4.14: For the second feature map Perform downsampling operation to obtain the second downsampling feature map , expressed as:

[0130] ;

[0131] Step S4.15: Subtract the second downsampled feature map Input the third layer of the MambaVision network to obtain the third feature map ; Among them, the third layer of the MambaVision network consists of the first hybrid module, the second hybrid module and the third hybrid module connected in sequence; first, the second down-sampled feature map Flatten it, and then After the first mixing module, the second mixing module and the third mixing module in sequence, the third characteristic map is obtained Specific:

[0132] For the second downsampled feature map Flatten in the spatial dimension to obtain the feature sequence , expressed as:

[0133] ;

[0134] Where, Represents a flatten operation.

[0135] The feature sequence Input the first mixing module and get the output of the first mixing module ; Among them, the first hybrid module consists of a first MLP branch and a Mamba branch connected in parallel.

[0136] The first MLP branch consists of two fully connected layers; the feature sequence After passing through two fully connected layers in sequence, the first feature enhancement sequence is obtained , which is the output of the first MLP branch, is expressed as:

[0137] ;

[0138] Where, represents the first fully connected layer in the first MLP branch; represents the second fully connected layer in the first MLP branch.

[0139] In the Mamba branch, the characteristic sequence The dimension is increased through a linear transformation layer; the feature sequence after dimension increase After a one-dimensional convolution layer, the second feature enhancement sequence is obtained ; Enhance the second feature sequence Generate a gating signal through a linear transformation layer ( Taking values ​​between 0 and 1, it is equivalent to assigning a learnable "switch" to each feature dimension, dynamically deciding which information needs to be retained and which information can be ignored, thereby enhancing the model's attention to important features and suppressing noise); through the gating signal To control the second feature enhancement sequence The information flow is obtained to obtain the third feature enhancement sequence , expressed as:

[0140] ;

[0141] ;

[0142] ;

[0143] Where, express Activation function; represents a one-dimensional convolutional layer; represents the linear transformation layer; Represents an element-wise multiplication operation.

[0144] Enhance the sequence for the third feature Perform selective scanning operation. During this process, the Mamba branch will combine the third feature enhancement sequence input at the current moment With the historical state (the third feature enhancement sequence input at the previous moment ), through a learnable timestep parameter ranging from 0 to 1 , to dynamically adjust the fusion ratio of new and old information (when When it is close to 1, it will pay more attention to the current input ,when When it is close to 0, it is more dependent on the third feature enhancement sequence input at the previous moment ); The third feature enhancement sequence after adjusting the fusion ratio After restoring the original dimension through a linear transformation layer and the second feature enhancement sequence Splice to get the fourth enhanced feature sequence , which is the output of the Mamba branch, is expressed as:

[0145] ;

[0146] ;

[0147] Where, Represents a splicing operation.

[0148] The fourth enhanced feature sequence After passing through a linear transformation layer and the first feature enhancement sequence Splicing is performed on the latitude to obtain the fifth enhanced feature sequence , the fifth enhanced feature sequence After the first MLP branch and random path discarding operation, the feature sequence Perform element-by-element addition to obtain the output of the first mixing module , expressed as:

[0149] ;

[0150] ;

[0151] Where, represents the first MLP branch.

[0152] The output of the first mixing module Input the second hybrid module; the second hybrid module consists of a parallel self-attention branch (Attention) and a second MLP branch; the second MLP branch of the second hybrid module has the same structure as the first MLP branch of the first hybrid module, and its processing flow is not described here; the output of the first hybrid module is After the self-attention branch and the second MLP branch respectively, the output of the self-attention branch and the output of the second MLP branch are obtained respectively. The output of the self-attention branch and the output of the second MLP branch are spliced ​​to obtain the sixth feature enhancement sequence , enhance the sixth feature sequence After passing through the second MLP branch and performing random path discarding operation, the output of the first mixing module Perform element-by-element addition to obtain the output of the second mixing module .

[0153] The structure of the third mixing module is the same as that of the second mixing module, and its processing flow will not be described in detail here.

[0154] The output of the third mixing module Convert the feature sequence into a feature map to get the third feature map .

[0155] Step S4.16: The third feature map Input the adaptive channel interaction attention module (ECA) to obtain the global semantic feature map .

[0156] The function of the adaptive channel interaction attention module (ECA) is to adaptively adjust the importance of each channel of the feature map, effectively enhance the distinction between the road area and the background, and improve the segmentation accuracy.

[0157] In the adaptive channel interaction attention module, the third feature map is pooled by global average pooling. The spatial dimension of is compressed and transformed to obtain the feature vector ; The feature vector After a one-dimensional convolution layer, the dimension is expanded and the output after dimension expansion is combined with the third feature map Perform element-by-element multiplication to obtain the global semantic feature map .

[0158] Among them, high-resolution road enhanced images are obtained through local collaborative perception branches Local detail feature map The specific process is:

[0159] Step S4.21: High-resolution road enhanced image Input the initialization layer of the ResNet18 network to obtain the fourth feature map The initialization layer of the ResNet18 network consists of a convolutional layer (7×7) with a stride of 2 and a padding of 3, and a maximum pooling layer. The high-resolution road enhancement image is sequentially passed through the convolutional layer (7×7) and the maximum pooling layer in the initialization layer of the ResNet18 network to obtain the fourth feature map. , expressed as:

[0160] ;

[0161] Where, represents the maximum pooling layer; Represents a 7×7 convolutional layer.

[0162] Step S4.22: The fourth feature map Input the residual layer and get the output of the residual layer ; Among them, the residual layer consists of the first residual block and the second residual block. The first residual block and the second residual block have the same structure, both consisting of two convolutional layers (3×3) with a stride of 1; the fourth feature map The output of the first residual block is obtained after two convolutional layers (3×3) with a stride of 1 in the first residual block. The output of the first residual block is added to the output of the first residual block after two convolutional layers (3×3) with a stride of 1 in the second residual block to obtain the output of the residual layer. .

[0163] Step S4.23: The output of the residual layer Input the multi-directional local feature enhancement module to obtain the local detail feature map The multi-directional local feature enhancement module is used to enhance the road-related local structural information in the extracted feature map through multi-directional feature extraction, local attention mechanism and residual connection, while retaining spatial details.

[0164] Among them, the multi-directional local feature enhancement module includes a depth-separable convolutional layer, a convolutional layer, and a local attention mechanism; the output of the residual layer is After passing through the horizontal depth separable convolution layer and one convolution layer in sequence, the horizontal features are obtained ; The output of the residual layer After passing through the vertical depth separable convolution layer and one convolution layer in sequence, the vertical features are obtained ; The horizontal feature , vertical features and the output of the residual layer Add together to get multi-directional fusion features , expressed as:

[0165] ;

[0166] ;

[0167] ;

[0168] Where, The convolution kernel is The depth-wise separable convolutional layer is the horizontal depth-wise separable convolutional layer. The convolution kernel is The depth-wise separable convolutional layer is the vertical depth-wise separable convolutional layer.

[0169] Generate multi-directional fusion features through local attention mechanism The channel weight , the channel weight and the output of the residual layer After multiplication, multi-directional fusion features Add together to get the local detail feature map .

[0170] Step S5: Transform the global semantic feature map After feature alignment and local detail feature map Add element by element to get the initial fusion feature map , the initial fusion feature map Input into the dynamic dual-path fusion module to obtain the road segmentation map .

[0171] Among them, the dynamic two-way fusion module consists of a spatial attention module, a channel attention module, an edge auxiliary branch, a pixel attention module and a segmentation head; the initial fusion feature map Input into the dynamic dual-path fusion module to obtain the road segmentation map The processing flow is:

[0172] Initially fusion feature map Input into the spatial attention module to obtain the spatial attention map , expressed as:

[0173] ;

[0174] Where, represents the average pooling layer; Represents the maximum pooling layer.

[0175] In order to further enhance the sensitivity to road edges, an edge auxiliary branch is introduced to the spatial attention map To enhance; specifically, the spatial attention map And the preliminary fusion feature map Input the edge auxiliary branch to obtain the edge enhanced spatial attention map , expressed as:

[0176] ;

[0177] Where, express function.

[0178] Initially fusion feature map Input to the channel attention module to obtain the channel attention map .

[0179] Edge-enhanced spatial attention map With channel attention map Add by broadcasting operation to get the spatial-channel attention map , expressed as:

[0180] ;

[0181] Where, Represents a broadcast operation.

[0182] The spatial-channel attention map With the preliminary fusion feature map At the same time, it is input into the pixel attention module to obtain the space-channel-pixel attention map , expressed as:

[0183] ;

[0184] Where, express Grouped convolutional layer.

[0185] Through spatial-channel-pixel attention map Global semantic feature map and local detail feature maps Perform dynamic adaptive fusion and combine the output of dynamic adaptive fusion with the preliminary fusion feature map After adding, input into the segmentation head to obtain the road segmentation map , expressed as:

[0186] ;

[0187] Where, Represents the segmentation head, which consists of two convolutional layers (1×1) connected in sequence.

[0188] In the road segmentation task, the cross entropy loss function is a commonly used optimization method, but this method has limitations when dealing with sample imbalance. Taking road segmentation as an example, the number of pixels in the road area is often significantly less than that in the non-road area. This makes the non-road area likely to dominate the model learning direction during model training, making it easier for the model to misclassify pixels as non-road areas during pixel classification, thereby affecting segmentation accuracy. To solve this problem, a loss function based on difficult sample mining is introduced. As the loss function of the road segmentation model, this loss function effectively improves the ability to handle sample imbalance by dynamically screening difficult samples and calculating the cross-entropy loss. DHM-CELoss is expressed as:

[0189] ;

[0190] Where, Represents the predicted probability distribution of the road segmentation model; represents the true category; represents the set of filtered difficult samples (high-resolution road images); express The corresponding category weight; represents the negative logarithmic function; Represents the road segmentation model prediction The probability distribution of samples; Indicates the The samples belong to probability; Indicates the The true category of the samples; Indicates the samples belong to the category probability; Indicates the total number of categories.

[0191] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A road segmentation method based on local-global dual-stream information collaborative perception, characterized in that: The steps include: Step S1: Construct a data set, which includes several high-resolution road images, and the road areas in the high-resolution road images are all marked with labels; Step S2: Constructing a road segmentation model; the road segmentation model consists of a dynamic shadow synthesis data enhancement module, a local collaborative perception branch, a global perception branch, and a dynamic dual-path fusion module; Step S3: Inputting the high-resolution road image into the dynamic shadow synthesis data enhancement module, first introducing elliptical or tree-shaped shadows into the high-resolution road image to obtain a high-resolution road enhanced image, and then reducing the high-resolution road enhanced image to obtain a low-resolution road enhanced image; Step S4: obtaining a global semantic feature map of the low-resolution road enhanced image through the global perception branch, and obtaining a local detail feature map of the high-resolution road enhanced image through the local collaborative perception branch; Step S5: After feature alignment, the global semantic feature map is added element-by-element to the local detail feature map to obtain a preliminary fusion feature map, which is then input into the dynamic two-way fusion module to obtain a road segmentation map. The dynamic two-way fusion module consists of a spatial attention module, a channel attention module, an edge auxiliary branch, a pixel attention module and a segmentation head; the initial fusion feature map Input into the dynamic dual-path fusion module to obtain the road segmentation map The specific process is: Initially fusion feature map Input into the spatial attention module to obtain the spatial attention map , expressed as: ； Where, express Convolutional layers; Represents a splicing operation; represents the average pooling layer; represents the maximum pooling layer; The spatial attention map And the preliminary fusion feature map Input the edge auxiliary branch to obtain the edge enhanced spatial attention map , expressed as: ； Where, express function; express Convolutional layers; Initially fusion feature map Input to the channel attention module to obtain the channel attention map ; Edge-enhanced spatial attention map With channel attention map Add by broadcasting operation to get the spatial-channel attention map ; The spatial-channel attention map With the preliminary fusion feature map At the same time, it is input into the pixel attention module to obtain the space-channel-pixel attention map , expressed as: ； Where, express Grouped convolutional layer; Represents batch normalization operation; Through spatial-channel-pixel attention map Global semantic feature map and local detail feature maps Perform dynamic adaptive fusion and combine the output of dynamic adaptive fusion with the preliminary fusion feature map After adding, input into the segmentation head to obtain the road segmentation map , expressed as: ； Where, Indicates a split header.

2. The road segmentation method based on local-global dual-stream information collaborative perception according to claim 1 is characterized by: The specific process of introducing elliptical or tree-shaped shadows into a high-resolution road image to obtain a high-resolution road enhanced image is as follows: Step S3.11: randomly selecting different types of shadows, including elliptical shadows and tree shadows; Step S3.12: After determining the shadow type, dynamically generate enhancement parameters for the corresponding shadow type based on the label information of the road area in the high-resolution road image; Step S3.13: Based on the generated enhancement parameters of the corresponding type of shadow, call the ellipse shadow mask generation function or the tree shadow mask generation function to generate a shadow mask of the corresponding shape , expressed as: ； Where, represents the ellipse shadow mask generation function; represents the tree shadow mask generation function; Represents the coordinates of any pixel in a high-resolution road image; Indicates the type of shadow; Indicates the enhancement parameters of the generated elliptical shadow; represents the enhancement parameters of the generated tree shadows; Represents the label information of the road area in the high-resolution road image; Represents an elliptical shadow; Step S3.14: Combine the shadow mask with the high-resolution road image through a weighted fusion formula Overlay according to transparency parameters to output high-resolution road enhanced images .

3. The road segmentation method based on local-global dual-stream information collaborative perception according to claim 2 is characterized by: The specific process of step S3.12 is: For the elliptical shadow, the center coordinates of the elliptical shadow are first randomly sampled within the road area in the high-resolution road image. ; Generate random size parameters for elliptical shadows , Represents the horizontal semi-axis length of the elliptical shadow, Indicates the vertical semi-axis length of the elliptical shadow; generates the transparency parameter used to control the depth of the elliptical shadow ; For tree shadows, we first set a fixed template of a preset size in the road area of ​​the high-resolution road image, and then superimpose three predefined elliptical shadows on the fixed template to generate tree shadows. Then, we calculate the scaling factor based on the direction of the road in the high-resolution road image. To control the size of the fixed template, and finally generate the transparency parameter used to control the depth of the tree shadow And the horizontal flip flag used to control the horizontal flip of the tree shadow ; Among them, the three predefined ellipse shadows include a center ellipse shadow, a left ellipse shadow, and a right ellipse shadow; The process of dynamically generating enhanced parameters for the corresponding type of shadow is expressed as: ； ； ； Where, Indicates the enhancement parameters of the corresponding type of shadow generated; Indicates a fixed template Position coordinates within the road area in the high-resolution road image; The ellipse shadow mask generation function is expressed as: ； Tree shadow mask generation function Expressed as: ； Where, Represents a resize operation, used to To adjust the fixed template size; Indicates a flip operation, used according to Flip the fixed template; Indicates a fixed template; Represents the coordinates of any pixel in the fixed template.

4. The road segmentation method based on local-global dual-stream information collaborative perception according to claim 3 is characterized by: The local collaborative perception branch has a three-layer structure. The first layer of the local collaborative perception branch adopts the initialization layer of the ResNet18 network, the second layer of the local collaborative perception branch adopts the residual layer, and the third layer of the local collaborative perception branch adopts the multi-directional local feature enhancement module. The global perception branch has a five-layer structure. The first layer of the global perception branch adopts the PatchEmbed module, the second, third and fourth layers of the global perception branch adopt the first, second and third layers of the MambaVision network respectively, and the fifth layer of the global perception branch adopts the adaptive channel interaction attention module.

5. The road segmentation method based on local-global dual-stream information collaborative perception according to claim 4 is characterized by: The specific process of the global perception branch to obtain the global semantic feature map of the low-resolution road enhanced image is as follows: Step S4.11: Enhance the low-resolution road image Input PatchEmbed module to get embedded features ; Among them, the PatchEmbed module consists of two convolutional layers with a stride of 2; After two convolutional layers with a step size of 2, the embedded features are obtained ; Step S4.12: Embedding features Input the first layer of the MambaVision network to get the first feature map ; Among them, the first layer of the MambaVision network consists of a convolution block, which contains two convolution layers with a stride of 1; the embedded features After passing through two convolutional layers with a step size of 1, the intermediate feature map is obtained , for the intermediate feature map The random path drop operation is used for regularization, and the regularized output is combined with the embedded features. Add and get the first feature map ; Step S4.13: For the first feature map Perform downsampling operation to obtain the first downsampling feature map ; Step S4.14: Subtract the first downsampled feature map Input the second layer of the MambaVision network to obtain the second feature map ; Among them, the second layer of the MambaVision network consists of three convolutional blocks connected in sequence. The convolutional blocks in the second layer of the MambaVision network have the same structure as the convolutional blocks in the first layer of the MambaVision network; the first down-sampled feature map After passing through the three convolution blocks in the second layer of the MambaVision network, the second feature map is obtained. ; Step S4.14: For the second feature map Perform downsampling operation to obtain the second downsampling feature map ; Step S4.15: Subtract the second downsampled feature map Input the third layer of the MambaVision network to obtain the third feature map ; Among them, the third layer of the MambaVision network consists of the first hybrid module, the second hybrid module and the third hybrid module connected in sequence; first, the second down-sampled feature map Flatten it, and then After the first mixing module, the second mixing module and the third mixing module in sequence, the third characteristic map is obtained ; Step S4.16: The third feature map Input the adaptive channel interaction attention module to obtain the global semantic feature map .

6. The road segmentation method based on local-global dual-stream information collaborative perception according to claim 5 is characterized by: The specific process of step S4.15 is as follows: For the second downsampled feature map Flatten in the spatial dimension to obtain the feature sequence ; The feature sequence Input the first mixing module and get the output of the first mixing module ; Wherein, the first hybrid module is composed of a first MLP branch and a Mamba branch connected in parallel; The first MLP branch consists of two fully connected layers; the feature sequence After passing through two fully connected layers in sequence, the first feature enhancement sequence is obtained , which is the output of the first MLP branch; In the Mamba branch, the characteristic sequence The dimension is increased through a linear transformation layer; the feature sequence after dimension increase After a one-dimensional convolution layer, the second feature enhancement sequence is obtained ; Enhance the second feature sequence Generate a gating signal through a linear transformation layer ; Through the gate signal To control the second feature enhancement sequence The information flow is obtained to obtain the third feature enhancement sequence ; Enhance the sequence for the third feature Perform a selective sweep operation with a learnable time step parameter ranging from 0 to 1 , to dynamically adjust the fusion ratio of the third feature enhancement sequence input at the current moment and the third feature enhancement sequence input at the previous moment; restore the original dimension of the third feature enhancement sequence after adjusting the fusion ratio through a linear transformation layer and combine it with the second feature enhancement sequence Splice to get the fourth enhanced feature sequence , which is the output of the Mamba branch; The fourth enhanced feature sequence After passing through a linear transformation layer and the first feature enhancement sequence Splicing is performed on the latitude to obtain the fifth enhanced feature sequence , the fifth enhanced feature sequence After the first MLP branch and random path discarding operation, the feature sequence Perform element-by-element addition to obtain the output of the first mixing module ; The output of the first mixing module Input the second hybrid module; the second hybrid module consists of a parallel self-attention branch and a second MLP branch; the second MLP branch of the second hybrid module has the same structure as the first MLP branch of the first hybrid module; the output of the first hybrid module is converted into After the self-attention branch and the second MLP branch respectively, the output of the self-attention branch and the output of the second MLP branch are obtained respectively. The output of the self-attention branch and the output of the second MLP branch are spliced ​​to obtain the sixth feature enhancement sequence , enhance the sixth feature sequence After passing through the second MLP branch and performing random path discarding operation, the output of the first mixing module Perform element-by-element addition to obtain the output of the second mixing module ; The structure of the third mixing module is the same as that of the second mixing module; the output of the second mixing module Input the third mixing module and get the output of the third mixing module ; The output of the third mixing module Convert the feature sequence into a feature map to get the third feature map .

7. The road segmentation method based on local-global dual-stream information collaborative perception according to claim 6 is characterized by: Obtaining high-resolution road enhanced images through local collaborative perception branches Local detail feature map The specific process is: Step S4.21: High-resolution road enhanced image Input the initialization layer of the ResNet18 network to obtain the fourth feature map The initialization layer of the ResNet18 network consists of a convolutional layer with a stride of 2 and a padding of 3, and a maximum pooling layer. The high-resolution road enhancement image is sequentially passed through the convolutional layer and the maximum pooling layer in the initialization layer of the ResNet18 network to obtain the fourth feature map. ; Step S4.22: The fourth feature map Input the residual layer and get the output of the residual layer ; Among them, the residual layer consists of the first residual block and the second residual block. The first residual block and the second residual block have the same structure, both consisting of two convolutional layers with a step size of 1; the fourth feature map The output of the first residual block is obtained after two convolutional layers with a step size of 1 in the first residual block. The output of the first residual block is added to the output of the first residual block after two convolutional layers with a step size of 1 in the second residual block to obtain the output of the residual layer. ; Step S4.23: The output of the residual layer Input the multi-directional local feature enhancement module to obtain the local detail feature map .

8. The road segmentation method based on local-global dual-stream information collaborative perception according to claim 7 is characterized by: The multi-directional local feature enhancement module includes a depth-separable convolutional layer, a convolutional layer, and a local attention mechanism; the output of the residual layer is Input the multi-directional local feature enhancement module to obtain the local detail feature map The specific process is: The output of the residual layer After passing through the horizontal depth separable convolution layer and one convolution layer in sequence, the horizontal features are obtained ; The output of the residual layer After passing through the vertical depth separable convolution layer and one convolution layer in sequence, the vertical features are obtained ; The horizontal feature , vertical features and the output of the residual layer Add together to get multi-directional fusion features ; Generate multi-directional fusion features through local attention mechanism The channel weight , the channel weight and the output of the residual layer After multiplication, fusion features are added to multiple directions Add together to get the local detail feature map .

Images