An unmanned vehicle-oriented road surface segmentation method

Abstract

This invention relates to a road segmentation method for autonomous driving, specifically in the field of road image segmentation technology. The invention addresses the problems of low accuracy in overlapping target segmentation, loss of segmentation boundaries, and low road segmentation accuracy in existing autonomous driving road image segmentation methods. The invention includes: reducing the feature resolution of the road image; performing dimensionality reduction on the reduced-resolution road image features to obtain feature X, thereby obtaining low-frequency features of the road image; using the low-frequency features and X to obtain high-frequency features of the road image; aggregating the low-frequency and high-frequency features to obtain aggregated frequency features, thereby obtaining frequency domain features; and then using the spatial domain features X to obtain the final spatial domain features. s Perform a dimensionality transformation to obtain feature X. A ; For X s A series of operations are performed to obtain the cross feature vector K. s V s Using K s V s and X A The invention obtains mixed-domain features and utilizes them to generate deep features, thereby generating a segmented image. This invention is used for segmenting road surface images.

Description

Technical Field

[0001] This invention relates to the field of road surface image segmentation technology, and in particular to a road surface segmentation method for autonomous driving. Background Technology

[0002] With the rapid development of autonomous driving, stable and real-time road scene segmentation is crucial for the safe operation of autonomous driving systems, and semantic segmentation provides technical support for road segmentation tasks. Semantic segmentation involves classifying all pixel labels in an image, and compared to other visual tasks, it has two key characteristics: pixel-wise dense prediction and multi-class representation. While deep learning-based semantic segmentation methods have achieved some success in general pixel classification tasks, they perform poorly in road segmentation tasks within complex urban road scenes. Road scene segmentation requires acquiring both pixel detail information and long-range contextual information from the image; however, most semantic segmentation methods only focus on one aspect. Therefore, improving the accuracy of road segmentation remains a challenge.

[0003] Traditional road segmentation methods rely on manual feature extraction to address pixel-level label assignment, such as threshold selection, superpixel extraction, and graph algorithms. With the continuous development of deep learning, a series of methods based on FCN (Fully Convolutional Neural Network) have achieved good performance in semantic segmentation tasks. DeepLabV3+ and PSPNet expand the receptive field by introducing spatial pyramid pooling modules to fuse features at different levels. HRNet enhances semantic information by parallelizing multiple high-resolution branches and simultaneously interacting branch features. OCNet enhances the features output by the backbone network by querying the global context. However, these semantic segmentation methods are limited by the receptive field of convolutional neural networks and cannot establish effective contextual dependencies between distant pixels in the image, resulting in low segmentation accuracy in complex and cluttered road scenes.

[0004] Recently, Transformer has demonstrated good performance in semantic segmentation. DPT uses Transformer as an encoder to improve performance in dense prediction tasks. SETR proposes a sequence-to-sequence approach using a pre-trained VisionTransformer (Vit) as a feature extraction network. However, since Viit only extracts features in the spatial domain, SETR loses a large amount of pixel detail information, leading to the loss of segmentation boundary information. SegFormer improves efficiency by introducing a hierarchical Transformer encoder and a lightweight fully connected neural network decoder. Existing Transformer-based semantic segmentation methods use attention mechanisms to update image semantic information; however, most attention mechanisms cannot simultaneously learn the global structure and local features of the image, resulting in incomplete local image structure and thus low segmentation accuracy.

[0005] In summary, current road segmentation methods for autonomous driving have the following main shortcomings:

[0006] (1) Existing road segmentation methods cannot establish contextual dependencies on remote pixels in the image, resulting in low segmentation accuracy for overlapping or incomplete objects in road scenes.

[0007] (2) Existing road segmentation methods only focus on the spatial domain features of the image and do not take into account the interaction between features of different domains. This limits the receptive field of a single spatial domain feature and makes it impossible to obtain the pixel details of the image, resulting in the loss of segmentation boundary information.

[0008] (3) Existing road segmentation methods cannot process pixel information at different locations and cannot learn the local structure of the image, resulting in low road segmentation accuracy in complex road scenes. Summary of the Invention

[0009] The purpose of this invention is to address the problems of low segmentation accuracy of overlapping targets, loss of segmentation boundaries, and low road segmentation accuracy in complex scenarios in existing autonomous driving road segmentation methods, and to propose a road segmentation method for autonomous driving.

[0010] The specific process of a road segmentation method for autonomous driving is as follows:

[0011] The road surface image to be segmented is obtained and input into the trained road surface segmentation network to obtain the segmented road surface image.

[0012] The road segmentation network includes: a feature extraction subnetwork, a cross-space-frequency domain cross-attention subnetwork, and a parallel gated feedforward subnetwork;

[0013] The feature extraction subnetwork includes: a feature resolution reduction module, a dynamic frequency capture kernel, and a linear attention operator module;

[0014] The feature resolution reduction module is used to reduce the resolution of road surface image features to a preset resolution value, and to convert the reduced resolution road surface image features... Send to the dynamic capture kernel and linear attention operator modules;

[0015] The dynamic frequency acquisition kernel includes: an adaptive low-frequency filtering layer, an adaptive high-frequency filtering layer, and a weight-sharing-based decomposition attention layer;

[0016] The adaptive low-frequency filtering layer is used to filter road surface image features with reduced resolution. Dimensionality reduction is performed to obtain feature X. Low-frequency features of the road surface image are obtained using feature X, and the low-frequency features of the road surface image are sent to the adaptive high-frequency filtering layer and the weight-sharing-based decomposition attention layer.

[0017] The adaptive high-frequency filtering layer uses the low-frequency features and feature X of the road surface image to obtain the high-frequency features of the road surface image, and sends the high-frequency features of the road surface image to the decomposition attention layer based on weight sharing.

[0018] The weight-sharing-based decomposition attention layer first aggregates the low-frequency and high-frequency features of the road image to obtain aggregated frequency features, and then performs a weight-sharing-based decomposition attention operation on the aggregated frequency features to generate frequency domain features. and frequency domain features Send to the cross-space-frequency domain cross-attention subnetwork;

[0019] The linear attention operator module utilizes road surface image features at reduced resolution. Obtaining spatial domain features X s And the spatial domain feature X s Send to the cross-space-frequency domain cross-attention subnetwork;

[0020] The cross-space-frequency domain cross-attention subnetwork utilizes frequency domain features and spatial domain features X s Obtaining hybrid domain features and hybrid domain features The data is sent to the parallel gated feedforward subnetwork; the cross-space-frequency domain cross-attention subnetwork includes: a frequency domain feature dimension transformation module, a cross feature vector acquisition module, and a hybrid domain feature acquisition module;

[0021] The frequency domain feature dimension conversion module is used to convert frequency domain features. Perform dimensional transformation to obtain feature X A and feature XA Send to the hybrid domain feature acquisition module;

[0022] The cross feature vector acquisition module is used to obtain spatial domain features X. s Perform normalization, pooling, convolution, splitting, and dimension transformation operations to obtain two cross-feature vectors K. s V s and K s V s Send to the hybrid domain feature acquisition module;

[0023] The hybrid domain feature acquisition module utilizes the cross feature vector K s V s and feature X A To obtain hybrid domain features and hybrid domain features Send to the parallel gated feedforward subnetwork;

[0024] The parallel gated feedforward subnetwork utilizes hybrid domain features Obtaining deep features And utilize deep features A segmented image is generated; the parallel gated feedforward subnetwork includes: a hybrid domain feature grouping module, a branch feature acquisition module, a feature stitching module, and a segmented image output module;

[0025] The hybrid domain feature grouping module is used to group hybrid domain features Split into feature X along the channel dimension F1 ,X F2 and feature X F1 ,X F2 Send branch feature acquisition module;

[0026] The branch feature acquisition module utilizes feature X F1 ,X F2 Generate branch features Y1 and Y2 respectively, and send the branch features to the feature concatenation module;

[0027] The feature concatenation module is used to aggregate Y1 and Y2 to generate deep features. and deep features Send to the segmented image output module;

[0028] The image segmentation output module utilizes deep features Generate a segmented image.

[0029] Furthermore, the feature resolution reduction module includes: a convolutional layer and three consecutive residual layers.

[0030] Furthermore, the adaptive low-frequency filtering layer is used to filter road surface image features with reduced resolution. Dimensionality reduction is performed to obtain feature X. Then, feature X is used to obtain low-frequency features of the road surface image, specifically:

[0031]

[0032]

[0033] in, reshape(·) represents the first dimension transformation operation. X represents the reduced-resolution road surface image features after dimensionality reduction, and ALF(X) represents the low-frequency features of the road surface image. m This indicates that the feature X is divided into m groups, Bilinear(·) represents the upsampling operation of bilinear interpolation, and concat(·) represents the concatenation operation. This represents an adaptive average pooling operation with an output size of s×s, where C4 is the number of channels for the feature, H is the height of the road image, W is the width of the road image, m and s are positive integers, and B is the number of samples selected in one training iteration.

[0034] Furthermore, the adaptive high-frequency filtering layer utilizes the low-frequency features and feature X of the road surface image to obtain the high-frequency features of the road surface image, specifically as follows:

[0035] AHF(X) = X*(X - ALF(X))

[0036] Here, AHF(X) is the high-frequency feature of the road surface image.

[0037] Furthermore, the weight-sharing-based decomposition attention layer first aggregates the low-frequency and high-frequency features of the road image to obtain aggregated frequency features, and then performs a weight-sharing-based decomposition attention operation on the aggregated frequency features to generate frequency domain features. Specifically:

[0038] A1. Aggregate the low-frequency and high-frequency features of the road surface image to obtain the aggregated frequency features. Use the aggregated frequency features to obtain the decomposed attention, specifically:

[0039]

[0040]

[0041] X LH =concat(ALF(X),AHF(X))

[0042] Among them, X LH These are the frequency features after aggregation; Linear(·) represents the linear layer. The one with the number 0 is a 3×3 depthwise separable convolution, where Q is the query vector, K is the key vector, V is the value vector, FA(Q,K,V) is the decomposed attention, and C is the channel dimension of the query vector Q.

[0043] A2. Obtaining weight-sharing-based decomposed attention using decomposed attention, specifically:

[0044] EFA(Q,K,V)=FA(Q,K,V)*Norm(V*R s )

[0045] Norm() is the normalization operation, R s It is an external weight;

[0046] A3. Generate frequency domain features using the weight-sharing-based decompositional attention EFA(Q,K,V) obtained in A2.

[0047]

[0048] in, It is a frequency domain characteristic.

[0049] Furthermore, the linear attention operator module utilizes road surface image features at reduced resolution. Obtaining spatial domain features X s Specifically:

[0050]

[0051] Among them, K e V e ∈R M×D K e V e These are learnable weight parameters, where DN is the double normalization operation and M is the weight parameter K. e V e The dimension is D, where D is the number of channels for the weighting parameter.

[0052] Furthermore, the cross-space-frequency domain cross-attention subnetwork is used to utilize frequency domain features. and spatial domain features X s Obtaining hybrid domain features Specifically:

[0053] B1. Frequency Domain Feature Dimension Conversion Module for Frequency Domain Features Perform dimensional transformation to obtain feature X A Specifically:

[0054]

[0055] in,

[0056] B2. Obtaining the module pair spatial domain feature X from cross feature vectors s After performing normalization, pooling, convolution, splitting, and dimension transformation operations, two cross-feature vectors K are obtained. s V s Specifically:

[0057] K s V s =σ(θ(W) 1×1 ·Pooling(Norm(X s ))))

[0058] Where σ(·) is the second-dimensional transformation operation, θ(·) is the matrix splitting operation, and W 1×1 It's a 1×1 convolution, Pooling() is the pooling operation, and Norm() is the normalization operation. C5 is the number of feature channels;

[0059] B3. The hybrid domain feature acquisition module utilizes the cross feature vector K. s V s and feature X A To obtain hybrid domain features Specifically:

[0060]

[0061]

[0062] Where, d f Representing feature X A The channel dimension, X F It is an interactive feature.

[0063] Furthermore, the parallel gated feedforward subnetwork utilizes hybrid domain features. Obtaining deep features And utilize deep features Generate a segmented image, specifically:

[0064] C1. The mixed-domain feature grouping module groups the mixed-domain features. Split into feature X along the channel dimension F1 ,X F2 Specifically:

[0065]

[0066] Split(·) is an operation that splits a feature into two parallel branches along the channel dimension.

[0067] C2. The branch feature acquisition module utilizes feature X. F1 ,X F2 Generate branch features Y1 and Y2 respectively, specifically as follows:

[0068]

[0069]

[0070] in, Represents a 1×1 pixel convolution. This represents a 3×3 depthwise separable convolution. GeLu activation function, BN is BatchNorm normalization function, and · represents element-wise multiplication;

[0071] C3. The feature concatenation module aggregates Y1 and Y2 to generate deep features. Specifically:

[0072]

[0073] Among them, EG(X) F ) is a branch splicing feature;

[0074] C4. The image segmentation output module utilizes deep features. Generate a segmented image.

[0075] Furthermore, the branch splicing feature EG(X) F Specifically:

[0076] EG(X F = concat(Y1,Y2)

[0077] Here, concat(·) is the concatenation operation.

[0078] The segmented image output module is a segmentation head, which includes a 3×3 convolutional layer and a 1×1 convolutional layer.

[0079] The beneficial effects of this invention are as follows:

[0080] This invention proposes a weight-sharing-based decompositional attention method to select important frequency features. By constructing a dynamic frequency capture kernel, it acquires high-level semantic information to enhance inter-category differences, thereby improving the segmentation accuracy of overlapping or incomplete objects in road scenes. Furthermore, this invention employs a cross-attention method combining spatial and frequency domain features to further extract pixel detail information. Through the interaction of spatial and frequency domain features, this invention fully utilizes high-resolution pixel information in the spatial domain and global contextual information in the frequency domain, solving the problem of lost image boundary information. Finally, this invention proposes a parallel gated feedforward sub-network segmentation method to encode pixel information at different locations. By learning the local structure of the road image, it enhances feature representation and improves the segmentation accuracy of targets in complex road scenes. Attached Figure Description

[0081] Figure 1 The flowchart shows the processing flow of the frequency-aware semantic segmentation method.

[0082] Figure 2 The flowchart shows the process of the weight-sharing-based decomposition attention importance frequency feature extraction method.

[0083] Figure 3 Flowchart of the cross-attention method for spatial and frequency domain features;

[0084] Figure 4 Here is a structural diagram of the linear attention operator module;

[0085] Figure 5 Flowchart of the parallel gated feedforward network segmentation method;

[0086] Figure 6 The graph shows the variation of MIoU of the segmentation method of the present invention under different groupings;

[0087] Figure 7 The graph shows the variation of MIoU of the segmentation method of the present invention under different sizes of cross feature space;

[0088] Figure 8 The graph shows the variation of MIoU of the segmentation method of the present invention under different values ​​of groups;

[0089] Figure 9(a) is a bar chart showing the changes in Recall during the ablation experiment;

[0090] Figure 9(b) is a bar chart showing the changes in Precision during the ablation experiment;

[0091] Figure 10 The graph shows the MIoU variation curves of various segmentation methods under different iteration numbers on the Cityscapes dataset;

[0092] Figure 11The graph shows the MIoU variation curves of various segmentation methods under different iteration numbers on the COCO-Stuff dataset.

[0093] Figure 12 A comparison chart of visualization results for a simple road scene;

[0094] Figure 13 This is a comparison chart of visualization results for complex road scenarios. Detailed Implementation

[0095] Recently, researchers have discovered that frequency domain features can better represent the semantic information of images. The WDSBLN network analyzes the low-frequency and high-frequency components of SAR images to preserve their deep features and achieve better image classification results. Rao et al. proposed a global filter network to acquire frequency domain features for more efficient image classification. Dong et al. utilized the frequency sensitivity of semantic segmentation to propose an adaptive frequency filter to learn feature representations from a frequency perspective. Li et al. proposed a novel frequency-aware discriminative feature learning framework to mine the frequency information of images. All of these methods establish long-range pixel dependencies from the frequency perspective of images; therefore, researching how to capture important frequency features has significant application value for semantic segmentation.

[0096] To facilitate information interaction between different branches or modules, researchers have proposed cross-attention. For example, Wang et al. proposed a cross-resolution cross-attention approach to fully realize the interaction of semantic information between low-resolution and high-resolution branches. Chen et al. designed an efficient token fusion strategy based on cross-attention to extract multi-scale feature representations. Wei used cross-attention to unify the correlation features within and between modalities. Zhu et al. proposed dual cross-attention to learn subtle features and identify fine-grained targets. Lin et al. applied attention alternately between internal patches to maintain good performance with low computational cost, while constructing a hierarchical network of a cross-attention Transformer (CAT). However, all of these methods only use cross-attention in the spatial domain, resulting in a limited receptive field for features and an inability to extract effective feature information. Therefore, researching how to use cross-attention across different domains is of significant importance for semantic segmentation.

[0097] As a core component of Transformer networks, feedforward networks suffer from poor generalization performance in segmentation methods due to their simple structure, which fails to generate high-quality features. Zamir et al. proposed a gated-depth convolutional feedforward network, introducing a gating mechanism for feature transformation. Xie et al. introduced 3x3 depth convolutions into feedforward networks to pass positional information, thus enhancing the output features to some extent. Dauphin et al. used a simplified gating mechanism in their feedforward networks to model the local contextual relationships of features. All of these methods use depth convolutions as the sole gating mechanism in the feedforward network, which cannot learn complex features resulting from interactions between different domains. Therefore, researching how to learn and enhance complex feature representations in feedforward networks is of significant research value.

[0098] Existing road segmentation methods fail to capture complete contextual semantic information, leading to decreased segmentation performance for overlapping objects in road scenes. Furthermore, they only focus on spatial domain features, neglecting the combination of spatial and frequency domain features, resulting in the loss of significant edge detail. Moreover, existing segmentation methods cannot encode the positional information of different pixels, causing a decline in target segmentation performance in complex road scenes. To address these issues, this invention proposes a frequency-aware semantic segmentation method, which will be described below with specific implementation details.

[0099] Specific implementation method one: as follows Figure 1 As shown, the specific process of a road segmentation method for autonomous driving in this embodiment is as follows:

[0100] The road surface image to be segmented is obtained and input into the trained road surface segmentation network to obtain the road surface image segmentation map.

[0101] The road segmentation network includes: a feature extraction subnetwork, a cross-space-frequency domain cross-attention subnetwork, and a parallel gated feedforward subnetwork;

[0102] The feature extraction subnetwork includes: a feature resolution reduction module, a dynamic frequency capture kernel, and a linear attention operator module;

[0103] The feature resolution reduction module includes: one convolutional layer and three consecutive residual layers; the feature resolution reduction module is used to reduce the resolution of road image features to a preset resolution value, and to convert the reduced resolution road image features... The image is sent to the dynamic capture kernel and the linear attention operator layer; the present invention sets the resolution to 1 / 16 of the original resolution of the road surface image;

[0104] The dynamic capture kernel is used to capture the frequency features of the road surface image and dynamically selects important frequency features based on weight-sharing decomposition attention; the dynamic capture kernel includes: an adaptive low-frequency filtering layer, an adaptive high-frequency filtering layer, and a weight-sharing decomposition attention layer;

[0105] The adaptive low-frequency filtering layer employs an adaptive average pooling layer; the adaptive low-frequency filtering layer is used to filter road surface image features with reduced resolution. Dimensionality reduction is performed to obtain feature X. An adaptive average pooling operation is applied to feature X to obtain low-frequency features of the road surface image. The low-frequency features of the road surface image are then sent to an adaptive high-frequency filtering layer and a weight-sharing-based decomposition attention layer.

[0106] The adaptive high-frequency filtering layer uses the low-frequency features and feature X of the road surface image to obtain the high-frequency features of the road surface image, and sends the high-frequency features of the road surface image to the decomposition attention layer based on weight sharing.

[0107] The weight-sharing-based decomposition attention layer first uses a feature aggregation method to aggregate low-frequency and high-frequency features of the road image to obtain aggregated frequency features. Then, it applies a weight-sharing-based decomposition attention operation to the aggregated frequency features to generate frequency domain features. and frequency domain features Send to the cross-space-frequency domain cross-attention subnetwork;

[0108] The linear attention operator module utilizes road surface image features at reduced resolution. Obtaining spatial domain features X s And the spatial domain feature X s Send to the cross-space-frequency domain cross-attention subnetwork;

[0109] The cross-space-frequency domain cross-attention subnetwork utilizes frequency domain features and spatial domain features X s Obtaining hybrid domain features and hybrid domain features The data is sent to the parallel gated feedforward subnetwork; the cross-space-frequency domain cross-attention subnetwork includes: a frequency domain feature dimension transformation module, a cross feature vector acquisition module, and a hybrid domain feature acquisition module;

[0110] The frequency domain feature dimension conversion module is used to convert frequency domain features. Perform dimensional transformation to obtain feature X A and feature X A Send to the hybrid domain feature acquisition module;

[0111] The cross feature vector acquisition module utilizes spatial domain feature X sAfter performing a series of matrix operations including normalization, pooling, convolution, splitting, and dimensionality transformation, two cross-feature vectors K are obtained. s V s and K s V s Send to the hybrid domain feature acquisition module;

[0112] The hybrid feature acquisition module utilizes cross-feature vector K based on cross-attention. s V s and feature X A To obtain hybrid domain features and hybrid domain features Send to the parallel gated feedforward subnetwork.

[0113] The parallel gated feedforward subnetwork utilizes hybrid domain features Obtaining deep features And utilize deep features Generate a segmented image; the parallel gated feedforward sub-network includes: a hybrid domain feature grouping module, a branch feature acquisition module, a feature stitching module, and a segmented image output module;

[0114] The hybrid domain feature grouping module applies a parallel mechanism to group hybrid domain features. Split into feature X along the channel dimension F1 ,X F2 and feature X F1 ,X F2 Send branch feature acquisition module;

[0115] The branch feature acquisition module is used to process feature X. F1 ,X F2 The gating mechanism is used to generate branch features Y1 and Y2 respectively, and the branch features are sent to the feature splicing module.

[0116] The feature concatenation module is used to aggregate Y1 and Y2 to generate deep features. and deep features Send to the segmented image output module;

[0117] The segmented image output module is a segmentation head, comprising: a 3×3 convolutional layer and a 1×1 convolutional layer; the segmented image output module utilizes deep features. Generate a segmented image.

[0118] like Figure 1 As shown, the image processing flow in this implementation is as follows: First, an image is input, and a convolutional layer and three consecutive residual layers are applied to obtain the features after reducing the resolution. Will Frequency domain features of the image generated by the dynamic frequency capture kernel Next Input to the linear attention operator module Transformed into spatial domain features Then the frequency domain features and Input a cross-spatial-frequency domain cross-attention subnetwork to generate hybrid domain features resulting from interactions between different domains. at last The input is fed into a parallel gated feedforward sub-network module to generate deep features. right The segmented image is then output using a segmentation head. Based on the above, this invention studies three parts: a weight-sharing-based decompositional attention important frequency feature extraction method, a cross-attention method combining spatial and frequency domain features, and a parallel gated feedforward network segmentation method. The weight-sharing-based decompositional attention important frequency feature extraction method improves the segmentation accuracy of overlapping targets by capturing high-level semantic information. The cross-attention method combining spatial and frequency domain features solves the problem of image boundary information loss by extracting pixel detail information. The parallel gated feedforward network segmentation method improves the segmentation performance of targets in complex road scenes by encoding the positional information of different pixels.

[0119] This invention proposes a weight-sharing-based decompositional attention method for important frequency features extraction. It directly captures image frequency features in the spatial domain by constructing a dynamic frequency capture kernel, and dynamically selects important frequency features using weight-sharing-based decompositional attention. The core components of the dynamic frequency capture kernel are as follows: Figure 2 As shown, it includes adaptive low-frequency filtering, adaptive high-frequency filtering, and weight-sharing-based decomposition attention. First, an image is input, and a convolutional layer and three consecutive residual layers are applied to obtain the reduced-resolution features. After dimensionality reduction to generate features X, adaptive low-frequency filtering is applied to X to obtain low-frequency features of the image. Next, the low-frequency features and the low-frequency features are projected onto the low-dimensional features and then subjected to adaptive high-frequency filtering to obtain high-frequency features of the image. Finally, feature aggregation is applied to the low-frequency and high-frequency features to generate frequency features, which are then subjected to weight-sharing-based decomposition attention to generate important frequency domain features.

[0120] Specific Implementation Method Two: The adaptive low-frequency filtering layer is used to filter road surface image features with reduced resolution. Dimensionality reduction is performed to obtain feature X. Adaptive average pooling is then applied to feature X to obtain low-frequency features of the road surface image. Specifically:

[0121]

[0122]

[0123] in, It represents the road surface image features with reduced resolution. `reshape(·)` represents the first-dimensional transformation operation. X represents the reduced-resolution road surface image features after dimensionality reduction, and ALF(X) represents the low-frequency features of the road surface image. m This means dividing feature X into m groups. This represents an adaptive average pooling operation with an output size of s×s, concat(·) represents a concatenation operation, Bilinear(·) represents a bilinear interpolation upsampling operation, H is the height of the road image, W is the width of the road image, C4 is the number of channels of feature X, m and s are positive integers, and B is the number of samples selected in one training session (the size of the model batch size); in this invention, m is set to 4, and s is set to 1, 2, 3, 6;

[0124] Low-frequency features contain most of the high-level contextual semantic information of an image. This invention employs average pooling as an adaptive low-frequency filter to dynamically capture low-frequency features. Since different images have different cutoff frequencies, "adaptive" means setting different groups of pooling operations based on kernel size and stride to capture low-frequency features suitable for different images.

[0125] Specific Implementation Method 3: The adaptive high-frequency filtering layer obtains the high-frequency features of the road surface image using the low-frequency features and feature X of the road surface image, specifically as follows:

[0126] High-frequency features of images are crucial for preserving image details in semantic segmentation. To reduce computational complexity, this invention directly utilizes low-frequency features to dynamically capture high-frequency features from different images. The formula for adaptive high-frequency filtering is shown below:

[0127] AHF(X)=X*(X-ALF(X)) (3)

[0128] This invention uses this method to suppress segmentation noise within the target.

[0129] Specific Implementation Method Four: The weight-sharing-based decomposition attention layer first uses a feature aggregation method to aggregate the low-frequency and high-frequency features of the road image to obtain aggregated frequency features. Then, it applies a weight-sharing-based decomposition attention operation to the aggregated frequency features to generate frequency domain features. Specifically:

[0130] For high-frequency and low-frequency features, the purpose of this invention is to select important frequency features that help capture global contextual semantic information. Therefore, this invention proposes a weight-sharing-based decomposition attention mechanism, by designing an external, learnable, and shared weight space R.s It implicitly considers the correlation between all image frequency features, thereby selecting important frequency features that are helpful for semantic segmentation from different frequency features.

[0131] A1. Aggregate the low-frequency and high-frequency features of the road surface image to obtain the aggregated frequency features, and use the aggregated frequency features to obtain the decomposed attention:

[0132] Decomposing attention uses the identity function and the Softmax() function to approximate the Softmax attention map in the self-attention mechanism. Decomposing attention first calculates the matrix multiplication of the key vector K and the value vector V; then, it activates the matrix product using the Softmax() function; finally, it calculates the matrix multiplication of the query vector Q with the result of the previous step, specifically:

[0133] X LH =concat(ALF(X),AHF(X)) (4)

[0134]

[0135]

[0136] Among them, X LH These are the aggregated frequency features. `concat(·)` represents the concatenation operation, and `Linear(·)` represents a learnable linear layer. This represents a 3×3 depthwise separable convolution with the number 0, where C is the channel dimension of the query vector Q, Q is the query vector, K is the key vector, V is the value vector, and FA(Q,K,V) is the decomposed attention.

[0137] This step of decomposing attention through factorization greatly reduces computational complexity.

[0138] A2. Obtaining weight-sharing-based decomposed attention using decomposed attention:

[0139] The weight-sharing-based decompositional attention mechanism introduces an external weight R based on decompositional attention. s External weights R s First, a matrix multiplication is performed with the value vector V to generate an attention map. Then, the attention map is multiplied with the feature FA(Q,K,V) generated by the decomposed attention to obtain the important frequency features. The formula for the decomposed attention EFA(Q,K,V) based on weight sharing is:

[0140] EFA(Q,K,V)=FA(Q,K,V)*Norm(V*R s (7)

[0141] Norm() is the normalization operation, R sThese are external weights; the external weights are randomly initialized and tend towards their optimal values ​​during the training process.

[0142] A3. Generate frequency domain features using the weight-sharing-based decompositional attention EFA(Q,K,V) obtained in A2.

[0143]

[0144] in, It is a frequency domain feature, and reshape(·) represents the first dimension transformation operation;

[0145] This invention proposes a weight-sharing-based decompositional attention method for important frequency feature extraction. In the dynamic frequency capture kernel, adaptive low-frequency filtering and adaptive high-frequency filtering are used to directly and dynamically capture high-frequency and low-frequency features of the image in the spatial domain. Furthermore, weight-sharing-based decompositional attention is employed to select important frequency features to capture high-level semantic information and enhance inter-category differences, thereby improving the segmentation accuracy of overlapping targets.

[0146] Specific implementation method five: The linear attention operator module utilizes road surface image features at reduced resolution. Obtaining spatial domain features X s Specifically:

[0147] Spatial domain features Generated by the linear attention operator module, such as Figure 4 As shown, the input to this module is the feature vector after resolution reduction. The output is the spatial domain feature X. s The formula is as follows:

[0148]

[0149] Among them, K e V e ∈R M×D These are learnable weight parameters, where M is the weight parameter K. e V e The dimension is D, where D is the number of channels for the weight parameters, and DN is the double normalization operation.

[0150] In order to reduce the computational complexity of the model, this implementation eliminates the multi-head mechanism of external attention in the prior art, thereby improving the computational speed of the model.

[0151] Specific Implementation Method Six: The data processing procedure of the cross-space-frequency domain cross-attention sub-network is as follows:

[0152] B1. Frequency domain characteristics Perform dimensional transformation to obtain features

[0153]

[0154] B2. To enable better interaction between spatial domain features and frequency domain features across various dimensions, spatial domain feature X is utilized. s After performing a series of matrix operations including normalization, pooling, convolution, splitting, and second-dimensional transformation, two cross-feature vectors are obtained.

[0155] K s V s =σ(θ(W) 1×1 ·Pooling(Norm(X s (11)

[0156] Where σ(·) and θ(·) represent the second-dimensional transformation and matrix splitting operations, respectively, and W 1×1 C represents a 1×1 convolution, Pooling() represents pooling operation, Norm() represents normalization operation, and C5 is the current number of feature channels.

[0157] B3. Based on cross-attention and utilizing cross-feature vector K s V s and feature X A To obtain hybrid domain features

[0158] Since simple feature aggregation operations cannot achieve the interaction of features between different domains, this invention uses cross features X from the spatial domain. s V s With frequency domain feature X A Apply cross-attention to generate interactive features Then for X F Perform inverse dimensional transformation to obtain hybrid domain features The formula is shown below:

[0159]

[0160]

[0161] Where, d f X represents A The channel dimension; when the spatial size of the cross feature is 12×12, the segmentation performance of the present invention is the best.

[0162] Existing road surface segmentation methods extract features only from the spatial domain of the image, ignoring the interaction between spatial and frequency domain features, thus leading to the loss of image edge detail information. Therefore, this invention proposes a cross-attention method for spatial and frequency domain features. By constructing a cross-space-frequency domain cross-attention module, the interaction between spatial and frequency domain features is achieved, thereby obtaining pixel detail information. The framework of the cross-space-frequency domain cross-attention module is as follows: Figure 3 As shown, the input is the spatial domain feature X. s and frequency domain characteristics Applying cross-attention mechanism to generate mixed features from different domains The generation of frequency domain features has already been introduced in the previous section. Next, we will introduce the process of generating spatial domain features by the linear attention operator module and the process of realizing the interaction of features from different domains by the cross-space-frequency domain attention module.

[0163] Specific Implementation Method Seven: The specific processing procedure of the parallel gated feedforward sub-network is as follows:

[0164] C1. Parallel mechanism refers to grouping and computing the feature information stream from the previous level in parallel. On the one hand, the parallel mechanism maintains the advantages of the multi-head mechanism in Transformer networks to some extent; on the other hand, it captures the relationships between pixels at different positions by generating two different paths. Given an input mixed domain feature... Applying parallel mechanisms to mix domain features Split into feature X along the channel dimension F1 ,X f2 Specifically:

[0165]

[0166] Where Split(·) represents splitting the feature along the channel dimension into two parallel branch features, X F1 ,X F2 The characteristics acquired after the split;

[0167] C2. To learn the local structure of an image, this invention designs the gating mechanism as an element-wise product of two parallel branches. First, the two parallel branches encode different pixel positions using the BatchNorm normalization function and depthwise convolution, respectively. Second, the GELU nonlinear function is used to activate the encoded features in the two parallel branches, respectively. Finally, the two parallel branches perform an interactive element-wise product operation on feature X. F1 ,X F2 The gating mechanism is used to generate branch features Y1 and Y2 respectively, as shown in the following equation:

[0168]

[0169]

[0170] in, Represents a 1×1 pixel convolution. This represents a 3×3 depthwise separable convolution. is the GeLu activation function, BN is the BatchNorm normalization function, and · represents element-wise multiplication. For two 3×3 depthwise separable convolutions with different labels, These are two 1×1 pixel convolutions with different labels.

[0171] C3. Concatenate the feature maps of the two parallel branches to output the deep features. As shown in the formula:

[0172] EG(X F = concat(Y1,Y2) (17)

[0173]

[0174] Here, concat(·) is the concatenation operation. It is a 1×1 pixel convolution with label 0.

[0175] C4, will Input the segmentation head to obtain the segmented road surface image.

[0176] The segmentation head includes: a 3×3 convolutional layer and a 1×1 convolutional layer;

[0177] In this embodiment, existing road segmentation methods achieve good target segmentation results in simple scenarios; however, due to poor model generalization performance, the segmentation effect deteriorates in complex road surfaces. Feedforward networks, as a core component of the Transformer, typically consist of two fully connected layers and a nonlinear activation function. However, this network structure can only perform the same processing on each pixel location in the features. Since local information differs at different locations, this structure cannot capture local location information, resulting in poor model generalization ability. Therefore, this invention proposes a parallel gated feedforward network segmentation method. By constructing a parallel gated feedforward network, it improves the feature information flow in the feedforward network from two aspects: parallel mechanism and gating mechanism based on the GeLu activation function. The proposed parallel gated feedforward sub-network architecture is as follows: Figure 5 As shown, the input mixture domain features are first... The feature X is divided into two groups using a parallel mechanism. F1 ,X F2 Then X F1 ,X F2As two parallel branches, gating mechanisms are applied simultaneously to generate features Y1 and Y2 respectively. Finally, Y1 and Y2 are aggregated to generate deep features. The parallel gated feedforward network segmentation method proposed in this invention learns the local structure of the image by encoding the position information of different pixels, thereby enhancing the feature representation and improving the segmentation performance of targets in complex road scenes.

[0178] Example:

[0179] To verify the beneficial effects of this invention, the effectiveness of the proposed model was validated on the Cityscapes and COCO-stuff datasets. This invention mainly focuses on three parts: parameter analysis of each module, ablation experiments, and comparative experiments.

[0180] 1.1 Experimental Setup

[0181] This invention uses the Paddle 1.8.0 framework for experiments and consistently employs the AdamW optimizer. In the Cityscapes dataset experiments, the initial learning rate was set to 0.0004, and the weight decay was set to 0.0125; on COCO-stuff, the learning rate was set to 0.0001, and the weight decay was set to 0.05. Furthermore, the semantic segmentation model proposed in this invention is not pre-trained. This embodiment uses MIoU, Precision, Recall, FPS, and Params to evaluate segmentation performance. MIoU is the average intersection-union ratio. Precision is the probability of correctly predicting a given class. Recall is the probability of correctly predicting a class from the ground truth. The formulas are defined as follows:

[0182]

[0183] Precision = TP / (TP + FP) (20)

[0184] Recall = TP / (TP + FN) (21)

[0185] In formula (19), the relationship between classes is defined as P, P ii p is used to represent the probability of a pixel being a true positive. ji The value t is used to represent the probability of a false positive for a pixel, where k represents the number of pixel categories. i The total number of pixels of category i is represented by , where i and j are pixel labels. The larger the MIoU value, the better the segmentation effect of the model. In formula (20), TP is used to represent a pixel as a true positive and FP is used to represent a pixel as a false positive. In formula (21), FN is used to represent a pixel as a false negative.

[0186] In addition, FPS represents the number of images inferred per second, and Params represents the number of model parameters.

[0187] 1.2 Analysis of Experimental Parameters

[0188] This invention proposes a weight-sharing-based decomposition attention important frequency feature extraction method to improve the accuracy of overlapping target segmentation; then, it employs a cross-attention method combining spatial and frequency domain features to address the problem of boundary information loss; finally, it proposes a parallel gated feedforward network segmentation method to improve the model's generalization ability. This embodiment will analyze the parameters of each of the above three modules.

[0189] 1.2.1 Parameter Analysis of the Number of Groups in the Weight-Sharing-Based Decomposition Attention Importance Frequency Feature Extraction Method

[0190] To investigate the impact of the number of groups M in the adaptive low-frequency filtering method based on weight sharing on semantic segmentation performance, this embodiment uses the number of groups M as a parameter for experimental analysis. The value of M is set to 2-9 respectively. Figure 6 This shows the changes in MIoU values ​​under different groupings. Figure 6 The experimental results show that when the number of groups in the adaptive low-frequency filter is 4, the MIoU value is the highest, reaching 73.38%, which is 2.25% higher than the lowest value. Figure 6 The trend shows that when the number of groups M is small, the model's MIoU value is low; then, as the value of M increases, the MIoU value also increases; when M is 4, it reaches a peak of 73.38%; afterwards, as the value of M increases, the model's MIoU value gradually decreases. Within a certain range, the larger the value of M, the more low-frequency features from different frequency bands are captured, so in the initial stage, the MIoU value increases with the increase of M; when it exceeds a certain range, the captured low-frequency features contain more segmentation noise, which affects the performance of semantic segmentation, thus causing the MIoU value to decrease later. The above experimental analysis proves that the extraction of frequency features is substantially helpful in improving the model's performance.

[0191] 1.2.2 Parameter Analysis of the Size of the Cross-Feature Space in the Spatial Domain and Frequency Domain Cross-Attention Method

[0192] To investigate the impact of different cross-feature space sizes on road segmentation under the cross-attention method combining spatial and frequency domain features, this embodiment sets the cross-feature space size S as an experimental parameter, in pixels, and sets eight experimental groups with S = 36, 64, 100, 144, 196, 256, 324, and 400 respectively. Here, S = 36 represents a cross-feature space with a length and width of 6×6, S = 64 represents a cross-feature space with a length and width of 8×8, and so on. The experimental results are as follows... Figure 7 As shown. By Figure 7 It can be seen that when the cross feature space size is 144 (12×12), the MIoU value reaches its highest level of 73.38%. From... Figure 7 The trend shows that when the cross-feature space S is small, the model's accuracy MIoU value is low; then, as S increases, the MIoU value also increases; it reaches a peak of 73.38% when S is 144; afterwards, as S increases further, the MIoU value of the segmentation method gradually decreases. When the size of the cross-feature space in the spatial domain is the same as that in the frequency domain, features from different domains can interact better, so the MIoU value of the segmentation method is the highest at this time. Through the above experimental analysis, the cross-attention method of spatial and frequency domain features can improve segmentation performance through the interaction of features from different domains.

[0193] 1.2.3 Parameter Analysis of Depthwise Separable Convolution in Parallel Gated Feedforward Network Segmentation Method

[0194] To investigate the impact of the `groups` parameter of depthwise separable convolution on the road segmentation method in the parallel gated feedforward network segmentation method, this embodiment sets the `groups` values ​​to 8, 16, 32, 64, 128, 256, 512, and 1024 for experiments. Figure 8 The overall trend shows that as the `groups` value increases, the MIoU value of the model also increases. However, the larger the `groups` value, the larger the overall model size and the heavier the computational burden. When the `groups` value exceeds a certain range, the overall computation speed of the model will decrease significantly. To balance the computation speed and accuracy of the model, this embodiment uses 1024 as the value of `groups`. When the value of `groups` is 1024, the MIoU value reaches a maximum of 73.38%, which is 1.74% higher than the lowest point. This also proves that using depthwise separable convolutions in the parallel gated feedforward network segmentation method has a certain impact on improving segmentation performance.

[0195] 1.3 Ablation Experiment

[0196] 1.3.1 Ablation Experiment Results of Each Module

[0197] To demonstrate the effectiveness of the proposed method for extracting important frequency features based on weight sharing, the cross-attention method for spatial and frequency domain features, and the parallel gated feedforward network segmentation method, this embodiment conducts experimental verification on the Cityscapes dataset.

[0198] This embodiment uses Transformer as the basic feature extraction network. The design and experiments of each module are based on the Transformer network. The experimental results of the road segmentation method proposed in this invention on the Cityscapes dataset are shown in Table 1.

[0199] Table 1. Experimental results of the road segmentation method proposed in this invention on the Cityscapes dataset.

[0200]

[0201] In Table 1, DFCM represents the Dynamic Frequency Capture Kernel module, CDCA represents the Cross-Spatial-Frequency Domain Cross-Attention module, and PGFFN represents the Parallel Gated Feedforward Network module. As shown in Table 1, the Dynamic Frequency Capture Kernel improves the accuracy of overlapping target segmentation by capturing high-level contextual semantic information, and the MIoU increases by 3.92% compared to the Transformer-based segmentation method. Then, from the data analysis in Figure 9, the design of the Cross-Spatial-Frequency Domain Cross-Attention module further improves the semantic segmentation performance by acquiring image boundary information, and the MIoU value increases by nearly 2%. Finally, although the Parallel Gated Feedforward Network module only improves the MIoU by 0.41%, it can be seen from Figure 9 that this module achieves the highest precision, meaning the segmentation result has the highest reliability, indirectly proving that improving the learning ability of local image structure has a substantial effect on improving the overall semantic segmentation performance.

[0202] 1.3.2 Advantages of the Weight-Sharing-Based Decomposition Attention Importance Frequency Feature Extraction Method

[0203] To demonstrate that the weight-sharing-based decompositional attention proposed in this invention can capture high-level contextual semantic information and effectively segment overlapping targets, this embodiment uses different types of attention in a dynamic frequency capture kernel for experimental verification. The comparison results are shown in Table 2.

[0204] Table 2 Comparison of different types of attention

[0205]

[0206] As shown in Table 2, we can see that weight-sharing-based decomposition attention outperforms self-attention in both speed and accuracy, improving the MIoU value by 4%. Furthermore, the proposed weight-sharing-based decomposition attention is far more efficient than self-attention; although the segmentation performance MIoU is only improved by 0.5%, the image processing speed is almost twice that of self-attention. These results indirectly indicate that weight-sharing-based decomposition attention can select important frequency features to improve the segmentation accuracy of overlapping targets.

[0207] 1.3.3 Advantages of the cross-attention method based on spatial and frequency domain features

[0208] To demonstrate that the cross-space-frequency domain cross attention proposed in this invention can acquire pixel detail information of an image, this embodiment uses different cross attention methods for experimental verification, and the results are shown in Table 3.

[0209] Table 3 Comparison of different types of cross-attention

[0210]

[0211] As shown in Table 3, the MIoU is improved by 2.03% when using ordinary cross-attention compared to not using cross-attention, indicating that the cross-attention mechanism is effective for the segmentation method in this embodiment. Cross-spatial-frequency domain cross-attention improves the MIoU by 3.26% compared to ordinary cross-attention, demonstrating that the cross-spatial-frequency domain cross-attention proposed in this invention can fully learn the local structure of the image, thereby improving semantic segmentation performance.

[0212] 1.3.4 Advantages of the Parallel Gated Feedforward Network Partitioning Method

[0213] To verify that the parallel gated feedforward network proposed in this invention can enhance feature representation and improve the generalization ability of the model, this embodiment studies the impact of whether the feedforward network adopts a parallel mechanism and a gating mechanism on the segmentation performance. The results are shown in Table 4.

[0214] Table 4 Comparison of different feedforward networks

[0215]

[0216] As shown in Table 4, compared to ordinary feedforward networks, the MIoU values ​​of the gated feedforward network and the parallel feedforward network were improved by 0.46% and 1.58%, respectively, with only a slight improvement in segmentation performance. The parallel gated feedforward network, however, showed a significant improvement in segmentation performance compared to the ordinary feedforward network, with an MIoU increase of 3.58%. These results indicate that the parallel gated feedforward network improves overall semantic segmentation performance by encoding information from different pixel positions.

[0217] 1.4 Comparison with other existing segmentation methods

[0218] This embodiment compares the semantic segmentation method proposed in this invention with mainstream segmentation methods, and conducts experiments on the Cityscapes dataset and the COCO-Stuff dataset, respectively.

[0219] 1.4.1 Comparison with other segmentation methods on the Cityscapes dataset

[0220] This embodiment uses the Cityscapes dataset for experiments. All semantic segmentation methods utilize the AdamW optimizer. Table 5 shows the experimental results for different segmentation methods, including Params and MIoU.

[0221] Table 5 compares other mainstream segmentation methods on the Cityscapes dataset.

[0222]

[0223]

[0224] As shown in Table 5, Transformer-based semantic segmentation methods (such as SegFormer) outperform convolution-based semantic segmentation methods (such as DeepLabV3+ and PSPNet). However, due to the quadratic computational complexity of Transformer, the computational cost of Transformer-based semantic segmentation methods remains very high. Furthermore, the semantic segmentation method proposed in this invention achieves 73.38% MIoU with only 7.8M parameters. Compared to SegFormer, FSSFormer reduces the number of parameters by 5M and improves MIoU by 1%. Moreover, compared to the lightweight semantic segmentation method RTFormer, the segmentation method of this invention improves MIoU by nearly 2% while using only half the number of parameters. To more intuitively reflect the training process of each segmentation method, Figure 10 The changes in MIoU value for each segmentation method as the number of iterations increases are shown.

[0225] like Figure 10 As shown, the overall curve of the segmentation method of this invention is smoother than other segmentation methods, indicating that the method is more stable throughout the training process. When iter is 80000, this invention achieves a high MIoU value, completing training faster than other segmentation methods. Moreover, compared to semantic segmentation methods such as RTFormer and SegFormer, the curve of this invention is flatter in the later stages of training, representing more stable segmentation performance. In summary, this invention outperforms other semantic segmentation methods during training.

[0226] 1.4.2 Comparison with other segmentation methods on the COCO-Stuff dataset

[0227] This embodiment conducts experiments on the COCO-Stuff dataset. Table 6 shows the experimental results for different segmentation methods.

[0228] Table 6 compares the results with other segmentation methods on the COCO-Stuff dataset.

[0229]

[0230] The COCO-stuff dataset contains a large number of difficult-to-process samples from the COCO dataset. As shown in Table 6, compared to lightweight segmentation methods (such as RTFormer), this invention, while achieving a lower MIoU, reduces the number of parameters by 10M. Furthermore, compared to semantic segmentation methods based on Transformer networks (such as SegFormer), this invention achieves 33.8% MIoU with only 6.8M params, reducing the number of parameters by a factor of 10. In summary, this invention achieves the best trade-off between computational speed and accuracy.

[0231] like Figure 11 As shown, the overall trend of the segmentation method of this invention is a steady increase in the early training stage, which then tends to level off in the later stage. Although some semantic segmentation methods (such as SegFormer) have higher MIoU than the method of this invention, the variation curves, especially in the later training stage, show a large range of MIoU changes, indicating that they are not stable throughout the training process. From the overall training results, the performance of this invention in terms of training stability and MIoU is generally better than other semantic segmentation methods.

[0232] 1.5 Visualization results of image segmentation using the present invention and existing segmentation methods

[0233] Visualization results comparison of simple road scenarios are shown in the following figure. Figure 12 As shown, Figure 12 The first row, from left to right, shows the original image, the segmented image from RTFormer, and the segmented image from DeepLabV3+. The second row, from left to right, shows the segmented image from SegFormer, the image from the FCN model, and the image segmented from PSPNet. The third row, from left to right, shows the segmented image from BiseNet2, the image segmented from OCRNet, and the segmented image from the segmentation method of this invention. Figure 12 As shown, compared to other segmentation methods, this invention exhibits better segmentation performance for overlapping vehicles and incomplete road surfaces in images. Furthermore, this invention can correctly segment people at a distance in the image. However, most other segmentation methods fail to segment small objects at a distance in the image. For example, in the visualization results of PSPNet, the category of people is mistakenly identified as the category of cars. Moreover, compared to the segmented images from BiseNet2, this invention does not show segmentation noise points at the edges of the image. This indicates that the important frequency features extracted by this invention contain contextual information that is helpful for semantic segmentation.

[0234] Visualization results of complex road scenes, such as Figure 13 As shown, Figure 13The first row, from left to right, shows the original image, the segmented image from RTFormer, and the segmented image from DeepLabV3+. The second row, from left to right, shows the segmented image from SegFormer, the image from the FCN model, and the image segmented from PSPNet. The third row, from left to right, shows the segmented image from BiseNet2, the image segmented from OCRNet, and the segmented image from the segmentation method of this invention. Figure 13 As shown, in complex scenes, compared with other segmentation methods, the segmentation edges of the method of this invention are clearer, such as the outlines of people, pedestrians, vehicles, and road surfaces. Visualization results show that the method of this invention can segment overlapping objects (such as overlapping pedestrians and vehicles) and can extract boundary information in the image (such as images without edge noise points). Furthermore, the segmentation method of this invention maintains good segmentation performance in different scenes.

[0235] To address the issue of declining road surface segmentation performance in complex road scenarios, this invention proposes a frequency-aware semantic segmentation model. First, to improve the accuracy of overlapping target segmentation, this invention uses weight-sharing-based decompositional attention to select important frequency features. Second, to solve the problem of boundary information loss, a cross-attention method combining spatial and frequency domain features is employed to further acquire pixel detail information. Finally, to improve target segmentation performance in complex scenes, a parallel gated feedforward network segmentation method is used to encode the positional information of different pixels. Extensive experiments demonstrate the feasibility of the proposed method. Furthermore, compared with other representative semantic segmentation methods currently available, the proposed method exhibits certain performance advantages and is of significant research value for improving road surface segmentation performance in complex scenarios.

Claims

1. A road segmentation method for autonomous driving, characterized in that... The specific process of the method is as follows: The road surface image to be segmented is obtained and input into the trained road surface segmentation network to obtain the segmented road surface image. The road segmentation network includes: a feature extraction subnetwork, a cross-space-frequency domain cross-attention subnetwork, and a parallel gated feedforward subnetwork; The feature extraction subnetwork includes: a feature resolution reduction module, a dynamic frequency capture kernel, and a linear attention operator module; The feature resolution reduction module is used to reduce the resolution of road surface image features to a preset resolution value, and to convert the reduced resolution road surface image features... Send to the dynamic capture kernel and linear attention operator modules; The dynamic frequency acquisition kernel includes: an adaptive low-frequency filtering layer, an adaptive high-frequency filtering layer, and a weight-sharing-based decomposition attention layer; The adaptive low-frequency filtering layer is used to filter road surface image features with reduced resolution. Dimensionality reduction is performed to obtain feature X. Low-frequency features of the road surface image are obtained using feature X, and the low-frequency features of the road surface image are sent to the adaptive high-frequency filtering layer and the weight-sharing-based decomposition attention layer. The adaptive high-frequency filtering layer uses the low-frequency features and feature X of the road surface image to obtain the high-frequency features of the road surface image, and sends the high-frequency features of the road surface image to the decomposition attention layer based on weight sharing. The weight-sharing-based decomposition attention layer first aggregates the low-frequency and high-frequency features of the road image to obtain aggregated frequency features, and then performs a weight-sharing-based decomposition attention operation on the aggregated frequency features to generate frequency domain features. and frequency domain features Send to the cross-space-frequency domain cross-attention subnetwork; The linear attention operator module utilizes road surface image features at reduced resolution. Obtaining spatial domain features X s And the spatial domain feature X s Send to the cross-space-frequency domain cross-attention subnetwork; The cross-space-frequency domain cross-attention subnetwork utilizes frequency domain features and spatial domain features X s Obtaining hybrid domain features and hybrid domain features The data is sent to the parallel gated feedforward subnetwork; the cross-space-frequency domain cross-attention subnetwork includes: a frequency domain feature dimension transformation module, a cross feature vector acquisition module, and a hybrid domain feature acquisition module; The frequency domain feature dimension conversion module is used to convert frequency domain features. Perform dimensional transformation to obtain feature X A and feature X A Send to the hybrid domain feature acquisition module; The cross feature vector acquisition module is used to obtain spatial domain features X. s Perform normalization, pooling, convolution, splitting, and dimension transformation operations to obtain two cross-feature vectors K. s V s and K s V s Send to the hybrid domain feature acquisition module; The hybrid domain feature acquisition module utilizes the cross feature vector K s V s and feature X A To obtain hybrid domain features and hybrid domain features Send to the parallel gated feedforward subnetwork; The parallel gated feedforward subnetwork utilizes hybrid domain features Obtaining deep features And utilize deep features A segmented image is generated; the parallel gated feedforward subnetwork includes: a hybrid domain feature grouping module, a branch feature acquisition module, a feature stitching module, and a segmented image output module; The hybrid domain feature grouping module is used to group hybrid domain features Split into feature X along the channel dimension F1 ,X F2 and feature X F1 ,X F2 Send branch feature acquisition module; The branch feature acquisition module utilizes feature X F1 ,X F2 Generate branch features Y1 and Y2 respectively, and send the branch features to the feature concatenation module; The feature concatenation module is used to aggregate Y1 and Y2 to generate deep features. and deep features Send to the segmented image output module; The image segmentation output module utilizes deep features Generate a segmented image.

2. The road segmentation method for autonomous driving according to claim 1, characterized in that: The feature resolution reduction module includes: a convolutional layer and three consecutive residual layers.

3. The road segmentation method for autonomous driving according to claim 2, characterized in that: The adaptive low-frequency filtering layer is used to filter road surface image features with reduced resolution. Dimensionality reduction is performed to obtain feature X. Then, feature X is used to obtain low-frequency features of the road surface image, specifically: in, reshape(·) represents the first dimension transformation operation. X represents the reduced-resolution road surface image features after dimensionality reduction, and ALF(X) represents the low-frequency features of the road surface image. m This indicates that the feature X is divided into m groups, Bilinear(·) represents the upsampling operation of bilinear interpolation, and concat(·) represents the concatenation operation. This represents an adaptive average pooling operation with an output size of s×s, where C4 is the number of channels for the feature, H is the height of the road image, W is the width of the road image, m and s are positive integers, and B is the number of samples selected in one training iteration.

4. The road segmentation method for autonomous driving according to claim 3, characterized in that: The adaptive high-frequency filtering layer obtains the high-frequency features of the road surface image using the low-frequency features and feature X of the road surface image, specifically as follows: AHF(X) = X*(X - ALF(X)) Here, AHF(X) is the high-frequency feature of the road surface image.

5. A road segmentation method for autonomous driving according to claim 4, characterized in that: The weight-sharing-based decomposition attention layer first aggregates the low-frequency and high-frequency features of the road image to obtain aggregated frequency features, and then performs a weight-sharing-based decomposition attention operation on the aggregated frequency features to generate frequency domain features. Specifically: A1. Aggregate the low-frequency and high-frequency features of the road surface image to obtain the aggregated frequency features. Use the aggregated frequency features to obtain the decomposed attention, specifically: X LH =concat(ALF(X),AHF(X)) Among them, X LH These are the frequency features after aggregation; Linear(·) represents the linear layer. The one with the number 0 is a 3×3 depthwise separable convolution, where Q is the query vector, K is the key vector, V is the value vector, FA(Q, K, V) is the decomposed attention, and C is the channel dimension of the query vector Q. A2. Obtaining weight-sharing-based decomposed attention using decomposed attention, specifically: EFA(Q, K, V)=FA(Q, K, V)*Norm(V*R s ) Norm() is the normalization operation, R s It is an external weight; A3. Generate frequency domain features using the weight-sharing-based decompositional attention EFA(Q, K, V) obtained in A2. in, It is a frequency domain characteristic.

6. The road segmentation method for autonomous driving according to claim 5, characterized in that: The linear attention operator module utilizes road surface image features at reduced resolution. Obtaining spatial domain features X s Specifically: Among them, K e V e ∈R M×D K e V e These are learnable weight parameters, where DN is the double normalization operation and M is the weight parameter K. e V e The dimension is D, where D is the number of channels for the weighting parameter.

7. A road segmentation method for autonomous driving according to claim 6, characterized in that: The cross-space-frequency domain cross-attention subnetwork is used to utilize frequency domain features. and spatial domain features X s Obtaining hybrid domain features Specifically: B1. Frequency Domain Feature Dimension Conversion Module for Frequency Domain Features Perform dimensional transformation to obtain feature X A Specifically: in, B2. Obtaining the module pair spatial domain feature X from cross feature vectors s Perform normalization, pooling, convolution, splitting, and dimension transformation operations to obtain two cross-feature vectors K. s V s Specifically: K s ,V s =σ(θ(W 1×1 ·Pooling(Norm(X s )))) Where σ(·) is the second-dimensional transformation operation, θ(·) is the matrix splitting operation, and W 1×1 It's a 1×1 convolution, Pooling() is the pooling operation, and Norm() is the normalization operation. C5 is the number of feature channels; B3. The hybrid domain feature acquisition module utilizes the cross feature vector k s V s and feature X A To obtain hybrid domain features Specifically: Where, d f Representing feature X A The channel dimension, X F It is an interactive feature.

8. A road segmentation method for autonomous driving according to claim 7, characterized in that: The parallel gated feedforward subnetwork utilizes hybrid domain features Obtaining deep features And utilize deep features Generate a segmented image, specifically: C1. The mixed-domain feature grouping module groups the mixed-domain features. Split into feature X along the channel dimension F1 ,X F2 Specifically: Split(·) is an operation that splits a feature into two parallel branches along the channel dimension. C2. The branch feature acquisition module utilizes feature X. F1 ,X F2 Generate branch features Y1 and Y2 respectively, specifically as follows: in, Represents a 1×1 pixel convolution. This represents a 3×3 depthwise separable convolution. GeLu activation function, BN is BatchNorm normalization function, and · represents element-wise multiplication; C3. The feature concatenation module aggregates Y1 and Y2 to generate deep features. Specifically: Among them, EG(X) F ) is a branch splicing feature; C4. The image segmentation output module utilizes deep features. Generate a segmented image.

9. A road segmentation method for autonomous driving according to claim 8, characterized in that: Branch splicing feature EG(X) F Specifically: EG(X F )=concat(Y1,Y2) Here, concat(·) is the concatenation operation.

10. A road segmentation method for autonomous driving according to claim 9, characterized in that: The segmented image output module is a segmentation head, which includes a 3×3 convolutional layer and a 1×1 convolutional layer.

Images