A multi-scale dual-flow fusion real-time semantic segmentation method for road scenes

Abstract

The present application relates to a kind of multi-scale dual-flow fusion real-time semantic segmentation method for road scene, belong to semantic segmentation field.The method includes: obtaining the image dataset under the driving road scene;Data is divided and data enhancement is carried out in proportion;Dual-flow fusion network model is built;The model is trained, verified and tested using dataset, and the optimal model is selected to obtain semantic segmentation result.Image is extracted by spatial branch and semantic branch respectively, spatial branch introduces residual basic block combined by spatial separable convolution and depth separable convolution, reduces parameter quantity, semantic branch introduces cascaded hollow convolution and wavelet down-sampling feature extraction, expands feature map receptive field, and the fusion strategy adopts optimization attention weighting mode, extracts more effective spatial and channel information, improves new feature pyramid pooling structure, and strengthens model generalization.The method improves segmentation accuracy while ensuring real-time, and ensures the balance of computational complexity and accuracy.

Description

Technical Field

[0001] This invention belongs to the field of semantic segmentation and relates to a real-time semantic segmentation method for multi-scale dual-stream fusion for road scenes. Background Technology

[0002] With the rapid advancement of computer data processing capabilities, artificial intelligence has experienced unprecedented development, and computer vision, as one of its most important branches, has garnered attention across various fields. Semantic segmentation has become crucial in autonomous driving tasks, requiring real-time segmentation of semantic information within road scenes to achieve pixel-level image classification and a detailed understanding of image content. In autonomous driving, the requirements for real-time scene perception speed and accuracy are extremely high, making improving model inference speed and accuracy the ideal choice. However, existing semantic segmentation methods still face challenges. For example, complex scenes such as lighting, occlusion, blurred edges, and small target sizes can cause significant differences between the actual target object and the image, thus affecting the segmentation results. In existing semantic segmentation methods, real-time performance and accuracy are often inversely proportional. Some high-accuracy network models often come at the cost of deep network layers and a large number of parameters, resulting in excessively long inference times, while some lightweight segmentation models often suffer a significant drop in accuracy. Therefore, designing an efficient real-time semantic segmentation network that simultaneously considers model accuracy and inference speed is an extremely challenging task.

[0003] In the development of image semantic segmentation, commonly used segmentation methods can be mainly divided into two categories: traditional image semantic segmentation methods and deep learning-based image semantic segmentation methods. Image segmentation methods before the emergence of deep learning are generally referred to as traditional methods. Based on different underlying theories, they can be roughly divided into image segmentation methods based on thresholding, regions, edges, or graph theory. These methods mainly utilize simple surface features such as texture, color, and shape to segment images into multiple differential regions. Their advantage is simplicity and speed, but the segmentation results are relatively coarse and cannot annotate semantic information, making accurate segmentation of complex images difficult. In the era of deep learning, convolutional neural networks have been applied in various fields. Fully convolutional networks have proposed segmentation methods that combine convolutional neural networks and semantic segmentation, achieving end-to-end pixel-level segmentation. Subsequently, the real-time semantic segmentation network Enet was proposed, which employs an asymmetric encoder-decoder architecture. Existing mainstream network architectures are generally divided into single-branch and dual-branch types. Single-branch architectures are mainly based on encoder-decoder architectures. Due to their simple structure, single-branch architectures have limited feature extraction capabilities and cannot capture detailed spatial information. Dual-branch networks consist of spatial and semantic branches. The spatial branch, due to its high resolution and computational inefficiency, faces a trade-off between semantic information capture capability and speed. Simultaneously, the semantic branch performs continuous downsampling operations, requiring sufficient contextual information for the features; existing methods inevitably lose some semantic information during the downsampling stage. Existing simple feature fusion methods struggle to simultaneously capture both local details and global contextual information. This makes it difficult for models to balance the relationship between local features and global context when dealing with small targets or complex scenes, necessitating multi-scale contextual fusion strategies. In summary, real-time semantic segmentation can be improved from the perspectives of feature information fusion and spatial information recovery, balancing the existing problems of accuracy and speed in semantic segmentation. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a multi-scale dual-stream fusion real-time semantic segmentation method for road scenarios. It employs a dual-branch data stream to implement the encoder-decoder structure and performs efficient cross-feature fusion between the two branches. Lightweighting is achieved through depthwise separable convolution and spatially asymmetric convolution. Lossless feature extraction is achieved using wavelet transform downsampling. The structure of pyramid pooling feature extraction is improved through cascaded dilated convolution, embedding a more accurate and generalizable structure into the model. Finally, efficient real-time semantic segmentation is achieved. The network structure framework is as follows: Figure 1 As shown.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A multi-scale dual-stream fusion real-time semantic segmentation method for road scenes includes the following steps:

[0007] Step 1) Obtain the urban road scene image dataset, divide the dataset into training, validation and test sets according to the proportion, and preprocess the data to obtain the initial images;

[0008] Step 2) Input the initial image into the primary feature extraction module. This module consists of one-step downsampling convolution and channel expansion convolution, which reduces the image resolution while increasing the channel depth of the feature map, thereby obtaining the primary feature map.

[0009] Step 3) The primary feature maps are input into the multi-stage parallel high-resolution spatial branch and low-resolution semantic branch network structure for feature extraction. The feature fusion of the two branches is performed at each stage of feature extraction, and the fusion result is incorporated into the input feature information of the spatial branch in the next stage. After multi-stage feature extraction and fusion, a fused feature map of deep semantic guidance spatial details is obtained.

[0010] Step 4) The fused feature map is input into the Spatial Cascaded Multi-Scale Pyramid Pooling Module (SCPPM) for contextual feature embedding representation. The embedded representation is then input into the optimized channel attention and global attention for attention calculation, which enriches the model's emphasis on different channels and the weight control of a single pixel on global information, and obtains the attention calculation result.

[0011] Step 5) The attention calculation result is input into the upsampling module to restore the spatial resolution. During the restoration process, long connections are maintained with the shallow feature map to enhance the feature expression ability. Finally, the resolution is restored to the initial image to obtain the segmentation result.

[0012] Step 6) Use the loss function to supervise the training and validation of the urban road scene semantic segmentation model to obtain the real-time semantic segmentation model with the best validation effect. Use this model to process the test dataset to obtain the final image segmentation result.

[0013] Furthermore, step 1) above includes:

[0014] Step 1.1 Download the Cityscapes and CamVid city road scene datasets from open-source dataset websites;

[0015] Step 1.2: Each dataset is proportionally divided into a training set, a validation set, and a test set;

[0016] Step 1.3 Perform data augmentation preprocessing on the allocated dataset, including random horizontal flipping, mean subtraction, and random scaling, where the random scaling factors are set to {0.75, 1.0, 1.25, 1.5, 1.75, 2.0} respectively.

[0017] Furthermore, step 3) includes: the high-resolution spatial branch consists of three stages. The first stage is constructed as the Basic Block in ResNet, which expands the output channels and reduces the resolution. The second stage is constructed as a deep spatial module (DS Unit) combining depthwise separable convolution and spatially asymmetric separable convolution, which effectively reduces the number of parameters and computation in the deep network. The third stage increases the number of channels on the deep spatial module to continue extracting features. The semantic branch also consists of three stages. The first stage uses a dual-mapping downsampling module (DM DownSampling) that combines convolution and wavelet downsampling to extract features. The second stage uses a multi-level grouped dilated downsampling module (MEDSB) to further increase the number of channels and gather semantic information. The third stage uses the dual-mapping downsampling module to further double the number of channels in the second stage for feature extraction. Feature fusion includes the fusion of semantic information and spatial details in the second stage and the fusion of semantic information and spatial information in the third stage, finally obtaining a fused feature map.

[0018] Furthermore, the depth spatial module specifically involves: first, performing a 1×1 convolution to expand the channels; then, summing the results pixel-by-pixel through a spatially separable extraction block and a 3×3 convolution block; finally, passing the results through a depth-separable extraction block and performing a residual connection with the initial input. The specific mathematical representation is as follows:

[0019] X s =F s (Conv 1×1 (x)) (1)

[0020] F ds (x)=F d (X s +Conv 3×3 (Conv 1×1 (x)))+Conv 1×1 (x) (2)

[0021] Among them, F s F represents a spatially separable extraction block. d F represents a depth-separable extraction block. ds This represents the depth space module.

[0022] Furthermore, the spatially separable extraction block and the depth-separable extraction block are as follows: Spatially separable extraction block: 1×1 convolution halves the number of channels, followed by 3×1 and 1×3 asymmetric convolutions and 5×1 and 1×5 asymmetric convolutions respectively. The elements of the two convolution results are summed, and finally, a 1×1 convolution halves the number of channels to restore the total number of channels. Depth-separable extraction block: Grouped convolutions with a group number equal to the number of input channels, and pointwise convolutions with a 1×1 kernel. The feature map is serially input to both the grouped and pointwise convolutions to form the depth-separable extraction block. The specific mathematical representation is as follows:

[0023] F1(x)=Conv 3×1 (Conv 1×3 (Conv 1×1 (x))) (3)

[0024] F2(x)=Conv 5×1 (Conv 1×5 (Conv 1×1 (x))) (4)

[0025] F s (x)=Conv 1×1 (F1(x)+F2(x)) (5)

[0026] F d (x)=Conv 1×1 (Conv g＝c (x)) (6)

[0027] Among them, Conv g＝c This indicates grouped convolution.

[0028] Furthermore, the semantic branch dual-mapping summation module and the multi-level grouped atrioventing downsampling module are as follows: Dual-mapping summation module: 1×1 convolution expands the channel, proportionally dividing the channel into pooling convolution downsampling and Haar wavelet transform downsampling channel groups. Pooling convolution downsampling and Haar wavelet transform downsampling are performed on the two parts respectively, and the results are summed pixel-by-pixel. Finally, after 3×3 convolution, the feature map resolution is reduced to 1 / 2. The Haar wavelet transform performs one-dimensional decomposition on the rows and columns of the image. Each decomposition generates four sub-bands: LL (low frequency-low frequency, approximation), LH (low frequency-high frequency, horizontal details), HL (high frequency-low frequency, vertical details), and HH (high frequency-high frequency, diagonal details). The downsampling low-pass filtering details are as follows:

[0029]

[0030] Among them, f j,n Let represent the approximation coefficients of the j-th level wavelet transform, h(k) be the low-pass filter coefficients, and n and l represent the scaling and translation parameters, respectively. Downsampling is also implicitly performed on the LL subband after each decomposition, halving its size and using it as input for the next level of decomposition. The mathematical representation of the dual-mapping summation module is:

[0031] F cd (x)=Cat d＝1 (Conv 3×3 , s＝2 (x),Maxpool 2×2 (x)) (8)

[0032] F dd (x)=Conv 1×1 (HWD(x)+F cd (x)) (9)

[0033] Cat d＝1 Indicates channel-level concatenation operation, F cd F represents pooling convolution downsampling, HWD represents wavelet transform downsampling, and F represents... dd This represents the dual-mapped summation module. The multi-level grouped dilated downsampling module consists of three dilated convolutional layers, each using a different dilation rate to expand the receptive field. After three convolutional operations, the intermediate feature maps are connected layer by layer, and channel compression is performed using 1×1 convolutions. The compressed feature map is then residually concatenated with the input feature map.

[0034] Furthermore, in step 3), the fusion of semantic information and spatial details adopts a multi-scale fusion strategy, specifically: the feature map with a resolution of H×W is adaptively pooled and mapped into regions of equal size (1×1, 4×4, 8×8, 16×16), then channel compression and expansion are performed, and then it is reshaped to a size of H×W. The regions are then accumulated to obtain region attention, which is mathematically represented as:

[0035] F Fuse =F Context +F Spatial (10)

[0036]

[0037] Where F Context and F Spatial These represent semantic branching and spatial branching information, respectively; Gonv represents channel expansion and compression operations; and UnPool represents the average solution pool. This represents region attention; subsequently, pixel attention is calculated using standard convolution, and the pixel and region attention are superimposed using a sigmoid activation function to obtain weights at different locations in the feature map. Finally, the two branches are fused using element-wise multiplication; further mathematical representation is as follows:

[0038]

[0039]

[0040] F Out =F Context ×2α+F Spatial ×2*(1-α) (14)

[0041] in 'Add' indicates pixel attention, and 'Add' indicates accumulation operation. In step 4), the spatial cascaded pyramid pooling module also uses a multi-scale strategy, specifically: the intermediate feature map obtained by superimposing the dilated convolution and the upsampled feature map after global pooling is connected, the intermediate feature map is compressed using a 1×1 convolution and residual connection is performed with the input. This method can provide richer contextual information without adding additional convolutional layers.

[0042] Further, in step 4), the channel and global attention are specifically as follows: Squeezed Channel Attention (SCA): Global average pooling is performed on the input features, channel compression convolution is used to reduce the number of channels to obtain the activation result, the activation result is normalized using the Sigmoid activation function to obtain weight information, the original feature map is convolved and multiplied with the weight information elements to obtain the channel attention map; Aggregated Global Attention (AGA): The tensor shape is adjusted to adapt to channel attention processing, channel attention is applied and the original tensor shape is restored, feature weighting is performed through the channel attention map, and finally a spatial attention map is generated and applied for feature weighting. This module enhances cross-dimensional interaction by preserving channel and spatial information, thereby improving the performance of deep neural networks. The global attention of the feature map can be represented as:

[0043]

[0044]

[0045] Where M c and M s This represents the channel and spatial attention mechanism, where F1 is the input feature map. This indicates element-wise multiplication.

[0046] Furthermore, in step 5), the upsampling module specifically replaces the conventional bilinear interpolation upsampling with a combination of transposed convolution and bilinear interpolation. At the same time, it maintains the skip connection with low-level features during the upsampling stage. In order to make the high-level information also rich in spatial information, the shallow feature map with the same resolution and sufficient spatial location information is accumulated and summed with the feature map to be upsampled through channel matching convolution, and then transposed convolution with a stride of 2 is performed to complete the upsampling operation. This method can learn new feature representations and obtain more accurate segmentation results.

[0047] The beneficial effects of this invention are as follows:

[0048] (1) This invention proposes a dual-branch data stream network structure based on the encoder-decoder segmentation architecture. Compared with the single-branch architecture, it utilizes dual paths for information extraction and integrates feature information from detail and semantic branches. It uses rich semantic information and attention mechanisms to calculate proportional weights, guides the learning of detail branches, strengthens the information representation of features, and enhances the understanding ability of deep networks.

[0049] (2) This invention utilizes a convolution decomposition method to decompose the standard convolution into depthwise separable convolution or spatially separable convolution, significantly reducing the number of parameters and computational cost. Simultaneously, it can increase network depth while maintaining a relatively constant number of parameters, thereby enhancing model performance. By combining depthwise and spatially separable convolutions to construct information extraction blocks for detail branches, it addresses the inefficiency of inference on spatial branches in most dual-branch networks.

[0050] (3) In the semantic information capture stage, the present invention designs wavelet transform downsampling and cascaded dilated convolution methods. The former can maintain more boundary, scale and texture features, and reduce the resolution by retaining more spatial information, thereby achieving lossless downsampling. At the same time, the method is simple and efficient, and achieves a good balance between segmentation accuracy and computational efficiency. The latter expands the receptive field, enabling the network to obtain better perception of different scales and effectively learn semantic information at various levels.

[0051] (4) This invention employs a more efficient multi-scale feature fusion strategy in the dual-branch fusion and global feature pooling stages, allowing the model to adaptively learn higher-order information at different semantic levels and selectively weight and fuse information at different scales. Spatial cascaded pyramid pooling operation obtains feature information with a larger receptive field, thus performing better on tasks requiring a large receptive field.

[0052] (5) This invention introduces channel attention to better learn the dependencies between channels and suppress redundant information. At the same time, it captures global context information through global attention, better understands the overall structure and context information of the target, improves the modeling of long-distance dependencies, and thus obtains more accurate feature representations.

[0053] (6) This invention designs a more lightweight and efficient real-time semantic segmentation method based on multi-scale bi-branch. The model does not require any pre-training or subsequent data processing operations, which significantly reduces the number of parameters and the size of the model, and can achieve a good balance between segmentation accuracy and inference speed.

[0054] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0055] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0056] Figure 1A flowchart of a multi-scale dual-stream fusion real-time semantic segmentation process for road scenarios;

[0057] Figure 2 A diagram of a multi-scale dual-stream fusion real-time semantic segmentation network framework for road scenarios;

[0058] Figure 3 This is a schematic diagram of the SCPPM spatial cascaded pyramid pooling module structure. Detailed Implementation

[0059] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0060] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0061] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0062] A multi-scale dual-stream fusion real-time semantic segmentation method for road scenes includes the following steps:

[0063] Step 1) Download the Cityscapes and CamVid city road scene datasets from open-source dataset websites. Divide the datasets into training, validation, and test sets according to the specified proportions, and preprocess the data for each set. Here, we take the Cityscapes dataset as an example, obtaining an initial training image resolution of 512×1024 and 3 channels.

[0064] Step 2) Input the initial training image into the primary feature extraction module. This module consists of two convolutions, both 3×3 downsampling convolutions with a stride of 2, and both convolutions have the standard number of channels C. Reduce the image resolution to 1 / 4 and increase the feature map channel depth to C, thus obtaining the primary feature map input1.

[0065] Step 3) Input1 is input to a multi-stage parallel high-resolution spatial branch and low-resolution semantic branch network structure for feature extraction. The high-resolution spatial branch consists of three stages. The first stage constructs a basic residual block (BasicBlock) in ResNet, expands the output channels, and reduces the resolution to obtain a feature map X1. The resolution is reduced to 1 / 8, and the number of channels is 2C. The second stage constructs a deep spatial module combining depthwise separable convolution and spatially asymmetric separable convolution, effectively reducing the number of parameters and computation in deep networks. Specifically, the deep spatial module first performs a 1×1 convolution to expand the channels, then sums the results pixel by pixel through the spatially separable extraction block and a 3×3 convolution block, and finally passes through the depthwise separable extraction block to form a residual connection with the initial input. The specific mathematical representation is as follows:

[0066]

[0067] F ds (x)=F d (X s +Conv 3×3 (Conv 1×1 (x)))+Conv 1×1 (x) (2)

[0068] Among them, F s F represents a spatially separable extraction block. d F represents a depth-separable extraction block. ds This represents the depth spatial module. The spatially separable extraction block is further constructed as follows: a 1×1 convolution halves the number of channels, followed by 3×1 and 1×3 asymmetric convolutions and 5×1 and 1×5 asymmetric convolutions respectively. The elements of the two convolution results are summed, and finally a 1×1 convolution halves the number of channels to restore the number of channels.

[0069] The operation is as follows:

[0070] F1(x)=Conv 3×1 (Conv 1×3 (Conv 1×1 (x))) (3)

[0071] F2(x)=Conv 5×1 (Conv 1×5 (Conv1×1 (x))) (4)

[0072] F s (x)=Conv 1×1 (F1(x)+F2(x)) (5)

[0073] The depthwise separable extraction block can be further constructed as follows: grouped convolutions with the number of groups equal to the number of input channels are used, and pointwise convolutions with 1×1 kernels are used. The feature maps are serially input to the grouped and pointwise convolutions to form the depthwise separable extraction block. The specific operation is as follows:

[0074] F d (x)=Conv 1×1 (Conv g＝c (x)) (6)

[0075] Among them, Conv g＝c This represents grouped convolution. The features are input into the two-stage depth space extraction module in the spatial branch without changing the resolution, resulting in two feature maps, X2 and X3. The tensor shapes of the two feature maps are as follows: the resolution is maintained at 1 / 8, and the channel depth is expanded to 4C in both cases.

[0076] Step 4) Input1, the semantic branch, also consists of three stages. The first stage uses a dual-mapping summation module combining convolution and wavelet downsampling for feature extraction. The specific design details are as follows: 1×1 convolution expands the channels, proportionally dividing the channels into pooling convolution downsampling and Haar wavelet transform downsampling channel groups. Pooling convolution downsampling and Haar wavelet transform downsampling are performed on the two parts respectively, and the results are summed pixel-by-pixel. Finally, after 3×3 convolution, the feature map Y1 is obtained, reducing the resolution to 1 / 8 and expanding the number of channels to 2C. The Haar wavelet transform performs one-dimensional decomposition on the rows and columns of the image. Each decomposition produces four sub-bands: LL (low frequency to low frequency, approximation), LH (low frequency to high frequency, horizontal details), HL (high frequency to low frequency, vertical details), and HH (high frequency to high frequency, diagonal details). The details of the downsampling low-pass filtering are as follows:

[0077]

[0078] Among them, f j,n Let represent the approximation coefficients of the j-th level wavelet transform, h(k) be the low-pass filter coefficients, and n and l represent the scaling and translation parameters, respectively. Downsampling is also implicitly performed on the LL subband after each decomposition, halving its size and using it as input for the next level of decomposition. Therefore, the operation of the dual-mapping summation module is as follows:

[0079] F cd (x)=Cat d＝1 (Conv 3×3, s＝2 (x),Maxpool 2×2 (x)) (8)

[0080] F dd (x)=Conv 1×1 (HWD(x)+F cd (x)) (9)

[0081] Cat d＝1 Indicates channel-level concatenation operation, F cd F represents pooling convolution downsampling, HWD represents wavelet transform downsampling, and F represents... dd The second stage of the semantic branch employs a multi-level grouped dilated downsampling module to further increase the number of channels and aggregate semantic information. Its implementation details consist of three dilated convolutional layers, each using a different dilation rate to expand the receptive field. The list of the three cascaded dilation rates used is d = {[1,3,5],[3,5,7],[5,7,9]}. After three convolutional operations, the intermediate feature maps are connected layer by layer, and 1×1 convolutions are used for channel compression. The compressed feature map is residually concatenated with the input feature map. The semantic information feature map Y2, after passing through the multi-level grouped dilated downsampling module, has a resolution of 1 / 16 and expands the number of channels to 4C. Y2 is further input to a higher-dimensional dual-mapping summing module, with its input channel number set to 4C. Similarly, in the third stage, the dual-mapping summing module further doubles the number of channels from the second stage for feature extraction, resulting in feature map Y3 with a resolution of 1 / 32 and expands the number of channels to 8C.

[0082] Step 5) Design a feature fusion strategy. In the second and third stages of feature extraction, perform bi-branch feature fusion. Multi-scale fusion (CSF) is used to fuse semantic information and spatial details. The detailed design is as follows: The semantic branch feature map is upsampled to the size of the spatial feature map using bilinear interpolation, and the number of channels is matched. Elements are added to obtain a unified feature. The unified feature with a resolution of H×W is adaptively pooled and mapped to regions of equal size (1×1, 4×4, 8×8, 16×16). Then, 1×1 convolution is used for channel compression and expansion, and the region is reshaped to H×W size. The regions are accumulated to obtain region attention. Its mathematical representation is:

[0083] F Fuse =F Context +F Spatial (10)

[0084]

[0085] Where F Context and F SpatialThese represent semantic and spatial branch information, respectively. Conv represents channel expansion and compression operations, and Unpool represents average pooling, i.e., upsampling. This represents region attention; subsequently, pixel attention is calculated through two 1×1 standard convolutions, and the pixel attention and region attention are superimposed using a sigmoid activation function to obtain the weights at different locations in the feature map. Finally, the two branches are fused through element-wise multiplication; further mathematical representation is as follows:

[0086]

[0087]

[0088] F Out =F Context ×2α+F Spatial ×2*(1-α) (14)

[0089] in 'Add' represents pixel attention, and 'Add' represents the accumulation operation. This feature fusion method is applied to the feature fusion of X2 and Y2, with the fusion result used as input to X3; and to the feature fusion of X3 and Y3, with the fusion result used as input feature map Z1 for the next stage.

[0090] Step 6) The fused feature map Z1 is input into the spatially cascaded multi-scale pyramid pooling module for contextual feature embedding representation. The spatially cascaded pyramid pooling module uses a multi-scale strategy similar to the fusion method, such as... Figure 2 As shown, this is further demonstrated by: firstly, stacking four dilated convolutions with dilation rates of [3, 6, 12, 18,]. Then, superimposing the dilated convolutions with the upsampled feature maps after global pooling yields an intermediate feature map. This intermediate feature map is then concatenated, and a 1×1 convolution is used to compress the channels, performing a residual connection with the input. During intermediate feature computation, the number of input channels at each scale is reduced from 4C to 2C, allowing the model to achieve lightweight expressive power. This method can provide richer contextual information without adding additional convolutional layers. Feature map Z1 is processed by this module to obtain the contextual feature embedding representation feature map Z2. This module does not change the feature map resolution or the number of channels, maintaining 1 / 8 and 4C.

[0091] Step 7) Further input the embedded representation Z2 into the optimized channel attention and global attention for attention calculation. Specifically, the channel and global attention are: squeezed channel attention: global average pooling is performed on the input features, and the number of channels is reduced through 1×1 channel compression convolution to obtain the activation result. The activation result is then normalized using the Sigmoid activation function to obtain the weight information map. The original features... Figure 3A ×3 convolution transformation is performed and element-wise multiplied with the weight information map to obtain the channel attention map Z3. The number of channels is halved to 2C, while maintaining a resolution of 1 / 8. Z3 is input to the aggregation global attention module. The tensor shape of Z3 is (b,c,h,w), which is further constructed by adjusting the tensor shape to (b,h*w,c) to adapt to channel attention processing. Channel attention is applied, and the original tensor shape (b,c,h,w) is restored. Element-wise feature weighting is performed through the channel attention map, and finally, a spatial attention map is generated. Further element-wise feature weighting is applied to obtain the feature map Z4, maintaining a resolution of 1 / 8 and a channel number of 2C. This module enhances cross-dimensional interaction by preserving channel and spatial information, thereby improving the performance of deep neural networks. The global attention of the feature map can be represented as:

[0092]

[0093]

[0094] Where M c and M s This represents the channel and spatial attention mechanism, where F1 is the input feature map. This indicates element-wise multiplication.

[0095] Step 8) The attention calculation result is input into the upsampling module to restore spatial resolution. During the restoration process, long skip connections are maintained with the shallow feature map to enhance feature representation. Specifically, the upsampling module replaces the conventional bilinear interpolation upsampling with a combination of transposed convolution and bilinear interpolation. Simultaneously, long skip connections are maintained with the low-level features during the upsampling stage. To ensure rich spatial information in the high-level information, the shallow, same-resolution feature map X1 with sufficient spatial location information is subjected to a 3×3 channel matched convolution, summed with the feature map Z4 to be upsampled, and then subjected to a transposed convolution with a stride of 2 to complete the upsampling operation, resulting in Z5 with a resolution of 1 / 4 and channels compressed to C. This upsampling method can learn new feature representations and obtain more accurate segmentation results. Feature map Z5 is then processed through the final segmentation layer: input1 is subjected to a 3×3 channel matched convolution, summed with Z5, and then subjected to a bilinear upsampling with a scaling factor of 4 to restore the original resolution to 512×1024, including a 3×3 convolution to further restore the original channel count to 3. The image segmentation training task is now complete. Finally, the segmentation model is tested using validation and test sets, achieving a balance between inference speed and accuracy on Cityscapes with 74.5% mIOU accuracy and 86 FPS, and with only 1.7M parameters.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A real-time semantic segmentation method for multi-scale dual-stream fusion in road scenes, characterized in that: The method includes the following steps: Step 1) Obtain the urban road scene image dataset, divide the dataset into training, validation and test sets according to the proportion, and preprocess the data to obtain the initial images; Step 2) Input the initial image into the primary feature extraction module. This module consists of two downsampling convolutions and channel expansion convolutions, which reduce the image resolution while increasing the channel depth of the feature map, thereby obtaining the primary feature map. Step 3) The initial feature maps are input into a multi-stage parallel high-resolution spatial branch and low-resolution semantic branch network structure for feature extraction. Feature fusion of the two branches is performed at each stage of feature extraction, and the fusion result is incorporated into the input feature information of the next stage spatial branch. After multi-stage feature extraction and fusion, a fused feature map of deep semantic-guided spatial details is obtained. In Step 3), the high-resolution spatial branch consists of three stages: the first stage is constructed using the BasicBlock in ResNet, expanding the output channels and reducing the resolution; the second stage is constructed using depthwise separable convolution and spatial asymmetry. The deep spatial module, which combines convolution and deep processing, effectively reduces the number of parameters and computation in deep networks. The third stage further increases the number of channels in the deep spatial module to extract features. The semantic branch also consists of three stages. The first stage uses a dual-mapping summation module that combines convolution and wavelet downsampling to extract features. The second stage uses a multi-level grouped dilated downsampling module to further increase the number of channels and gather semantic information. The third stage uses a dual-mapping summation module to double the number of channels in the second stage for feature extraction. Feature fusion includes the fusion of semantic information and spatial details in the second stage and the fusion of semantic information and spatial information in the third stage, finally obtaining a fused feature map. Step 4) The fused feature map is input into the spatial cascaded multi-scale pyramid pooling module for contextual feature embedding representation. The embedded representation is then input into the optimized channel attention and global attention for attention calculation, which enriches the model's emphasis on different channels and the weight control of a single pixel on global information, and obtains the attention calculation result. Step 5) The attention calculation result is input into the upsampling module to restore the spatial resolution. During the restoration process, long connections are maintained with the shallow feature map to enhance the feature expression ability. Finally, the resolution is restored to the initial image to obtain the segmentation result. Step 6) Use the loss function to supervise the training and validation of the semantic segmentation model for urban road scenes, and obtain the real-time semantic segmentation model with the best validation effect. Use this model to process the test dataset to obtain the final image segmentation result.

2. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1, characterized in that: Step 1) includes: Step 1.1 Download the Cityscapes and CamVid city road scene datasets from open-source dataset websites; Step 1.2 Each dataset is proportionally divided into training, validation, and test sets; Step 1.3 Perform data augmentation preprocessing on the allocated dataset, including random horizontal flipping, mean subtraction, and random scaling, where the random scaling factors are set to {0.75, 1.0, 1.25, 1.5, 1.75, 2.0} respectively.

3. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1, characterized in that: The depth spatial module is specifically described as follows: First, a 1×1 convolution is performed to expand the channel. Then, the result is summed pixel by pixel through a spatially separable extraction block and a 3×3 convolution block. Finally, the result is passed through a depth-separable extraction block and residually connected to the initial input.

4. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1, characterized in that: The dual-mapping summation module and the multi-level grouped hole downsampling module are respectively: Dual-mapping summation module: The 1×1 convolution expands the channels, proportionally dividing the channels into pooling convolution downsampling and Haar wavelet transform downsampling channel groups. The two parts are subjected to pooling convolution downsampling and Haar wavelet transform downsampling respectively. The results are summed pixel by pixel, and finally passed through 3×3 convolution, reducing the feature map resolution to 1 / 2. The multi-level grouped dilated downsampling module consists of three dilated convolutional layers. Each convolutional layer uses a different dilation rate to expand the receptive field. After three convolution operations, the intermediate feature maps are connected layer by layer, and 1×1 convolution is used for channel compression. The compressed feature map is residually connected to the input feature map.

5. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1, characterized in that: The fusion of semantic information and spatial details adopts a multi-scale fusion strategy, specifically: The feature map with a resolution of H×W is adaptively pooled and mapped into regions of equal size: 1×1, 4×4, 8×8, and 16×16. Then, channel compression and expansion are performed, and the region is reshaped to H×W size. Region attention is accumulated from each region, and pixel attention is calculated through standard convolution. Pixel attention and region attention are superimposed through the sigmoid activation function to obtain weights at different positions in the feature map. Finally, the two branches are fused through element-wise multiplication. In step 4), the spatial cascaded pyramid pooling module also uses a multi-scale strategy. Specifically, the intermediate feature map obtained by superimposing the dilated convolution and the upsampled feature map after global pooling is connected. The intermediate feature map is compressed using 1×1 convolution and residual connection is performed with the input to provide contextual information without the need to add additional convolutional layers.

6. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1, characterized in that: In step 4), the channel and global attention are specifically as follows: global average pooling is performed on the input features, the number of channels is reduced by channel compression convolution to obtain the activation result, the activation result is normalized by applying the Sigmoid activation function to obtain weight information, and the original feature map is convolved and multiplied with the weight information elements to obtain the channel attention map; global attention is as follows: the tensor shape is adjusted to adapt to the channel attention processing, channel attention is applied and the original tensor shape is restored, feature weighting is performed through the channel attention map, and finally a spatial attention map is generated and applied for feature weighting. This module enhances cross-dimensional interaction by preserving channel and spatial information, thereby improving the performance of deep neural networks.

7. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1, characterized in that: In step 5), the upsampling module specifically involves replacing the conventional bilinear interpolation upsampling with a combination of transposed convolution and bilinear interpolation. Simultaneously, during the upsampling stage, it maintains skip connections with lower-level features. To ensure that higher-level information also possesses rich spatial information, the shallow, same-resolution feature map with spatial location information is summed with the feature map to be upsampled via channel-matched convolution, and then subjected to a transposed convolution with a stride of 2 to complete the upsampling operation. This process learns new feature representations and yields more accurate segmentation results.

Images