Multi-scale double-flow fusion real-time semantic segmentation method for road scene

Abstract

The invention relates to a road scene-oriented multi-scale double-flow fusion real-time semantic segmentation method, and belongs to the field of semantic segmentation. The method comprises the following steps: acquiring an image data set in a driving road scene; dividing the data in proportion and performing data enhancement; building a double-flow fusion network model; and training, verifying and testing the model by using the data set, and selecting the model with the optimal effect to obtain a semantic segmentation result. Features of the image are extracted through a space branch and a semantic branch, the space branch introduces a residual basic block formed by combining space separable convolution and depth separable convolution, the parameter quantity is reduced, the semantic branch introduces cascade cavity convolution and wavelet down-sampling feature extraction, the receptive field of the feature image is expanded, the fusion strategy adopts an attention weighting optimization mode, and the fusion speed is improved. More effective space and channel information is extracted, a new feature pyramid pooling structure is improved, and model generalization is enhanced. According to the method, the segmentation precision is improved while the real-time performance is ensured, and the balance between the calculation complexity and the precision is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation and relates to a multi-scale dual-stream fusion real-time semantic segmentation method for road scenes. Background Art

[0002] With the rapid progress of computer data processing capabilities, artificial intelligence has achieved unprecedented development. Computer vision, as one of the most important branches, has also received attention from various fields. Among them, semantic segmentation has been valued in autonomous driving tasks, which requires real-time segmentation of semantic information in road scenes. In order to achieve image classification at the pixel level, and then achieve a detailed understanding of the image content. In the field of autonomous driving, the perception speed and accuracy of real-time scenes are extremely high. These characteristics determine that improving the reasoning speed and accuracy of the model is the ideal choice. However, the existing semantic segmentation methods still face some challenges. For example, complex scenes such as lighting, occlusion, blurred edges and small targets make the actual target objects and pictures very different, which affects the segmentation results. In the existing semantic segmentation methods, the real-time and accuracy of segmentation are often inversely proportional. Some network models with higher segmentation accuracy often come at the cost of deeper network layers and larger parameters, resulting in too long reasoning time, while some segmentation models that pursue lightweight are often accompanied by a significant decrease in accuracy. Therefore, it is a very challenging task to design an efficient real-time semantic segmentation network considering both model accuracy and reasoning speed.

[0003] In the development of image semantic segmentation, commonly used segmentation methods can be mainly divided into two categories: traditional image semantic segmentation methods and image semantic segmentation methods based on deep learning. Image segmentation methods before the emergence of the deep learning wave are usually called traditional methods. According to different basic theories, they can be roughly divided into image segmentation methods based on thresholds, regions, edges or graph theory. These methods mainly use simple surface features such as texture, color, and shape to segment images into multiple differential regions. The advantages are simplicity and speed, but the segmentation results are relatively rough and cannot mark semantic information, making it difficult to accurately segment complex images. In the era of deep learning, convolutional neural networks have been applied in various fields. The full convolutional network proposed a segmentation method that combines convolutional neural networks and semantic segmentation, realizing end-to-end pixel-level segmentation. Subsequently, the real-time semantic segmentation network Enet was proposed, which adopts an asymmetric encoder-decoder architecture. The existing mainstream network architectures are usually divided into single-branch and dual-branch categories. The single branch is mainly based on the encoder-decoder architecture. Due to its single structure and limited feature extraction capability, the single branch cannot capture detailed spatial information. The dual-branch network consists of a spatial branch and a semantic branch. The spatial branch has a contradiction between its ability to capture semantic information and its speed of information capture due to its high resolution and high computational inefficiency. At the same time, the semantic branch performs continuous downsampling operations, which requires sufficient contextual information of the features. Existing methods inevitably lose some semantic information during the downsampling stage. Existing simple feature fusion is difficult to capture local details and global context information at the same time, which makes it difficult for the model to balance the relationship between local features and global context when dealing with small targets or complex scenes. This requires a multi-scale context fusion strategy. In summary, the approach to real-time semantic segmentation can be improved from the perspective of feature information fusion and spatial information recovery, balancing the existing semantic segmentation accuracy and speed issues. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a multi-scale dual-stream fusion real-time semantic segmentation method for road scenes. The codec structure is implemented by using a dual-branch data stream, and efficient cross-feature fusion is performed between the dual branches. The model is lightweighted by using depth-separable convolution and spatial asymmetric convolution. The wavelet transform downsampling is used to achieve lossless feature extraction. The structure block of pyramid pooling feature extraction is improved by cascaded dilated convolution, and a structure with higher precision and high generalization performance is embedded into the model. Finally, efficient real-time semantic segmentation is achieved. The network structure framework is as follows: Figure 1 shown.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A multi-scale dual-stream fusion real-time semantic segmentation method for road scenes includes the following steps:

[0007] Step 1) obtaining a dataset of urban road scene images, dividing the dataset into training, validation and test sets in proportion, and preprocessing the data to obtain initialization images;

[0008] Step 2) Input the initialized image into the primary feature extraction module, which consists of a one-step downsampling convolution and a channel expansion convolution, which reduces the image resolution and increases the channel depth of the feature map, thereby obtaining a primary feature map;

[0009] Step 3) The primary feature map is input into the multi-stage parallel high-resolution spatial branch and low-resolution semantic branch network structures for feature extraction, and the dual-branch feature fusion is performed at each stage of feature extraction, and the fusion result is incorporated into the input feature information of the spatial branch of the next stage. After multi-stage feature extraction and fusion, a fused feature map of deep semantics guiding spatial details is obtained;

[0010] Step 4) The fused feature map is input into the spatial cascade multi-scale pyramid pooling module (SCPPM) for context feature embedding representation, and the embedded representation is further input into the optimized channel attention and global attention for attention calculation, enriching the model's emphasis on different channels and the weight control of a single pixel on global information, and obtaining the attention calculation result;

[0011] Step 5) The attention calculation result is input to the upsampling module to restore the spatial resolution. During the restoration process, the long skip connection with the shallow feature map is maintained to enhance the feature expression ability. Finally, the resolution of the initialization image is restored to obtain the segmentation result;

[0012] Step 6) Use the loss function to supervise the training and verification of the urban road scene semantic segmentation model to obtain the real-time semantic segmentation model with the best verification effect. Use this model to process the test data set to obtain the final image segmentation result.

[0013] Furthermore, the above step 1) includes:

[0014] Step 1.1 Download the Cityscapes and CamVid urban road scene datasets from the open source dataset website;

[0015] Step 1.2 Each data set is divided into training set, validation set and test set in proportion;

[0016] Step 1.3 performs data augmentation preprocessing on the assigned dataset, including random horizontal flipping, mean subtraction, and random scaling, where the random scaling factors are set to {0.75, 1.0, 1.25, 1.5, 1.75, 2.0} respectively.

[0017] Furthermore, the step 3) includes: the high-resolution spatial branch is composed of three stages, the first stage is constructed as the basic residual block BasicBlock in ResNet, the output channel is expanded and the resolution is reduced, the second stage is constructed as a deep spatial module (DS Unit) combining deep separable convolution and spatial asymmetric separation convolution, which effectively reduces the number of parameters and calculations in the deep network, and the third stage deepens the number of channels in the deep spatial module to continue to extract features. The semantic branch is also composed of three stages. The first stage is a double mapping sum module (DM DownSampling) combining convolution and wavelet downsampling for feature extraction, the second stage adopts a multi-level grouping hole downsampling module (MEDSB) to further deepen the number of channels and gather semantic information, and the third stage uses a double mapping sum module to further double the number of channels in the second stage for feature extraction. Feature fusion includes the fusion of semantic information and spatial details in the second stage, and the fusion of semantic information and spatial information in the third stage, and finally obtains a fused feature map.

[0018] Furthermore, the depth space module is specifically as follows: first, a 1×1 convolution expansion channel is performed, and the results are summed pixel by pixel through the spatial separable extraction block and the 3×3 convolution block respectively, and then the results are continued through the depth separable extraction block to perform a residual connection with the initial input. The specific mathematical representation is as follows:

[0019] X s =F s (Conv 1×1 (x)) (1)

[0020] F ds (x) = F d (X s +Conv 3×3 (Conv 1×1 (x)))+Conv 1×1 (x) (2)

[0021] Among them, F s represents a spatially separable extraction block, F d represents the deep separable extraction block, F ds Represents a deep spatial module.

[0022] Furthermore, the spatially separable extraction block and the depth-separable extraction block are respectively: spatially separable extraction block: 1×1 convolution halve the number of channels, respectively through 3×1, 1×3 asymmetric convolution and 5×1, 1×5 asymmetric convolution, the elements of the two convolution results are summed, and finally 1×1 convolution halve the number of channels to restore the number of channels; depth-separable extraction block: set the number of groups to the number of input channels for grouped convolution, set the convolution kernel to 1×1 point-by-point convolution, and the feature map is serially input to the grouped and point-by-point convolution to form a depth-separable extraction block. The specific mathematical representation is as follows:

[0023] F1(x)=Conv 3×1 (Conv 1×3 (Conv 1×1 (x))) (3)

[0024] F2(x)=Conv 5×1 (Conv 1×5 (Conv 1×1 (x))) (4)

[0025] F s (x) = Conv 1×1 (F1(x)+F2(x)) (5)

[0026] F d (x) = Conv 1×1 (Conv g＝c (x)) (6)

[0027] Among them, Conv g＝c Represents grouped convolution.

[0028] Furthermore, the semantic branch dual mapping addition module and the multi-level grouping hole downsampling module are respectively: dual mapping addition module: 1×1 convolution expansion channel, divide the channel into pooling convolution downsampling and Haar wavelet transform downsampling channel groups in proportion, and perform pooling convolution downsampling and Haar wavelet transform downsampling on the two parts respectively. The results are summed pixel by pixel, and finally after 3×3 convolution, the feature map resolution is reduced to 1 / 2. The Haar wavelet transform will perform one-dimensional decomposition on the rows and columns of the image respectively. Each decomposition will produce four sub-bands: LL (low frequency-low frequency, approximation), LH (low frequency-high frequency, horizontal details), HL (high frequency-low frequency, vertical details), HH (high frequency-high frequency, diagonal details). The details of the downsampling low-pass filter are as follows:

[0029]

[0030] Among them, f j,n represents the approximate coefficient of the j-th layer wavelet transform, h(k) is the low-pass filter coefficient, n and l represent the scale and translation parameters respectively. Downsampling also implicitly operates on the LL subband after each decomposition, halving its size as the input of the next level of decomposition. The mathematical representation of the double mapping summation module is:

[0031] F cd (x) = Cat d＝1 (Conv 3×3 , s＝2 (x),Maxpool 2×2 (x)) (8)

[0032] F dd (x) = Conv 1×1 (HWD(x)+F cd (x)) (9)

[0033] Among them Cat d＝1 represents the channel dimension concatenation operation, F cd represents pooling convolution downsampling, HWD represents wavelet transform downsampling, F dd Represents a dual-map sum module. Multi-level grouped dilated downsampling module: It consists of three dilated convolutional layers, each of which uses a different dilation rate to expand the receptive field. After three convolution operations, the intermediate feature maps are connected layer by layer and channel compressed using 1×1 convolution. The compressed feature maps are residually connected to the input feature maps.

[0034] Furthermore, in step 3), the fusion of semantic information and spatial details adopts a multi-scale fusion strategy, specifically: adaptively pooling the feature map with a resolution of H×W into 1×1, 4×4, 8×8, and 16×16 regions of equal size, then compressing and expanding the channels, and then reshaping it to H×W size, accumulating each region to obtain regional attention, which is mathematically expressed as:

[0035] F Fuse =F Context +F Spatial (10)

[0036]

[0037] Among them, F Context and F Spatial Represents semantic branch and spatial branch information respectively, Gonv represents channel expansion and compression operations, UnPool represents average solution pooling, Represents regional attention; subsequently, pixel attention is calculated through standard convolution, and the weights of different positions in the feature map are obtained by superimposing pixel attention and regional attention through Sigmoid activation function, and finally the two branches are fused through element multiplication; the further mathematical representation is:

[0038]

[0039]

[0040] F Out =F Context ×2α+F Spatial ×2*(1-α) (14)

[0041] in Indicates pixel attention, and Add indicates accumulation operation. The spatial cascade pyramid pooling module in step 4) also uses a multi-scale strategy, specifically: superimposing the dilated convolution with the upsampled feature map after global pooling to obtain the intermediate feature map, connecting the intermediate feature maps, using 1×1 convolution to compress the channel and perform residual connection with the input. This method can provide richer contextual information without adding additional convolutional layers.

[0042] Furthermore, the channel and global attention described in step 4) are specifically as follows: Squeezed channel attention (SCA): Global average pooling of input features, channel compression convolution to reduce the number of channels to obtain excitation results, the excitation results are normalized by applying the Sigmoid activation function to obtain weight information, the original feature map is convolved and transformed and multiplied with the weight information elements to obtain the channel attention map; Aggregated global attention (AGA) is: adjust the tensor shape to adapt to the channel attention processing, apply channel attention and restore the original tensor shape, perform feature weighting through the channel attention map, and finally generate a spatial attention map and apply feature weighting. This module enhances cross-dimensional interaction by retaining channel and spatial information, thereby improving the performance of deep neural networks. The global attention of the feature map can be expressed as:

[0043]

[0044]

[0045] Among them, M c and M s represents the channel and spatial attention mechanism, F1 is the input feature map, Represents element-wise multiplication.

[0046] Furthermore, the upsampling module in step 5) is specifically as follows: conventional bilinear interpolation upsampling is replaced by a combination of transposed convolution and bilinear interpolation, while maintaining skip connections with low-level features in the upsampling stage. In order to make high-level information also have rich spatial information, the shallow feature map of the same resolution with sufficient spatial position information is convolved through channel matching, and is accumulated and summed with the feature map to be upsampled, and then a transposed convolution with a step size of 2 is performed to complete the upsampling operation. This method can learn new feature representations and obtain more accurate segmentation results.

[0047] The beneficial effects of the present invention are:

[0048] (1) Based on the encoder-decoder segmentation architecture, the present invention proposes a dual-branch data flow network structure. Compared with a single branch, it uses dual paths to extract information and integrates feature information from detail and semantic branches. It uses rich semantic information and attention mechanism to calculate proportional weights, guide the learning of detail branches, strengthen the information expression of features, and enhance the understanding ability of deep networks.

[0049] (2) The present invention uses the convolution decomposition method to decompose the standard convolution into depth-separable convolution or spatial-separable convolution, which significantly reduces the number of parameters and the amount of calculation. At the same time, the network depth can be increased while ensuring that the number of parameters remains basically unchanged, thereby enhancing the model performance. The information extraction block of the detail branch is constructed by combining depth-separable and spatial-separable convolution, solving the problem of inefficient reasoning on the spatial branch of most dual-branch networks.

[0050] (3) The present invention designs wavelet transform downsampling and cascaded dilated convolution methods in the semantic information capture stage. The former can maintain more boundary, scale and texture features, and reduce the resolution by retaining more spatial information, thereby achieving lossless downsampling. At the same time, this method is simple and efficient, and achieves a good balance between segmentation accuracy and computational efficiency; the latter expands the receptive field, enabling the network to obtain better perception of different scales and effectively learn semantic information at all levels.

[0051] (4) The present invention adopts a more efficient multi-scale feature fusion strategy in the dual-branch fusion and global feature pooling stages, allowing the model to adaptively learn high-order information at different semantic levels and selectively weight and fuse information at different scales. The spatial cascade pyramid pooling operation obtains feature information with a larger receptive field, thereby performing better on tasks that require a large receptive field.

[0052] (5) The present invention introduces channel attention to better learn the dependencies between channels and suppress redundant information. At the same time, it captures global context information through global attention, better understands the overall structure and context information of the target, and improves the modeling of long-distance dependencies, thereby obtaining more accurate feature representation.

[0053] (6) The present invention designs a more lightweight and efficient real-time semantic segmentation method based on multi-scale dual branches. The model does not require any pre-training and subsequent data processing operations, which significantly reduces the number of parameters and the size of the model, and can achieve a better balance between segmentation accuracy and inference speed.

[0054] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0056] Figure 1It is a flowchart of multi-scale dual-stream fusion real-time semantic segmentation for road scenes;

[0057] Figure 2 This is a multi-scale dual-stream fusion real-time semantic segmentation network framework for road scenes;

[0058] Figure 3 Schematic diagram of the SCPPM spatial cascade pyramid pooling module structure. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0060] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0061] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0062] A multi-scale dual-stream fusion real-time semantic segmentation method for road scenes includes the following steps:

[0063] Step 1) Download the Cityscapes and CamVid urban road scene datasets from the open source dataset website, divide the datasets into training, validation, and test sets in proportion, and preprocess the data respectively. Here, the Cityscapes dataset is taken as an example, and the initial training image resolution is 512×1024 and the number of channels is 3.

[0064] Step 2) Input the initialized training image into the primary feature extraction module, which consists of two convolutions, both of which are 3×3 downsampling convolutions with a step size of 2, and the number of channels for both convolutions is the standard channel C. Reduce the image resolution to 1 / 4 and increase the feature map channel depth to C, thereby obtaining the primary feature map input1.

[0065] Step 3) Input 1 is respectively input to the multi-stage parallel high-resolution spatial branch and low-resolution semantic branch network structures for feature extraction. The high-resolution spatial branch consists of three stages. The first stage is constructed as the basic residual block BasicBlock in ResNet, which expands the output channel and reduces the resolution to obtain the feature map X1. The resolution is reduced to 1 / 8 and the number of channels is 2C; the second stage is constructed as a deep spatial module combining deep separable convolution and spatial asymmetric separation convolution, which effectively reduces the number of parameters and calculations in the deep network. The deep spatial module is specifically as follows: first, a 1×1 convolution expansion channel is performed, and the results are summed pixel by pixel through the spatial separable extraction block and the 3×3 convolution block respectively, and then the deep separable extraction block is continued to be residually connected with the initial input. The specific mathematical representation is as follows:

[0066]

[0067] F ds (x) = F d (X s +Conv 3×3 (Conv 1×1 (x)))+Conv 1×1 (x) (2)

[0068] Among them, F s represents a spatially separable extraction block, F d represents the deep separable extraction block, F ds Represents the deep spatial module. The spatially separable extraction block is further constructed as follows: 1×1 convolution reduces the number of channels by half, and then 3×1, 1×3 asymmetric convolutions and 5×1, 1×5 asymmetric convolutions are performed respectively. The elements of the two convolution results are summed, and finally 1×1 convolution reduces the number of channels by half to restore the number of channels.

[0069] The operation is as follows:

[0070] F1(x)=Conv 3×1 (Conv 1×3 (Conv 1×1 (x))) (3)

[0071] F2(x)=Conv 5×1 (Conv 1×5 (Conv1×1 (x))) (4)

[0072] F s (x) = Conv 1×1 (F1(x)+F2(x)) (5)

[0073] The depth-separable extraction block can be further constructed as follows: set the group convolution with the number of groups equal to the number of input channels, set the convolution kernel to 1×1 point-by-point convolution, and input the feature map serially to the group and point-by-point convolution to form a depth-separable extraction block. The specific operations are as follows:

[0074] F d (x) = Conv 1×1 (Conv g＝c (x)) (6)

[0075] Among them, Conv g＝c Represents grouped convolution. The feature is input into the two-stage deep spatial extraction module in the spatial branch without changing the resolution, and two feature maps X2 and X3 are obtained. The shapes of the two feature map tensors are: the resolution is kept at 1 / 8, and the channel depth is expanded to 4C.

[0076] Step 4) Input 1 inputs the semantic branch, which also consists of three stages. The first stage is a dual-map summation module combining convolution and wavelet downsampling for feature extraction. The specific design details are: 1×1 convolution expansion channel, the channel is divided into pooled convolution downsampling and Haar wavelet transform downsampling channel groups in proportion, the two parts are respectively subjected to pooled convolution downsampling and Haar wavelet transform downsampling, the results are summed pixel by pixel, and finally after 3×3 convolution, the feature map Y1 is obtained, the resolution is reduced to 1 / 8, and the number of channels is expanded to 2C. The Haar wavelet transform will perform one-dimensional decomposition on the rows and columns of the image respectively. Each decomposition will produce four sub-bands: LL (low frequency-low frequency, approximate), LH (low frequency-high frequency, horizontal details), HL (high frequency-low frequency, vertical details), HH (high frequency-high frequency, diagonal details). The details of the downsampling low-pass filter are as follows:

[0077]

[0078] Among them, f j,n represents the approximate coefficient of the j-th layer wavelet transform, h(k) is the low-pass filter coefficient, n and l represent the scale and translation parameters respectively. Downsampling also implicitly operates on the LL subband after each decomposition, halving its size as the input of the next level of decomposition. The operation of the double mapping addition module is obtained as follows:

[0079] F cd (x) = Cat d＝1 (Conv 3×3, s＝2 (x),Maxpool 2×2 (x)) (8)

[0080] F dd (x) = Conv 1×1 (HWD(x)+F cd (x)) (9)

[0081] Among them Cat d＝1 represents the channel dimension concatenation operation, F cd represents pooling convolution downsampling, HWD represents wavelet transform downsampling, F dd Represents the dual-map sum module. The second stage of the semantic branch adopts a multi-level grouping hole downsampling module to further deepen the number of channels and gather semantic information. Its implementation details consist of three hole convolution layers. Each convolution layer uses a different hole rate to expand the receptive field. The three cascaded hole rate lists used are d = {[1,3,5], [3,5,7], [5,7,9]}. After three convolution operations, the intermediate feature maps are connected layer by layer and channel compression is performed using 1×1 convolution. The compressed feature map is residually connected with the input feature map. After the multi-level grouping hole downsampling module, the semantic information feature map Y2 has a resolution of 1 / 16 and the number of channels is expanded to 4C. Y2 is further input to a higher-dimensional dual-map sum module, and its input channel number is set to 4C. Similarly, the third stage uses the dual-map sum module to further double the number of channels in the second stage for feature extraction, obtaining the feature map Y3 with a resolution of 1 / 32 and an expanded number of channels to 8C.

[0082] Step 5) Design a feature fusion strategy, perform dual-branch feature fusion in the second and third stages of feature extraction, and use multi-scale fusion (CSF) to fuse semantic information and spatial details. The detailed design is: upsample the semantic branch feature map to the size of the spatial feature map by bilinear interpolation, match the number of channels, add element by element to obtain a unified feature, and adaptively pool the unified feature with a resolution of H×W to 1×1, 4×4, 8×8, and 16×16 regions of equal size, then perform channel compression and expansion through 1×1 convolution, and then reshape it to H×W size, accumulate each region to obtain regional attention, and its mathematical expression is:

[0083] F Fuse =F Context +F Spatial (10)

[0084]

[0085] Among them, F Context and F SpatialThey represent the semantic branch and spatial branch information respectively. Conv represents the channel expansion and compression operations. Unpool represents average unpooling, i.e. upsampling. Represents regional attention; subsequently, pixel attention is calculated through two 1×1 standard convolutions, and the weights of different positions in the feature map are obtained by superimposing pixel attention and regional attention through the Sigmoid activation function. Finally, the two branches are fused through element multiplication; the further mathematical representation is:

[0086]

[0087]

[0088] F Out =F Context ×2α+F Spatial ×2*(1-α) (14)

[0089] in Represents pixel attention, and Add represents the accumulation operation. This feature fusion method is applied to the feature fusion of X2 and Y2, and the fusion result is used as the input of X3; and the feature fusion of X3 and Y3, and the fusion result is used as the input feature map Z1 of the next stage.

[0090] Step 6) The fused feature map Z1 is input to the spatial cascade multi-scale pyramid pooling module for context feature embedding representation. The spatial cascade pyramid pooling module uses a multi-scale strategy similar to the fusion method, such as Figure 2 As shown in the figure, it is further expressed as follows: first, 4 dilated convolutions are stacked, with dilation rates of [3, 6, 12, 18] respectively, and the dilated convolution is superimposed with the upsampled feature map after global pooling to obtain an intermediate feature map, which is then connected. The channels are compressed using 1×1 convolution and residual connections are performed with the input. When calculating the intermediate features, the number of input channels at each scale is reduced from 4C to 2C, allowing the model to achieve lightweight expression capabilities. This method can provide richer contextual information without adding additional convolutional layers. The feature map Z1 is embedded with the contextual features to represent the feature map Z2 through this module. This module does not change the resolution and number of channels of the feature map, and keeps 1 / 8 and 4C.

[0091] Step 7) The embedded representation Z2 is further input into the optimized channel attention and global attention for attention calculation. The channel and global attention are specifically as follows: Squeeze channel attention: perform global average pooling on the input features, reduce the number of channels by 1×1 channel compression convolution to obtain the excitation result, apply the Sigmoid activation function to the excitation result for normalization to obtain the weight information map, the original feature Figure 3×3 convolution transformation and element-by-element multiplication with the weight information map to obtain the channel attention map Z3; the number of channels is halved to 2C, and the resolution is maintained at 1 / 8; Z3 is input to the aggregated global attention module, the Z3 tensor shape is (b, c, h, w), and it is further constructed to adjust the tensor shape to (b, h*w, c) to adapt to the channel attention processing, apply channel attention and restore the original tensor shape (b, c, h, w), perform element-wise feature weighting through the channel attention map, and finally generate a spatial attention map and apply further element-wise multiplication for feature weighting to obtain the feature map Z4, maintaining the resolution of 1 / 8 and the number of channels as 2C. This module enhances cross-dimensional interactions by retaining channel and spatial information, thereby improving the performance of deep neural networks. The global attention of the feature map can be expressed as:

[0092]

[0093]

[0094] Among them, M c and M s represents the channel and spatial attention mechanism, F1 is the input feature map, Represents element-wise multiplication.

[0095] Step 8) The attention calculation result is input to the upsampling module to restore the spatial resolution. During the restoration process, the long skip connection with the shallow feature map is maintained to enhance the feature expression ability. The upsampling module is specifically as follows: the conventional bilinear interpolation upsampling is replaced by a combination of transposed convolution and bilinear interpolation, and the skip connection with the low-level features is maintained during the upsampling stage. In order to make the high-level information also have rich spatial information, the shallow feature map X1 with sufficient spatial position information is convolved through 3×3 channels, and the feature map Z4 to be upsampled is accumulated and summed, and then the upsampling operation is completed by transposed convolution with a step size of 2 to obtain Z5, with a resolution of 1 / 4 and a channel compression to C. This upsampling method can learn new feature representations and obtain more accurate segmentation results. The feature map Z5 passes through the last segmentation layer, specifically: input1 passes through 3×3 channel matching convolution, and is accumulated and summed with Z5. The skip connection is finally restored to the original resolution of 512×1024 through bilinear upsampling with a scaling factor of 4, including 3×3 convolution to further restore the original channel to 3. At this point, the image segmentation training task is completed. Finally, the segmentation model is tested using the validation set and the test set, achieving a balance between inference speed and accuracy with an mIOU accuracy of 74.5% and a speed of 86FPS on Cityscapes, and the number of parameters is only 1.7M.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-scale dual-stream fusion real-time semantic segmentation method for road scenes, characterized by:

1. The method comprises the following steps: Step 1) obtaining a dataset of urban road scene images, dividing the dataset into training, validation and test sets in proportion, and preprocessing the data to obtain initialization images; Step 2) Input the initialized image into the primary feature extraction module, which consists of a one-step downsampling convolution and a channel expansion convolution, which reduces the image resolution and increases the channel depth of the feature map, thereby obtaining a primary feature map; Step 3) The primary feature map is input into the multi-stage parallel high-resolution spatial branch and low-resolution semantic branch network structures for feature extraction, and the dual-branch feature fusion is performed at each stage of feature extraction, and the fusion result is incorporated into the input feature information of the spatial branch of the next stage. After multi-stage feature extraction and fusion, a fused feature map of deep semantics guiding spatial details is obtained; Step 4) The fused feature map is input into the spatial cascade multi-scale pyramid pooling module for context feature embedding representation, and the embedded representation is further input into the optimized channel attention and global attention for attention calculation, so as to enrich the model's emphasis on different channels and the weight control of a single pixel on global information, and obtain the attention calculation result; Step 5) The attention calculation result is input to the upsampling module to restore the spatial resolution. During the restoration process, the long skip connection with the shallow feature map is maintained to enhance the feature expression ability. Finally, the resolution of the initialization image is restored to obtain the segmentation result; Step 6) Use the loss function to supervise the training and verification of the urban road scene semantic segmentation model to obtain the real-time semantic segmentation model with the best verification effect. Use this model to process the test data set to obtain the final image segmentation result.

2. According to claim 1, a multi-scale dual-stream fusion real-time semantic segmentation method for road scenes is characterized by: The step 1) comprises: Step 1.1 Download the Cityscapes and CamVid urban road scene datasets from the open source dataset website; Step 1.2 Each data set is divided into training set, validation set and test set in proportion; Step 1.3 performs data augmentation preprocessing on the assigned dataset, including random horizontal flipping, mean subtraction, and random scaling, where the random scaling factors are set to {0.75, 1.0, 1.25, 1.5, 1.75, 2.0} respectively.

3. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1 is characterized by: In the step 3), the high-resolution spatial branch consists of three stages. The first stage is constructed as the basic residual block BasicBlock in ResNet, which expands the output channel and reduces the resolution. The second stage is constructed as a deep spatial module combining deep separable convolution and spatial asymmetric separation convolution, which effectively reduces the number of parameters and calculations in the deep network. The third stage deepens the number of channels in the deep spatial module to continue to extract features. The semantic branch also consists of three stages. The first stage uses a double mapping addition module combining convolution and wavelet downsampling to extract features. The second stage adopts a multi-level grouping hole downsampling module to further deepen the number of channels and gather semantic information. The third stage uses a double mapping addition module to further double the number of channels in the second stage for feature extraction, and feature fusion It includes the fusion of semantic information and spatial details in the second stage, and the fusion of semantic information and spatial information in the third stage, and finally obtains a fused feature map.

4. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 3 is characterized by: The depth space module is specifically as follows: first, a 1×1 convolution expansion channel is performed, and the results are summed pixel by pixel through a spatial separable extraction block and a 3×3 convolution block respectively, and then a residual connection is performed with the initial input through a depth separable extraction block.

5. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 3 is characterized by: The dual mapping summing module and the multi-level grouping hole downsampling module are respectively: dual mapping summing module: 1×1 convolution expands the channel, and divides the channel into pooled convolution downsampling and Haar wavelet transform downsampling channel groups in proportion. The two parts are subjected to pooled convolution downsampling and Haar wavelet transform downsampling respectively, and the results are summed pixel by pixel. Finally, after 3×3 convolution, the feature map resolution is reduced to 1 / 2; Multi-level grouped hole downsampling module: It consists of three hole convolution layers, each convolution layer uses a different hole rate to expand the receptive field. After three convolution operations, the intermediate feature maps are connected layer by layer, and 1×1 convolution is used for channel compression. The compressed feature map is residually connected to the input feature map.

6. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 3 is characterized by: The fusion of semantic information and spatial details adopts a multi-scale fusion strategy, specifically: The feature map with a resolution of H×W is adaptively pooled and mapped into 1×1, 4×4, 8×8, and 16×16 regions of equal size, and then the channels are compressed and expanded, and then reshaped to H×W size, and the regional attention is accumulated in each region to obtain the regional attention, the pixel attention is calculated by standard convolution, and the pixel attention and regional attention are superimposed by the Sigmoid activation function to obtain the weights of different positions in the feature map, and finally the two branches are fused by element multiplication; the spatial cascade pyramid pooling module in step 4) also uses a multi-scale strategy, specifically: the intermediate feature map obtained by superimposing the void convolution with the feature map upsampled after global pooling, connecting the intermediate feature maps, using 1×1 convolution to compress the channel and perform residual connection with the input to provide context information without adding additional convolutional layers.

7. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1, characterized in that: In the step 4), the channel and global attention are specifically as follows: globally average pooling the input features, reducing the number of channels through channel compression convolution to obtain the excitation result, applying the Sigmoid activation function to the excitation result to normalize the weight information, and convolutionally transforming the original feature map and multiplying it with the weight information element to obtain the channel attention map; the global attention is as follows: adjusting the shape of the tensor to adapt to the channel attention processing, applying the channel attention and restoring the original tensor shape, performing feature weighting through the channel attention map, and finally generating a spatial attention map and applying it for feature weighting. This module enhances cross-dimensional interaction by retaining channel and spatial information, thereby improving the performance of deep neural networks.

8. The multi-scale dual-stream fusion real-time semantic segmentation method for road scenes according to claim 1, characterized in that: In the step 5), the upsampling module is specifically as follows: conventional bilinear interpolation upsampling is replaced by a combination of transposed convolution and bilinear interpolation, while maintaining a skip connection with low-level features in the upsampling stage. In order to make high-level information also have rich spatial information, the shallow feature map of the same resolution with spatial position information is convolved through channel matching, and is accumulated and summed with the feature map to be upsampled, and then a transposed convolution with a step size of 2 is performed to complete the upsampling operation, learn a new feature representation, and obtain a better segmentation result.

Images