An urban road scene segmentation method based on improved DeepLabV3+

Abstract

The application discloses a kind of urban road scene segmentation methods based on improved DeepLabV3+, first in the backbone part of network, replace original network structure using lightweight MobileNetV2 network, reduce parameter quantity and complexity, reduce the running time of network;Secondly, a low-level feature is extracted in the model, so that the decoding part fuses more scale features, reduces the spatial information loss caused by downsampling;The ASPP module is improved in the form of dense link and replaced with the original ordinary convolution using depth separable convolution, and the attention mechanism module is introduced in multiple places on the model.After optimization, the network structure effectively reduces the calculation time while maintaining high classification accuracy, has good performance and strong applicability.The application focuses on the scene understanding of urban road, has high complexity, diversity and data volume, and has a certain effect on the environment perception part of unmanned driving.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an improved urban road scene segmentation method based on DeepLabV3+. Background Technology

[0002] Road environment perception is a key technology in vehicle-assisted driving technology, and the analysis and understanding of road scenes is an important component of in-vehicle intelligent systems. In autonomous driving, the goal of road scene recognition is to automatically acquire high-level semantic information from road images, determine the scene category of the image, and extract drivable areas from the image. For example, classifying and recognizing road scenes such as streets, rural roads, and highways under complex conditions like rain, snow, fog, and haze, and providing warnings such as speed adjustments to vehicles, will contribute to achieving driverless driving. In automotive driver assistance systems, deep semantic understanding of road scenes aids in vehicle navigation and intelligent warnings. Using labeled road scene images can better depict the driver's environment, playing a crucial role in driving safety.

[0003] Currently, semantic segmentation networks in deep learning have a large number of parameters, long computation time, low segmentation accuracy, and are not suitable for mobile devices. Summary of the Invention

[0004] The purpose of this invention is to provide an improved urban road scene segmentation method based on DeepLabV3+.

[0005] The technical solution adopted in this invention is:

[0006] A method for segmenting urban road scenes based on an improved DeepLabV3+ includes the following steps:

[0007] Step 1: Acquire urban traffic scene images, and then perform random scaling, cropping, horizontal flipping, and data preprocessing to obtain training images;

[0008] Step 2: Construct the DeeplabV3+ model. The DeeplabV3+ model includes an encoder and a decoder. An additional feature extraction branch is added to the encoder to transfer information between shallow and deep features, thereby improving the model's feature learning ability.

[0009] At the encoding end, the DeepLabV3+ backbone network is replaced with the MobileNetV2 network. The MobileNetV2 network includes seven linear bottleneck structures, each of which contains several inverted residual structures.

[0010] Step 2: The original image is processed by a deep convolutional neural network at the encoding end of the DeeplabV3+ model to extract information. The image is downsampled by 16 times after passing through the backbone network. The extracted information is then processed by 1x1 convolutions, depthwise separable convolutional layers with dilatancy of 6, 12, and 18, and pooling layers, with an attention mechanism added. The results are concatenated along the channel dimension, and then the number of channels is reduced by 1x1 convolutions to obtain the feature map at the encoding end.

[0011] Step 2: The DeeplabV3+ model decoder obtains the output of the first inverse residual structure from the encoder, passes it through the SE attention mechanism and a 1*1 convolution, and concatenates it with the output of the second inverse residual structure through the SE attention mechanism to fuse multi-scale information and obtain low-level features. Then, the encoder feature map is bilinearly interpolated by 4 times and concatenated with the low-level features. After passing through the SE attention mechanism and a 3*3 convolution to adjust the number of channels and add contextual information, it is finally returned after 4 times bilinear interpolation to obtain the predicted segmentation map with the same size as the original image.

[0012] Furthermore, the data preprocessing in step 1 includes the following steps:

[0013] Step 1-1: First, perform random rotation, translation, and cropping operations on the traffic scene image;

[0014] Steps 1-2 involve standardizing the data by subtracting the mean and dividing by the variance to keep the input data within [0,1], preventing data oscillations at the beginning of training, accelerating model convergence, and improving stability.

[0015] Furthermore, the MobileNetV2 network has a total of 17 inverted residual structures.

[0016] Furthermore, the stride of the 6th linear bottleneck structure in MobileNetV2 is changed to 1, and the depthwise separable convolutions of the 8th-17th inverse residual structures are replaced with depthwise separable convolutions with holes to increase the receptive field; attention mechanism modules are added to multiple parts of the network to improve the segmentation accuracy of the model.

[0017] By reconstructing the ASPP in the network into a dense ASPP and replacing ordinary convolutions with depthwise separable convolutions, the segmentation accuracy is improved while the segmentation time of the network is greatly reduced. The reconstructed model is named DDASPP (DepthwiseDense ASPP).

[0018] Furthermore, the specific steps of step 3 are as follows:

[0019] Step 3-1: The training image is input into the encoder of the DeeplabV3+ model. After strided convolution with a size of 3x3, a stride of 2, and a number of kernels of 32, the first output feature map is obtained with an image size of half that of the training image and a number of output channels of 32.

[0020] Step 3-2: Pass the first output feature map through two linear bottleneck structures to obtain an image with the first output feature map as its size. Figure 1 The second output feature map has 24 output channels and is half-sized;

[0021] Step 3-3: Pass the second output feature map through a linear bottleneck structure to obtain an image with the second output feature map as its size. Figure 1 The third output feature map with half output channels and 32 output channels;

[0022] Steps 3-4 involve passing the third output feature map through a linear bottleneck structure to obtain an image with the third output feature map as its size. Figure 1 The fourth output feature map with half output channels and 64 output channels;

[0023] Steps 3-5 involve passing the fourth output feature map through three linear bottleneck structures to obtain an image with the fourth output feature map as its size. Figure 1 The fifth output feature map with half output channels and 320 output channels;

[0024] Steps 3-6 involve passing the fifth output feature map through a DDASPPP structure and then performing 1x1 convolution for dimensionality reduction to obtain the image size and the fifth output feature map. Figure 1 The output channel is a feature map of the encoding end with 256 channels.

[0025] Furthermore, the specific steps of step 4 are as follows:

[0026] Step 4-1: The output of the first linear bottleneck structure of the encoder, after passing through the SE attention mechanism and 1*1 convolution, is concatenated with the output of the second linear bottleneck structure of the encoder after passing through the SE attention mechanism to obtain a low-dimensional output feature map.

[0027] Step 4-2: Adjust the number of channels of the low-dimensional output feature map through a 1x1 convolution to obtain the sixth output feature map;

[0028] Step 3-3: After upsampling the encoding end feature map through bilinear interpolation, it is concatenated with the sixth output feature map to obtain the seventh output feature map;

[0029] Step 4-4: The seventh output feature map is processed by an attention mechanism and convolved with a 3x3 size, and then upsampled back to the original image size by bilinear interpolation to obtain the predicted segmentation map.

[0030] Furthermore, the SE attention mechanism is implemented in the following steps: Obtain input X input The squeezing and excitation module selectively emphasizes interdependent channel maps by integrating features from all channel maps, input X. input By extruding F in sequence sq (Global Average Pooling) and F ex (Fully connected components and activation functions) are used to obtain the channel attention coefficients; finally, they are compared with X. input Matrix multiplication is used to further improve the representation of channel dimension features, which contributes to more accurate segmentation results.

[0031] This invention employs the above technical solutions. First, the lightweight MobileNetV2 network is used to replace the original network structure in the backbone, reducing the number of parameters and complexity, and shortening the network runtime. Second, an additional low-level feature is extracted from the model, allowing the decoding part to integrate more scale features and reducing spatial information loss due to downsampling. The ASPP module is improved with dense connections and the original ordinary convolution is replaced with depthwise separable convolution. Attention (SE) mechanisms are introduced in multiple places on the model. The optimized network structure effectively reduces computation time while maintaining high classification accuracy, exhibiting good performance and strong applicability. The Cityscapes and Camvid datasets used in this invention focus on scene understanding of urban roads, possessing high complexity, diversity, and data volume, which plays a certain role in the environmental perception part of autonomous driving. Attached Figure Description

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0033] Figure 1 This is a schematic diagram of the improved DeepLabV3+ network structure for an urban road scene segmentation method based on the improved DeepLabV3+ of the present invention.

[0034] Figure 2 This is a schematic diagram illustrating the principle and structure of the SE attention mechanism;

[0035] Figure 3 This is a schematic diagram of the segmentation results for the Cityscapes dataset;

[0036] Figure 4 This is a schematic diagram of the segmentation results for the Camvid dataset. Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0038] like Figures 1 to 4 As shown in the figure, this invention discloses an urban road scene segmentation method based on an improved DeepLabV3+, which includes the following steps:

[0039] Step 1: Acquire urban traffic scene images, and then perform random scaling, cropping, horizontal flipping, and data preprocessing to obtain training images;

[0040] Step 2: Construct the DeeplabV3+ model. The DeeplabV3+ model includes an encoder and a decoder. An additional feature extraction branch is added to the encoder to transfer information between shallow and deep features, thereby improving the model's feature learning ability.

[0041] At the encoding end, the DeepLabV3+ backbone network is replaced with the MobileNetV2 network. The MobileNetV2 network includes seven linear bottleneck structures, each of which contains several inverted residual structures.

[0042] Step 2: The original image is processed by a deep convolutional neural network at the encoding end of the DeeplabV3+ model to extract information. The image is downsampled by 16 times after passing through the backbone network. The extracted information is then processed by 1x1 convolutions, depthwise separable convolutional layers with dilatancy of 6, 12, and 18, and pooling layers, with an attention mechanism added. The results are concatenated along the channel dimension, and then the number of channels is reduced by 1x1 convolutions to obtain the feature map at the encoding end.

[0043] Step 2: The DeeplabV3+ model decoder obtains the output of the first inverse residual structure from the encoder, passes it through the SE attention mechanism and a 1*1 convolution, and concatenates it with the output of the second inverse residual structure through the SE attention mechanism to fuse multi-scale information and obtain low-level features. Then, the encoder feature map is bilinearly interpolated by 4 times and concatenated with the low-level features. After passing through the SE attention mechanism and a 3*3 convolution to adjust the number of channels and add contextual information, it is finally returned after 4 times bilinear interpolation to obtain the predicted segmentation map with the same size as the original image.

[0044] Furthermore, the data preprocessing in step 1 includes the following steps:

[0045] Step 1-1: First, perform random rotation, translation, and cropping operations on the traffic scene image;

[0046] Steps 1-2 involve standardizing the data by subtracting the mean and dividing by the variance to keep the input data within the range [0,1]. This prevents data oscillations at the beginning of training, accelerates model convergence, and improves stability. The mean values ​​are 0.485, 0.456, and 0.406, and the variances are 0.229, 0.224, and 0.225, respectively.

[0047] Furthermore, the MobileNetV2 network has a total of 17 inverted residual structures.

[0048] Furthermore, the stride of the 6th linear bottleneck structure in MobileNetV2 is changed to 1, and the depthwise separable convolutions of the 8th-17th inverse residual structures are replaced with depthwise separable convolutions with holes to increase the receptive field; attention mechanism modules are added to multiple parts of the network to improve the segmentation accuracy of the model.

[0049] By reconstructing the ASPP in the network into a dense ASPP and replacing ordinary convolutions with depthwise separable convolutions, the segmentation accuracy is improved while the segmentation time of the network is greatly reduced. The reconstructed model is named DDASPP (DepthwiseDense ASPP).

[0050] Furthermore, the specific steps of step 3 are as follows:

[0051] Step 3-1: The training image is input into the encoder of the DeeplabV3+ model. After strided convolution with a size of 3x3, a stride of 2, and a number of kernels of 32, the first output feature map is obtained with an image size of half that of the training image and a number of output channels of 32.

[0052] Step 3-2: Pass the first output feature map through two linear bottleneck structures to obtain an image with the first output feature map as its size. Figure 1 The second output feature map has 24 output channels and is half-sized;

[0053] Step 3-3: Pass the second output feature map through a linear bottleneck structure to obtain an image with the second output feature map as its size. Figure 1 The third output feature map with half output channels and 32 output channels;

[0054] Steps 3-4 involve passing the third output feature map through a linear bottleneck structure to obtain an image with the third output feature map as its size. Figure 1 The fourth output feature map with half output channels and 64 output channels;

[0055] Steps 3-5 involve passing the fourth output feature map through three linear bottleneck structures to obtain an image with the fourth output feature map as its size. Figure 1 The fifth output feature map with half output channels and 320 output channels;

[0056] Steps 3-6 involve passing the fifth output feature map through a DDASPPP structure and then performing 1x1 convolution for dimensionality reduction to obtain the image size and the fifth output feature map. Figure 1 The output channel is a feature map of the encoding end with 256 channels.

[0057] Furthermore, the specific steps of step 4 are as follows:

[0058] Step 4-1: The output of the first linear bottleneck structure of the encoder, after passing through the SE attention mechanism and 1*1 convolution, is concatenated with the output of the second linear bottleneck structure of the encoder after passing through the SE attention mechanism to obtain a low-dimensional output feature map.

[0059] Step 4-2: Adjust the number of channels of the low-dimensional output feature map through 1x1 convolution to obtain the sixth output feature map; that is, the low-dimensional output feature map obtained by splicing has 40 channels, and after 1x1 convolution, it has 48 channels.

[0060] Step 3-3: After upsampling the encoding end feature map through bilinear interpolation, it is concatenated with the sixth output feature map to obtain the seventh output feature map;

[0061] Step 4-4: The seventh output feature map is processed by an attention mechanism and convolved with a 3x3 size, and then upsampled back to the original image size by bilinear interpolation to obtain the predicted segmentation map.

[0062] Furthermore, such as Figure 2 As shown, the SE attention mechanism is executed in the following steps: Obtain input X input The squeezing and excitation module selectively emphasizes interdependent channel maps by integrating features from all channel maps, input X. input By extruding F in sequence sq (Global Average Pooling) and F ex (Fully connected components and activation functions) are used to obtain the channel attention coefficients; finally, they are compared with X. input Matrix multiplication is used to further improve the representation of channel dimension features, which contributes to more accurate segmentation results.

[0063] This invention improves the DeepLabV3+ model, which consists of an encoder and a decoder. It can be applied to other segmentation datasets such as the Cityscapes dataset and the Camvid dataset, achieving segmentation performance superior to other networks like FCN and BiseNet. The network comprises an encoder and a decoder. An additional feature extraction branch is added to the encoder to facilitate information transfer between shallow and deep features, improving the model's feature learning ability. In the encoder, the DeepLabV3+ backbone network is replaced with MobileNetV2. The stride of the 6th linear bottleneck structure in MobileNetV2 is changed to 1, and the depthwise separable convolutions of the 8th-17th inverse residual structures are replaced with depthwise separable convolutions with holes to increase the receptive field. Attention mechanisms are added to multiple points in the network to improve segmentation accuracy. The ASPP in the network is reconstructed into a dense ASPP, and ordinary convolutions are replaced with depthwise separable convolutions, significantly reducing segmentation time while improving accuracy. The reconstructed model is named DDASPP (Depthwise Dense ASPP).

[0064] Results Description: In semantic segmentation, the evaluation metrics are mean intersection-union ratio (mIoU), mean pixel accuracy (MPA), and number of parameters.

[0065] The experiment used a 64-bit Windows 10 operating system, an 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz CPU, 16GB of RAM, and an NVIDIA GeForce RTX 3060 Laptop GPU with 6GB of VRAM. The development environment was PyCharm, and the deep learning frameworks used were PyTorch 3.8, CUDA 11.3, and CUDNN 8.2.

[0066] Other experimental parameters are set as shown in Table 1 below.

[0067] Table 1 Parameter settings

[0068] Parameter name Cityscape Parameter Selection Camvid Parameter Selection crop size 768*768 720*720 Validation size 1024*2048 960*720 Loss function Cross entropy Cross entropy Batch size 2 2 Optimizer SGD SGD Scheduler Poly Poly Max iteration 161000 30000 Output stride 16 16

[0069] When the crop size was set to 513*513, convergence was slow and segmentation accuracy was low. Considering memory resources, the crop size for Cityscapes was set to 768*768, and the crop size for Camvid was set to 720*720, with a batch size of 2 for both. Single images were used for validation, and the validation size was the same as the original image size. The optimizer used SGD, with a momentum of 0.9, and a poly learning strategy. The learning rate can be expressed as:

[0070]

[0071] In the formula: base_lr = 0.01 is the initial learning rate, cur_itr is the current iteration number, max_itr is the maximum iteration number, and power = 0.9 is the decay exponent.

[0072] The mIoU value is an important indicator of segmentation accuracy, and it has become the most commonly used metric due to its simplicity and strong representativeness. It is calculated by averaging the IoU of each class, where IoU is (true positive samples / (true positive samples + false negative samples + false positive samples)). The mIoU calculation formula can be expressed as:

[0073]

[0074] In the formula: k+1 represents k semantic categories and 1 background, P ij To predict category i as category j.

[0075] The performance comparison of different algorithms on Cityscapes is shown in Table 2 below:

[0076] Table 2: Performance of different algorithms on Cityscapes

[0077] Backbone mIoU Parameters / MB Enet - 47.46 0.336 FCB-8s VGG-16 62.21 30.036 Bisenet Resnet18 67.96 12.796 DeeplabV3+ MobilenetV2 71.53 5.225 ours MobilenetV2 73.90 3.492

[0078] Furthermore, MPA calculates the correct pixel proportion for each category and then averages it, which can be expressed as:

[0079]

[0080] The comparison of different algorithm models is shown in Table 3:

[0081] Table 3: Comparison of different algorithms

[0082] Model Dataset MPA mIoU DeeplabV3+ CamVid 83.94 78.00 ours CamVid 86.08 80.13 DeeplabV3 Cityscapes 81.15 71.53 ours Cityscapes 83.00 73.90

[0083] This invention employs the above technical solutions. First, the lightweight MobileNetV2 network is used to replace the original network structure in the backbone, reducing the number of parameters and complexity, and shortening the network runtime. Second, an additional low-level feature is extracted from the model, allowing the decoding part to integrate more scale features and reducing spatial information loss due to downsampling. The ASPP module is improved with dense connections and the original ordinary convolution is replaced with depthwise separable convolution. Attention (SE) mechanisms are introduced in multiple places on the model. The optimized network structure effectively reduces computation time while maintaining high classification accuracy, exhibiting good performance and strong applicability. The Cityscapes and Camvid datasets used in this invention focus on scene understanding of urban roads, possessing high complexity, diversity, and data volume, which plays a certain role in the environmental perception part of autonomous driving.

[0084] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. An improved DeepLabV3+ based urban road scene segmentation method, characterized in that: It comprises the following steps: Step 1, obtaining urban traffic scene image, and obtaining training image after random scaling, cropping, horizontal flipping and data preprocessing; Step 2, constructing DeeplabV3+ model, the DeeplabV3+ model includes an encoding end and a decoding end, an additional feature extraction branch is added to the encoding end to realize information transmission of shallow features and deep features; In the encoding end, the backbone network of DeepLabV3+ is replaced by MobileNetV2 network, and the MobileNetV2 network includes 7 linear bottleneck structures, each linear bottleneck structure includes a plurality of reverse residual structures, Step 3, the training image is extracted by the deep convolutional neural network of the encoding end of the DeeplabV3+ model, and the training image is down-sampled by 16 times after passing through the backbone network; the down-sampled information is respectively subjected to 1x1 convolution, deep separable convolution with a hole rate of 6, 12 and 18, and pooling layer, and the results of adding attention mechanism are spliced in the channel dimension, and then the channel number is reduced by 1x1 convolution to obtain the encoding end feature map; Step 4, the output of the first reverse residual structure of the encoding end of the DeeplabV3+ model is subjected to SE attention mechanism and 1*1 convolution, and the output of the second reverse residual structure is subjected to SE attention mechanism, and then the two are spliced to fuse multi-scale information to obtain low-level features; Then the encoding end feature map is spliced with the low-level features after being up-sampled by 4 times by bilinear interpolation, and then subjected to SE attention mechanism and 3*3 convolution to adjust the channel number and increase the context information, and finally subjected to 4 times of bilinear interpolation to return to the prediction segmentation map with the size of the original image.

2. The improved DeepLabV3+ based urban road scene segmentation method according to claim 1, characterized in that: The data preprocessing of step 1 includes the following steps: Step 1-1, first, randomly rotate, translate and crop the traffic scene image to obtain initial image data; Step 1-2, the initial image data is subjected to standardization of mean value subtraction and variance division to keep the image data input into the network within [0, 1].

3. The improved DeepLabV3+ based urban road scene segmentation method according to claim 1, characterized in that: The MobileNetV2 network has a total of 17 reverse residual structures.

4. The improved DeepLabV3+ based urban road scene segmentation method according to claim 3, characterized in that: The stride of the 6th linear bottleneck structure of the MobileNetV2 is changed to 1, the depth separable convolution of the 8th-17th reverse residual structure is replaced by the depth separable convolution with holes to increase the receptive field, and the attention mechanism module is added to the network at multiple places to improve the segmentation accuracy of the model.

5. The improved DeepLabV3+ based urban road scene segmentation method according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3-1, the training image is input into the encoder of the DeeplabV3+ model, and after the cross-step convolution with a size of 3x3, a stride of 2 and a convolution kernel number of 32, a first output feature map with an image size of half of the training image and an output channel number of 32 is obtained; step 3-2, the first output feature map is subjected to two linear bottleneck structures to obtain a second output feature map with an image size of half of the first output feature map and an output channel number of 24; Step 3-3, the second output feature map is subjected to a linear bottleneck structure to obtain a third output feature map with an image size of half of the second output feature map and an output channel number of 32; Step 3-4, the third output feature map is passed through a linear bottleneck structure to obtain a fourth output feature map with an image size of half of the third output feature map and an output channel number of 64; Step 3-5, the fourth output feature map is passed through three linear bottleneck structures to obtain a fifth output feature map with an image size of half of the fourth output feature map and an output channel number of 320; Step 3-6, the fifth output feature map is passed through a DDASPP structure and a 1*1 convolution dimension reduction to obtain an encoding end feature map with an image size same as the fifth output feature map and an output channel number of 256.

6. The improved DeepLabV3+ based urban road scene segmentation method according to claim 1, characterized in that: The specific steps of step 4 are as follows: Step 4-1, the output of the first linear bottleneck structure of the encoder is passed through the SE attention mechanism and the result of the 1*1 convolution, and the output of the second linear bottleneck structure of the encoder is passed through the SE attention mechanism to obtain a low-dimensional output feature map; Step 4-2, the low-dimensional output feature map is adjusted by 1x1 convolution to obtain a sixth output feature map; Step 3-3, the encoding end feature map is passed through bilinear interpolation up-sampling and spliced with the sixth output feature map to obtain a seventh output feature map; Step 4-4, the seventh output feature map is passed through the attention mechanism and the 3x3 convolution, and then the bilinear interpolation up-sampling is performed to return to the original image size to obtain the predicted segmentation map.

7. The improved DeepLabV3+ based urban road scene segmentation method according to claim 6, characterized in that: The channel number of the low-dimensional output feature map is 40, and the channel number of the sixth output feature map is 48.

8. The improved DeepLabV3+ based urban road scene segmentation method according to claim 1, characterized in that: The specific execution of the SE attention mechanism is as follows: Obtain input features X input It utilizes a squeezing and excitation module to selectively emphasize interdependent channel maps by integrating features from all channel maps; input feature X input F is squeezed sequentially through global average pooling. sq and fully connected and activation function F ex Obtain the channel attention coefficients; finally, compare them with the input feature X. input Matrix multiplication is used to further improve the representation of channel dimension features, which contributes to more accurate segmentation results.

Images