A road scene segmentation method based on improved Deeplabv3+ network model

Abstract

The present invention discloses a road scene segmentation method based on an improved Deeplabv3+ network model, comprising the steps of obtaining and preprocessing road scene image data to generate a labeled image; combining the preprocessed image and the labeled image to obtain a data set and dividing the data set into a training set, a validation set, and a test set; constructing an improved Deeplabv3+ network model; using the training set and the validation set to train the improved Deeplabv3+ network model; and inputting the preprocessed images of the test set into the trained model to obtain a segmentation result of the road scene image. The present invention replaces two-dimensional convolution with quaternion convolution to improve the network's fitting ability and reduce the number of parameters; replaces dilated convolution with large convolution to reconstruct the spatial pyramid pooling module, thereby increasing the effective receptive field and improving the reuse efficiency of the information to be characterized; introducing the feature map output by the downsampling layer of the encoding end and the attention mechanism module at the decoding end to achieve multi-scale feature fusion and detail perception; and decoupling the training and inference of the model to reduce the amount of computation and memory consumption.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation technology, and in particular to a road scene segmentation method based on an improved DeepLabV3+ network. Background Art

[0002] Semantic segmentation of urban road images is a technology that divides each pixel in an image into different color regions based on its category. This technology aims to provide a more detailed understanding of image semantics, enabling better applications such as road traffic management, autonomous driving, and intelligent transportation. Traditional clustering segmentation algorithms struggle to subdivide categories, thus failing to capture useful semantic information. A common technique currently employed is the use of deep learning models, such as convolutional neural networks, which can learn from large amounts of data to perform image recognition and segmentation, leading to more refined semantic segmentation. For example, segmenting road areas can enable functions such as traffic congestion detection and intelligent streetlight control, while segmenting objects such as vehicles and pedestrians can enable traffic violation detection and pedestrian safety monitoring.

[0003] DeepLabv3+ is a semantic segmentation model based on deep learning. It uses advanced technologies such as dilated convolution and multi-scale fusion to process images of different scales, and can adapt to different data sets and tasks. Compared with other models, DeepLabv3+ has higher accuracy and precision, and can accurately identify different objects and regions in the image. However, when segmenting object boundaries, discontinuities and segmentation errors may occur. In order to solve this problem, the present invention improves the DeepLabv3+ algorithm based on the reuse of underlying features and the capture of multi-scale semantic information. By constructing a quaternion convolutional neural network to extract shallow network detail information, and then performing cascade feature fusion to complete the reuse of underlying information, the spatial pyramid structure is finally used to further obtain multi-scale feature information to achieve a more accurate object boundary segmentation effect. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies in the prior art and provide a road scene segmentation method based on an improved DeepLabV3+ network, which can effectively reuse underlying features and capture multi-scale semantic information to achieve more accurate object boundary segmentation.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solution: a road scene segmentation method based on an improved DeepLabv3+ network model, characterized by comprising the following steps:

[0006] (1) Acquire road scene image data;

[0007] (2) preprocessing the road scene image data, generating a label image based on the preprocessed image; combining the preprocessed image and the corresponding label image in a one-to-one correspondence to obtain a data set; and dividing the data set into a training set, a validation set, and a test set;

[0008] (3) Introducing quaternion convolution to improve the ConvNeXt network; using the Deeplabv3+ network as a prototype, the improved ConvNeXt network replaces the Xception network in the backbone network, and introduces the feature map output by the downsampling layer in the encoding end and the attention mechanism module at the decoding end to construct an improved Deeplabv3+ network model;

[0009] (4) The training set and validation set are used to train the improved Deeplabv3+ network model;

[0010] (5) The preprocessed images of the test set are input into the trained improved Deeplabv3+ network model to obtain the segmentation results of the road scene images.

[0011] Furthermore, in step (2), the preprocessing includes resizing, flipping, standard normalization and cropping operations.

[0012] Furthermore, in step (2), the method of generating the label image includes manual drawing, mask drawing, semi-automatic annotation and AI-assisted annotation.

[0013] Furthermore, in step (3), the introduction of quaternion convolution to improve the ConvNeXt network is specifically as follows: the convolution kernel of the ConvNeXt convolution layer is represented as a quaternion form, and quaternion multiplication is used to implement the convolution operation to obtain an improved ConvNeXt network.

[0014] Furthermore, the step (3) also includes: reconstructing the ASPP module in the Deeplabv3+ network by replacing the dilated convolution with a large convolution to obtain a RepASPP module, wherein the RepASPP module is used to decouple the training and inference of the improved Deeplabv3+ network model.

[0015] Furthermore, the step (4) is specifically as follows: inputting the training set into the improved Deeplabv3+ network model, performing calculations and obtaining prediction results; calculating the error between the prediction results and the label image, and using the gradient optimization algorithm to reduce the error and update the model weight parameters; through the iterative optimization process, the model gradually converges.

[0016] Furthermore, the step (4) includes the following sub-steps:

[0017] (4.1) Initialize the improved Deeplabv3+ network model using weights pre-trained on the ImageNet dataset;

[0018] (4.2) Input the preprocessed images in the training set into the improved Deeplabv3+ network model to obtain the prediction results of road scene image segmentation;

[0019] (4.3) Use the cross entropy loss function to calculate the error between the prediction result and the corresponding label image;

[0020] (4.4) Based on the calculated error, the gradient descent algorithm is used to update the weight parameters of the improved Deeplabv3+ network;

[0021] (4.5) Repeat steps (4.2) to (4.4). At the end of each iteration, use the preprocessed images in the validation set to evaluate the improved Deeplabv3+ network model and select the optimal model; until the iteration is completed, save the trained model.

[0022] The beneficial effects of the present invention are:

[0023] 1. The present invention uses quaternion convolution to replace two-dimensional convolution, which improves the fitting ability of the network and reduces the number of parameters;

[0024] 2. The present invention uses large convolution to replace the dilated convolution to reconstruct the spatial pyramid pooling module, thereby increasing the effective receptive field and improving the reuse efficiency of the information to be tested;

[0025] 3. The present invention introduces the feature information output by the downsampling layer in the encoding end and the attention mechanism module at the decoding end to achieve better multi-scale feature fusion and detail perception;

[0026] 4. The present invention uses structural reparameterization technology to decouple model training and inference, reducing computational complexity and memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a basic flow chart of a road scene segmentation method using an improved Deeplabv3+ network model provided by an embodiment of the present invention;

[0028] Figure 2 Schematic diagram of the improved DeepLabV3+ network model structure provided by an embodiment of the present invention;

[0029] Figure 3 2 is a schematic diagram of an improved spatial pyramid pooling module provided by an embodiment of the present invention;

[0030] Figure 4 This is an architecture diagram for decoupling training and reasoning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0032] Example 1

[0033] like Figure 1 FIG. 1 is a schematic diagram of a basic flow chart of a road scene segmentation method using an improved Deeplabv3+ network according to an embodiment of the present invention. The road scene segmentation method using an improved Deeplabv3+ network according to the present invention comprises the following steps:

[0034] Step 1: Acquire road scene image data;

[0035] In this embodiment, a camera device is used to capture image data in actual road scenes. The camera device is mounted on a collection platform, such as a vehicle or mobile robot, ensuring proper sensor installation and calibration to obtain accurate image data. Sensor parameters, including exposure time, focal length, and field of view, are then adjusted to ensure the captured image quality meets the required requirements. The location and path for data collection are then determined based on the required road scene data range and coverage area. Different road types, such as urban streets, rural roads, or highways, can be selected as needed. The camera device continuously captures image data and stores it on appropriate media, such as a hard drive, memory card, or cloud storage. Relevant information, such as timestamps, location information, and sensor parameters, is recorded for subsequent processing and analysis. Finally, the captured image data undergoes a quality check, including assessments of image clarity, exposure, and occlusion. Low-quality or abnormal data is excluded to ensure that the captured image data possesses reliable quality characteristics.

[0036] Step 2: Preprocess the road scene image data and generate a labeled image based on the preprocessed image; the methods for generating the labeled image include manual drawing, mask drawing, semi-automatic annotation, and AI-assisted annotation; the preprocessed image and the corresponding labeled image are combined in a one-to-one correspondence to obtain a dataset; the dataset is divided into a training set, a validation set, and a test set;

[0037] In this embodiment, the road scene image collected in step 1 is resized to a size of 224×224×3. This ensures image consistency during subsequent processing and training. The resized road scene image is preprocessed using the transform module, performing random flipping, standard normalization, and random cropping operations to enhance the generalization ability of the model. The preprocessed image is processed using a mask annotation method to mark the pixel positions of each area in the image. This can also be achieved through manual annotation or automatic annotation algorithms. The generated labeled image has the same size as the preprocessed image, where the value of each pixel represents the category to which the pixel belongs. The preprocessed image and the corresponding labeled image are combined in a one-to-one correspondence, that is, each preprocessed image corresponds to a labeled image, thus obtaining a dataset. 60% of the samples in the dataset are randomly selected as the training set, 20% of the samples are used as the validation set, and the remaining 20% ​​of the samples are used as the test set, thereby obtaining training samples and test samples.

[0038] Step 3: Quaternion convolution is introduced to improve the ConvNeXt network. Taking the Deeplabv3+ network as the prototype, the improved ConvNeXt network replaces the Xception network in the backbone network. The feature map output by the downsampling layer in the encoder and the attention mechanism module are introduced at the decoding end to build an improved Deeplabv3+ network model. The improved Deeplabv3+ network model can also be constructed by replacing the atrous convolution with large convolution to reconstruct the Atrous Spatial Pyramid Pooling (ASPP) module, and using structural reparameterization technology to decouple training and inference.

[0039] See Figure 2 , Figure 2 This is a schematic diagram of the improved DeepLabV3+ network model structure provided by an embodiment of the present invention.

[0040] (3.1) Quaternion convolution is introduced to improve the ConvNeXt network; taking the Deeplabv3+ network as the prototype, the improved ConvNeXt network replaces the Xception network in the backbone network.

[0041] This paper introduces quaternion convolution to improve the ConvNeXt network, represents the convolution kernel of the ConvNeXt convolution layer in quaternion form, and uses quaternion multiplication to implement the convolution operation, successfully extending the traditional two-dimensional convolution to three-dimensional data processing. By using quaternion convolution, the weight ratio of different training targets is optimized, thereby better adapting to the feature distribution of various training targets. The improved ConvNeXt network replaces the Xception network in the backbone network. The improved ConvNeXt network can better extract and represent image features, thereby enhancing the performance and effect of the improved Deeplabv3+ network model in image segmentation tasks.

[0042] like Figure 2 As shown in the figure, the improved Deeplabv3+ network model mainly consists of three modules: Encoder, RepASPP and Decoder.

[0043] First, given an input image, the improved Deeplabv3+ network model uses a modified ConvNeXt network as an encoder to extract hierarchical features. The improved ConvNeXt network has five downsampling layers, each with a stride of 2, meaning that the size of the input image is reduced by 1 / 2 after each downsampling layer. After the first downsampling layer, the image size becomes 1 / 2 of the input image, and after the second downsampling layer, the image size becomes 1 / 4 of the input image. As a result, the final feature map output by the encoder is 1 / 32 of the input image size, indicating that after five downsampling layers, the feature map size is reduced by a factor of 32 relative to the input image size.

[0044] Then, the improved Deeplabv3+ network model uses RepASPP to model random dependencies. RepASPP takes the encoder output features as input and generates a feature that contains global context information.

[0045] Finally, the improved Deeplabv3+ network model uses a decoder to progressively fuse multi-level features and output the resulting image. The decoder consists of two fused ECANet (Efficient Channel Attention Network) attention modules and a segmentation head. Each ECANet takes two features as input: low-level feature maps extracted by each encoder stage and high-level feature maps generated by RepASPP or a deeper fusion module. Resize and Predict operations are performed in the segmentation head to reduce the channels of the 1 / 8 downsampled features to the number of classes. An upsampling operation is used to scale the features to the input image size, and an argmax operation is used to predict the label for each pixel.

[0046] See Figure 3 , Figure 3 This is an improved spatial pyramid pooling module provided by an embodiment of the present invention.

[0047] (3.2) Reconstruct the spatial pyramid pooling module by replacing the dilated convolution with a large convolution to increase the effective receptive field and improve the reuse efficiency of the information to be tested. The specific reconstruction steps are as follows:

[0048] (3.2.1) Determine the size of the large convolution kernel used to reconstruct the ASPP module. Figure 3 As shown, the embodiment of the present invention proposes using convolution kernels of sizes 1, 13, 25, and 31. The sizes of these convolution kernels can capture semantic information of different scales.

[0049] (3.2.2) The original dilated convolution in the ASPP module is replaced with a new convolution kernel using a large convolution kernel, resulting in the RepASPP module. Compared to small convolution kernels and dilated convolution, large convolution kernels have a larger effective receptive field and are more suitable for capturing shape deviations rather than texture deviations.

[0050] (3.3) The feature map and attention mechanism module of the downsampling layer output in the encoding end are introduced at the decoding end to achieve better multi-scale feature fusion and detail perception.

[0051] To achieve better multi-scale feature fusion and detail perception, the decoder incorporates feature information from the downsampling layer output by the encoder. The encoder uses a modified ConvNeXt network to extract features from the input image. The downsampling layer typically reduces the size of the feature map to increase the receptive field and extract higher-level semantic information. Due to the scale change, the output feature maps of different downsampling layers have different spatial resolutions and semantic information. After each deconvolution layer in the decoder, the feature map of the corresponding downsampling layer is fused with the feature map of the decoder. The decoder uses upsampling and skip connections to restore the encoder feature map to the original image size and perform semantic segmentation prediction. Skip connections can be implemented through element-by-element addition or element-by-element concatenation.

[0052] To more effectively utilize the feature maps output by the encoder, the ECANet attention mechanism module is used during the fusion process. This module uses one-dimensional convolution to achieve local cross-channel interactions and extract dependencies between channels. This improvement helps the network focus more closely on the target to be detected, highlighting important features and further improving the accuracy of the network model in object detection.

[0053] See Figure 4 , Figure 4 This is a diagram of the decoupled training and reasoning architecture provided by an embodiment of the present invention.

[0054] (3.4) Using structural reparameterization technology, the RepASPP module is adopted to decouple the training and inference of the model.

[0055] like Figure 4 As shown in , the RepASPP structure consists of a common basic structure and a set of learnable parameters. The basic structure includes the Concat operation for feature fusion and the 1x1 convolution operation, which are shared during training and inference. Figure 4 As shown in Figure A, RepASPP uses standard convolution calculations, including backpropagation and parameter updates, to train learnable parameters. In the inference phase, as Figure 4 As shown in Figure B, RepASPP replaces the parameters of four parallel convolutional layers with a fixed convolution kernel parameter, greatly reducing the amount of computation and memory consumption.

[0056] By decoupling training and inference, the improved Deeplabv3+ network model can fully utilize backpropagation and parameter updates during training, improving the model's expressiveness and performance. During inference, the use of fixed convolution kernel parameters significantly reduces computational complexity and memory consumption, improving inference efficiency. This decoupled design optimizes training and inference for different needs, ensuring both the model's learning capabilities and improving computational efficiency in practical applications.

[0057] Step 4: Input the preprocessed images of the training set into the improved Deeplabv3+ network model, perform calculations and obtain prediction results; calculate the error between the prediction results and the label images, and use the gradient optimization algorithm to reduce the error and update the model weight parameters; through the iterative optimization process, the model gradually converges;

[0058] The specific process of model training is as follows:

[0059] (4.1) Weight initialization: Initialize the improved Deeplabv3+ network model using weights pre-trained on the ImageNet dataset.

[0060] (4.2) Forward propagation: The preprocessed images in the training set are fed into the improved Deeplabv3+ network model and processed to obtain the predicted image segmentation results. The predicted results have the same size as the preprocessed images, where each pixel is assigned a class label indicating the class to which the pixel belongs.

[0061] (4.3) Calculation error: The cross entropy loss function is used to calculate the error between the prediction result and the label image. The prediction probability of the improved Deeplabv3+ network model for the i-th pixel is p i ,y iis the true label, and the cross entropy loss function is recorded as :

[0062]

[0063] In the formula, ∑ represents the sum of all pixels, y i Represents the one-hot encoding of the true label (the position belonging to the category is set to 1, and the other positions are set to 0), p i Represents the model's predicted probability for the i-th pixel.

[0064] (4.4) Back propagation: Based on the loss function, the chain rule is used to calculate the gradient of the loss with respect to the network weight parameters. The weight parameter of the network model is ω, and the loss function is , calculated by the back-propagation algorithm The formula for the gradient of ω is:

[0065]

[0066] Where, Represents the gradient of the loss function to the prediction result p, Represents the gradient of the prediction result p to the weight parameter ω. The backpropagation algorithm calculates the gradient of each layer through backpropagation layer by layer and passes the gradient back to each layer of the network.

[0067] (4.5) Update weight parameters: Based on the calculated error, use the gradient descent algorithm to update the weight parameters of the network model. The weight parameter of the network model is ω, the learning rate is η, and the formula for updating the weight parameters is:

[0068]

[0069] in, Represents the gradient of the error with respect to the weight parameter, obtained by calculating the back propagation.

[0070] (4.6) Repeat steps (4.2) to (4.3). After each iteration, use the validation set to evaluate the performance of the model on unseen data. Based on the performance on the validation set, adjust and optimize the model, such as adjusting hyperparameters, modifying the model structure, etc. Ensure the generalization ability of the model on unseen data and prevent overfitting. If the loss function does not change much in several consecutive iterations, that is, it tends to be stable, it can be considered that the model has converged and the model training stops. Through multiple iterative optimizations, the model gradually converges, that is, the error between the prediction result and the label image is continuously reduced, thereby improving the accuracy of segmentation.

[0071] Through the iterative optimization process, the model gradually converges. The present invention can effectively reuse underlying features and capture multi-scale semantic information to achieve a more accurate object boundary segmentation effect.

[0072] Step 5: Input the preprocessed images of the test set into the trained improved Deeplabv3+ network model to obtain the segmentation results of the test set image predictions; compare the obtained test set image prediction results with the corresponding label images, calculate the intersection over union (IoU), and take the average of the IoU values ​​of all images to obtain the mean intersection over union (mIoU) as a performance evaluation indicator to measure the accuracy and effect of the improved Deeplabv3+ network model on the road scene image segmentation task.

[0073] The formula for calculating the intersection-over-union (IoU) index is:

[0074] IoU = TP / (TP+FP+FN)

[0075] Where TP represents the true positive, that is, the number of pixels predicted to be positive and whose corresponding position in the label image is positive; FP represents the false positive, that is, the number of pixels predicted to be positive but whose corresponding position in the label image is negative; FN represents the false negative, that is, the number of pixels predicted to be negative but whose corresponding position in the label image is positive.

[0076] The calculation formula of mean intersection over union (mIoU) is:

[0077] mIoU=(IoU1+IoU2+…+IoUn) / n

[0078] Where IoU1, IoU2, …, IoUn represent the intersection-over-union (IoU) of each image, and n represents the total number of images. mIoU ranges from 0 to 1, with values ​​closer to 1 indicating better accuracy and performance of the improved DeepLabV3+ network model on the road scene image segmentation task.

[0079] The present embodiment provides a road scene segmentation method based on an improved Deeplabv3+ network model, which comprises the following steps: using a camera device to collect image data in an actual road scene; performing preprocessing operations on the collected image, including resizing, random flipping, standard normalization, and random cropping operations; processing the preprocessed image using a mask annotation method, annotating the pixel positions of each area in the image, and generating a corresponding labeled image; combining the preprocessed image and the corresponding labeled image in a one-to-one correspondence manner to obtain a data set; dividing the data set into a training set and a test set; and using a pre-trained image on the ImageNet data set. The improved Deeplabv3+ network model is initialized with weights; the preprocessed images of the training set are input into the improved Deeplabv3+ network model, and the calculation is performed to obtain the prediction results; the error between the prediction results and the label images is calculated, and the gradient optimization algorithm is used to reduce the error and update the model weight parameters; through the iterative optimization process, the model gradually converges; the preprocessed images of the test set are input into the trained improved Deeplabv3+ network, and the average intersection-over-union index performance evaluation index is calculated to measure the accuracy and effect of the improved Deeplabv3+ network in the road scene image segmentation task.

[0080] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A road scene segmentation method based on an improved Deeplabv3+ network model, characterized in that: The following steps are involved: (1) Acquire road scene image data; (2) preprocessing the road scene image data, generating a label image based on the preprocessed image; combining the preprocessed image and the corresponding label image in a one-to-one correspondence manner to obtain a data set; Divide the dataset into training, validation, and test sets; (3) Introducing quaternion convolution to improve the ConvNeXt network; Using the Deeplabv3+ network as a prototype, the improved ConvNeXt network replaces the Xception network in the backbone network. The feature map output by the downsampling layer in the encoder and the attention mechanism module are introduced to the decoder to build an improved Deeplabv3+ network model. (4) The training set and validation set are used to train the improved Deeplabv3+ network model; (5) The preprocessed images of the test set are input into the trained improved Deeplabv3+ network model to obtain the segmentation results of the road scene images.

2. The road scene segmentation method based on the improved Deeplabv3+ network model according to claim 1 is characterized in that: In the step (2), the preprocessing includes resizing, flipping, standard normalization and cropping operations.

3. The road scene segmentation method based on the improved Deeplabv3+ network model according to claim 1 is characterized in that: In the step (2), the method of generating the label image includes manual drawing, mask drawing, semi-automatic labeling and AI-assisted labeling.

4. The road scene segmentation method based on the improved Deeplabv3+ network model according to claim 1, characterized in that: In the step (3), the introduction of quaternion convolution to improve the ConvNeXt network is specifically as follows: the convolution kernel of the ConvNeXt convolution layer is represented as a quaternion form, and quaternion multiplication is used to implement the convolution operation to obtain an improved ConvNeXt network.

5. The road scene segmentation method based on the improved Deeplabv3+ network model according to claim 1, characterized in that: The step (3) further includes: reconstructing the ASPP module in the Deeplabv3+ network by replacing the dilated convolution with a large convolution to obtain a RepASPP module, wherein the RepASPP module is used to decouple the training and inference of the improved Deeplabv3+ network model.

6. The road scene segmentation method based on the improved Deeplabv3+ network model according to claim 1, characterized in that: The step (4) is specifically as follows: inputting the training set into the improved Deeplabv3+ network model, performing calculations and obtaining prediction results; calculating the error between the prediction results and the label image, and reducing the error using a gradient optimization algorithm, and updating the model weight parameters; through an iterative optimization process, the model gradually converges.

7. The road scene segmentation method based on the improved Deeplabv3+ network according to claim 1 or 6, characterized in that: The step (4) includes the following sub-steps: (4.1) Initialize the improved Deeplabv3+ network model using weights pre-trained on the ImageNet dataset; (4.2) Input the preprocessed images in the training set into the improved Deeplabv3+ network model to obtain the prediction results of road scene image segmentation; (4.3) Use the cross entropy loss function to calculate the error between the prediction result and the corresponding label image; (4.4) Based on the calculated error, the gradient descent algorithm is used to update the weight parameters of the improved Deeplabv3+ network; (4.5) Repeat steps (4.2) to (4.4). At the end of each iteration, use the preprocessed images in the validation set to evaluate the improved Deeplabv3+ network model and select the optimal model; until the iteration is completed, save the trained model.

Images