Urban streetscape semantic segmentation method based on improved DeepLabV3

Abstract

The invention discloses a city streetscape semantic segmentation method based on improved DeepLabV3, comprising the following steps: step 1, acquiring a city streetscape image data set including training data, verification data and test data, and performing data enhancement on the training data; step 2, establishing an improved DeepLabV3 semantic segmentation network model; step 3, training the improved DeepLabV3 semantic segmentation network model by using the training data obtained in the step 1, and evaluating and adjusting the model by using verification data in the training process to obtain a target improved DeepLabV3 semantic segmentation network model; and 4, inputting the test data obtained in the step 1 into the improved DeepLabV3 semantic segmentation network model in the step 3 to obtain a semantic segmentation result. According to the method, the fineness of feature extraction is improved, and the extraction capability of the model on key features in the channel dimension and the space dimension is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method for semantic segmentation of urban street scenes based on improved DeepLabV3. Background Art

[0002] Semantic segmentation is a computer vision technique that aims to classify each pixel in an image into different object categories. It has a wide range of applications in urban landscapes, such as autonomous driving, traffic control, and environmental monitoring. However, the complexity and diversity of urban landscapes increase the accuracy of the model's classification and segmentation of different categories of elements.

[0003] DeepLabV3 is a network model with good performance in semantic segmentation. The shapes, sizes and appearances of the elements in the complexity and diversity of urban landscapes vary significantly. Existing semantic segmentation methods based on DeepLabV3 are difficult to accurately segment objects of different sizes and shapes at the same time. In addition, objects are often occluded and overlapped in urban environments, making the boundaries of objects complex and blurred, reducing segmentation accuracy.

[0004] In view of this, the inventors provide a city street scene semantic segmentation method based on improved DeepLabV3 to solve the above technical problems. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and propose a semantic segmentation method for urban street scenes based on improved DeepLabV3. The method comprises an improved DeepLabV3 semantic segmentation network model. The model adopts ResNet50 with a smaller step size to form a feature extraction network, which not only ensures the computational efficiency but also significantly improves the precision of feature extraction. A CBAM (convolutional block attention) module is embedded after the feature extraction network to enhance the model's ability to extract key features in both channel and spatial dimensions. The expansion rate of the ASPP (spatial pyramid pooling) module is also increased to enable the model to capture richer contextual information while making up for the detail features that may be missed during the feature extraction process.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a method for semantic segmentation of urban street scenes based on improved DeepLabV3, comprising the following steps:

[0008] Step 1: Obtain a city street view image dataset, including training data, verification data, and test data, and perform data enhancement on the training data;

[0009] Step 2: Establish and improve the DeepLabV3 semantic segmentation network model;

[0010] Step 3: Use the training data obtained in step 1 to train the improved DeepLabV3 semantic segmentation network model, and use the verification data to evaluate and adjust the model during the training process to obtain the target improved DeepLabV3 semantic segmentation network model;

[0011] Step 4: Input the test data obtained in step 1 into the target improved DeepLabV3 semantic segmentation network model in step 3 to obtain the semantic segmentation result.

[0012] Furthermore, in step 1, the city street view image dataset uses the Cityscapes dataset and / or the GTA5 dataset, and randomly divides the city street view image dataset into training data, verification data and test data according to a ratio of 1.9-2.1:0.9-1.1:0.9-1.1.

[0013] Furthermore, in step 1, the data enhancement is to randomly crop and randomly horizontally flip the training data using average fusion, and the calculation formula of the average fusion is as follows:

[0014]

[0015] Among them, P final is the final prediction result, P i is the prediction result of the i-th scale or flipped image, and N is the total number of prediction results.

[0016] Furthermore, in step 2, establishing an improved DeepLabV3 semantic segmentation network model includes the following steps:

[0017] Step 2.1: Input the city street view image into the improved ResNet50 feature extraction network to obtain feature maps F of different resolutions;

[0018] Step 2.2, integrate the CBAM module into the DeepLabV3 network architecture, and use the CBAM module to process the obtained feature map F to obtain the enhanced feature map F″;

[0019] Step 2.3: Expand and concatenate the enhanced feature map F″ through the improved ASPP module to generate a feature map of multi-scale information.

[0020] Furthermore, in step 2.1, the improved ResNet50 feature extraction network includes a first convolution layer and a maximum pooling layer, the average pooling layer and the fully connected layer are removed, and the step size of the first convolution layer and the maximum pooling layer is adjusted from 2 to 1.

[0021] Furthermore, in step 2.2, the CBAM module is located between the improved ResNet50 and the improved ASPP module, and the CBAM module includes a channel attention module and a spatial attention module, and the specific process is as follows:

[0022] Step 2.2.1. Compress the input feature map F of size H×W×C to 1×1×C in the channel attention module, and then transform the feature map F after average pooling and maximum pooling through two fully connected layers. The first fully connected layer reduces the number of channels C to 1 / 16 of the original number, and then restores the number of channels C to the original number through the second fully connected layer.

[0023] Step 2.2.2, add the average pooling and maximum pooling results of the two fully connected layers respectively, and obtain the channel attention weight through the sigmoid activation function, and multiply the channel attention weight with the feature map F to obtain the feature map F′. The calculation formula is as follows:

[0024] Mc(F)=σ(MLp(Avgpool(F)) + MLp(Maxpool(F)) (2);

[0025]

[0026] Where H is the height, W is the width, C is the number of channels, σ is the Sigmoid function, MLP is used to generate channel attention weights, Mc(F) is the channel attention weight, and F' is the result of multiplying the feature map F by the channel attention weight;

[0027] Step 2.2.3: The spatial attention module performs average pooling and maximum pooling on the channel dimension on the input feature map F′ to generate two single-channel feature maps of size H×W×1. The two single-channel feature maps are then connected along the channel dimension to form a dual-channel feature map of size H×W×2. The dual-channel feature map is then convolved with a 7×7 convolution to fuse the spatial information. Finally, the spatial attention weight is obtained through the sigmoid activation function, and the spatial attention weight is multiplied by the feature map F′ to obtain the feature map F″. The calculation formula is as follows:

[0028] Ms (F′) = σ(f 7×7 ([(Avgpool(F));(Maxpool(F))])) (4);

[0029]

[0030] Among them, f 7×7represents a convolution operation with a convolution kernel of 7×7, Ms(F′) is the spatial attention weight, and F″ is the result of multiplying F′ by the spatial attention weight.

[0031] Furthermore, in step 2.3, 3×3 convolutions with dilation rates of 24 and 30 are added to the 3×3 convolutions with dilation rates of (6, 12, 18) of the ASPP module, thereby obtaining an improved ASPP module with dilation rates of (6, 12, 18, 24, 30).

[0032] Furthermore, in step 3, the data-enhanced training data is input into the established improved DeepLabV3 semantic segmentation network model for multiple cycles of training, and L2 regularization is introduced to prevent the model from overfitting.

[0033] Furthermore, in step 3, the verification data is used to evaluate the performance of the improved DeepLabV3 semantic segmentation network model during the training process, and the hyperparameters are adjusted according to the evaluation results to help obtain the target improved DeepLabV3 semantic segmentation network model.

[0034] Furthermore, the class weights are adjusted during the training process of multiple cycles, using the following formula:

[0035]

[0036] Among them, Prob(j) is the probability of the jth category occurring in the training data; Weight(i) is the weighted result of Prob(i); the constant 1.02 is used for smoothing to prevent the logarithmic function from producing infinite values ​​when Prob(j) is close to zero.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The present invention provides a method for semantic segmentation of urban street scenes based on improved DeepLabV3, which includes improving the DeepLabV3 semantic segmentation network model, wherein the model selects ResNet50 as a feature extraction network, sets a smaller step size, removes the last average pooling layer and the fully connected layer, and retains the main convolutional layer for feature extraction to obtain a feature map with higher resolution and more spatial information, thereby ensuring computational efficiency and significantly improving the precision of feature extraction; integrating a CBAM module after the feature extraction network to enhance the feature representation in the channel and spatial dimensions, improve the recognition ability of semantic information, and solve the problem that the existing semantic segmentation method based on DeepLabV3 is difficult to accurately segment objects of different sizes and shapes at the same time; and gradually adjusting the expansion rate of the ASPP module from (6, 12, 18) to (6, 12, 18, 24, 30) to enable it to capture more contextual information while compensating for details that may be ignored in the feature extraction process, thereby enhancing the feature extraction ability and segmentation accuracy of the model.

[0039] (2) The method for semantic segmentation of urban street scenes based on improved DeepLabV3 provided by the present invention improves the mean intersection over union (mIOU) on Cityscapes test data from 81.3% to 82.8%, and also reaches 81.6% mIOU on GTA5 verification data. The experimental results show that the present invention not only enhances the model's focus on important features, but also significantly improves the segmentation accuracy of detailed information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are incorporated in and constitute a part of this specification and, together with the description, serve to explain the principles of the present invention.

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 An improved DeepLabV3 semantic segmentation network model for the urban street scene semantic segmentation method of the present invention;

[0043] Figure 2 The present invention improves the channel attention module of the CBAM module of the DeepLabV3 semantic segmentation network model;

[0044] Figure 3 The spatial attention module of the CBAM module of the DeepLabV3 semantic segmentation network model improved by the present invention;

[0045] Figure 4 The improved ASPP module of the present invention has an expansion ratio of (6, 12, 18, 24, 30);

[0046] Figure 5 The segmentation results of the ASPP module with different expansion rates on the Cityscapes validation data;

[0047] in, Figure 5 (a) is Cityscapes test data;

[0048] Figure 5 (b) ASPP module with expansion ratio (6, 12, 18) Figure 5 (a) The segmentation result on the graph;

[0049] Figure 5 (c) ASPP modules with expansion ratios (6, 12, 18, 24) Figure 5 (a) The segmentation result on the graph;

[0050] Figure 5 (d) ASPP modules with expansion rates (6, 12, 18, 24, 30) Figure 5 (a) The segmentation result on the graph;

[0051] Figure 6 (a) is Cityscapes test data;

[0052] Figure 6 (b) is the semantic segmentation method of the present invention. Figure 6 (a) The segmentation result on the graph;

[0053] Figure 7 (a) is the GTA5 test data;

[0054] Figure 7 (b) is the semantic segmentation method of the present invention. Figure 7 Segmentation results on (a). DETAILED DESCRIPTION

[0055] Here, exemplary embodiments will be described in detail, and the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples consistent with some aspects of the present invention described in detail in the appended claims.

[0056] See also Figures 1 to 7 The embodiment of the present invention provides a method for semantic segmentation of urban street scenes based on improved DeepLabV3, comprising the following steps:

[0057] Step 1: Obtain a city street view image dataset, including training data, verification data, and test data, and perform data enhancement on the training data;

[0058] Step 2: Establish and improve the DeepLabV3 semantic segmentation network model;

[0059] Step 3: Use the training data obtained in step 1 to train the improved DeepLabV3 semantic segmentation network model, and use the verification data to evaluate and adjust the model during the training process to obtain the target improved DeepLabV3 semantic segmentation network model;

[0060] Step 4: Input the test data obtained in step 1 into the target improved DeepLabV3 semantic segmentation network model in step 3 to obtain the semantic segmentation result.

[0061] Furthermore, in step 1, the city street view image dataset uses Cityscapes and GTA5 datasets, and randomly divides the city street view image dataset into training data, verification data, and test data according to the ratio of 1.9-2.1:0.9-1.1:0.9-1.1.

[0062] Specifically, Cityscapes is a city street dataset widely used in computer vision tasks. The GTA5 dataset consists of 24,966 city street images. These images are rendered from the game Grand Theft Auto V, and all images are captured from a car's perspective on virtual city streets. In this embodiment, the city street view image dataset includes 12,403 training images (training data), 6382 verification images (verification data), and 6181 test images (test data). 19 semantic categories correspond to the labels in the Cityscapes dataset.

[0063] In order to enhance the generalization ability and robustness of the improved DeepLabV3 semantic segmentation network model, in step 1, the data enhancement is to use average fusion to randomly crop the training data and randomly flip the image horizontally, which is helpful for multi-scale reasoning. The calculation formula of the average fusion is as follows:

[0064]

[0065] Among them, P final is the final prediction result, P i is the prediction result of the i-th scale or flipped image, and N is the total number of prediction results. The final prediction result is obtained by aggregating predictions from different scales and flipped versions and calculating the average.

[0066] Furthermore, in step 2, establishing an improved DeepLabV3 semantic segmentation network model includes the following steps:

[0067] Step 2.1: Input the city street view image into the improved ResNet50 feature extraction network to obtain feature maps F of different resolutions;

[0068] Step 2.2, integrate the CBAM module into the DeepLabV3 network architecture, and use the CBAM module to process the feature maps F of different resolutions to obtain the enhanced feature map F″;

[0069] Step 2.3: Expand and concatenate the enhanced feature map F″ through the improved ASPP module to generate a feature map of multi-scale information.

[0070] It should be noted that in the feature extraction network, the step size setting is crucial for performing high-precision segmentation tasks in complex urban environments. Smaller step sizes produce larger feature maps, retaining more spatial details, but also require more computing power and memory. Larger step sizes reduce the overlap of feature maps, but may lose some spatial information. ResNet50 is selected as the backbone network for feature extraction. Compared with shallower networks, it can capture more complex features; compared with deeper networks, it is more efficient in computing resources and training time, and requires fewer model parameters and storage space. The present invention improves the step size of the ResNet50 feature extraction network and optimizes the network space.

[0071] Furthermore, in step 2.1, the improved ResNet50 feature extraction network includes the first convolution layer and the maximum pooling layer, removes the average pooling layer and the fully connected layer, and the step size of the first convolution layer and the maximum pooling layer is adjusted from 2 to 1, and the step size of other layers remains unchanged, thereby setting the step size of the entire feature extraction network to 8. The improved ResNet50 feature extraction network retains the main convolution layer of the feature extraction network, which retains more spatial details while balancing computing resources.

[0072] like Figure 2 and Figure 3 As shown, the CBAM module is located between the improved ResNet50 and the improved ASPP module. The CBAM module includes a channel attention module and a spatial attention module. The specific process is as follows:

[0073] Step 2.2.1. Compress the input feature map F of size H×W×C to 1×1×C in the channel attention module, and then transform the feature map F after average pooling and maximum pooling through two fully connected layers. The first fully connected layer reduces the number of channels C to 1 / 16 of the original number, and then restores the number of channels C to the original number through the second fully connected layer.

[0074] Step 2.2.2, add the average pooling and maximum pooling results of the two fully connected layers respectively, and obtain the channel attention weight through the sigmoid activation function, and multiply the channel attention weight with the feature map F to enhance the important feature channels, and finally obtain the feature map F′. The calculation formula is as follows:

[0075] Mc(F)=σ(MLp(Avgpool(F)) + MLp(Maxpool(F)) (2);

[0076]

[0077] Where H is the height, W is the width, C is the number of channels, σ is the Sigmoid function, MLP is used to generate channel attention weights, Mc(F) is the channel attention weight, and F' is the result of multiplying the feature map F by the channel attention weight;

[0078] Step 2.2.3: The spatial attention module performs average pooling and maximum pooling on the channel dimension on the input feature map F′ to generate two single-channel feature maps of size H×W×1. The two single-channel feature maps are then connected along the channel dimension to form a dual-channel feature map of size H×W×2. The dual-channel feature map is then convolved with a 7×7 convolution to fuse the spatial information. Finally, the spatial attention weight is obtained through the sigmoid activation function, and the spatial attention weight is multiplied by the feature map F′ to enhance the important spatial area. Finally, the feature map F″ is obtained. The calculation formula is as follows:

[0079] Ms (F′) = σ(f 7×7 ([(Avgpool(F));(Maxpool(F))])) (4);

[0080]

[0081] Among them, f 7×7 represents a convolution operation with a convolution kernel of 7×7, Ms(F′) is the spatial attention weight, and F″ is the result of multiplying F′ by the spatial attention weight.

[0082] It should be noted that as a lightweight attention module CBAM (Convolutional Attention Module), it can be inserted into the existing DeepLabV3 network model as needed. The CBAM module enhances the feature representation of the DeepLabV3 network model through its channel attention and spatial attention modules, enabling the DeepLabV3 network model to pay more attention to important features and suppress unimportant information. At the same time, the CBAM module helps the DeepLabV3 network model to pay attention to features of different scales, which is very useful for processing images with multi-scale objects.

[0083] The channel attention module effectively uses global spatial information to adjust the importance of each channel. This approach enables the DeepLabV3 network model to better focus on key feature channels. Specifically, the channel attention module applies adaptive average pooling and adaptive maximum pooling to each channel independently. The spatial attention module effectively uses global information in the channel dimension to adjust the importance of each spatial position, enabling the DeepLabV3 network model to better focus on key spatial areas. This spatial attention module is particularly suitable for tasks that require fine spatial information.

[0084] Considering the long-range dependencies in urban street scenes and potential features that may be ignored during feature extraction, the present invention increases the dilation rate in the ASPP module to capture greater context information without increasing the number of parameters and computational cost. This multi-scale feature extraction is very useful for capturing global context information while maintaining high-resolution features, enabling the model to better adapt to complex scenes with objects of different sizes and shapes.

[0085] like Figure 4 As shown, further, in step 2.3, 3×3 convolutions with dilation rates of 24 and 30 are added to the 3×3 convolutions with dilation rates of (6, 12, 18) of the ASPP module, thereby obtaining an improved ASPP module with dilation rates of (6, 12, 18, 24, 30), which can extract more features of different scales.

[0086] Specifically, the feature map F″ is subjected to channel adjustment through a 1×1 convolution to reduce the channel dimension, and then subjected to the above-mentioned 3×3 convolution with the expansion rate of (6, 12, 18, 24, 30). The feature map of the last layer (i.e., the feature map output when the expansion rate is 30) is subjected to a pooling operation, and the feature map after convolution and pooling is further concatenated in the channel dimension to form a feature map that integrates multi-scale information, generating a more compact feature representation, and finally a 1×1 convolution layer is used for fusion and the feature map is output as the segmentation result.

[0087] Furthermore, in step 3, the data-enhanced training data is input into the established improved DeepLabV3 semantic segmentation network model for multiple cycles of training, and L2 regularization is introduced to prevent model overfitting and improve training efficiency.

[0088] Furthermore, the verification data is used to evaluate the performance of the improved DeepLabV3 semantic segmentation network model during the training process, and the hyperparameters are adjusted according to the evaluation results to help obtain the target improved DeepLabV3 semantic segmentation network model.

[0089] The specific steps to obtain the target improved DeepLabV3 semantic segmentation network model are as follows:

[0090] 1. After each training cycle, use the validation data to evaluate the performance of the model, including calculating the accuracy of the model, loss function value and other indicators;

[0091] 2. Manually adjust the model’s external configuration variables (called hyperparameters) based on the performance of the validation data. The settings of hyperparameters, such as learning rate, number of iterations, etc., have a significant impact on the model’s training process and final performance;

[0092] 3. After multiple training and validation, select the model with the best performance on the validation data, that is, the target improved DeepLabV3 semantic segmentation network model. This model will be used for the final test data evaluation to determine its generalization ability on unknown data.

[0093] In addition, the model error is obtained through validation data, and the error inflection point is identified to avoid model overfitting. It should be noted that preventing model overfitting here is different from introducing L2 regularization during training to prevent model overfitting.

[0094] Among them, increasing validation data: evaluating the performance of the classifier by increasing the amount of data and test data is a direct means to avoid overfitting. This method provides more data to train the model so that the model can be verified on more data, thereby reducing the over-adaptation of the model to specific training data and improving the generalization ability of the model. Increasing data can help the model learn more changes and patterns, reduce the model's dependence on noise or outliers, and make the model more robust; L2 regularization is a method that is implemented by adding the sum of the squares of parameter weights to the loss function. L2 regularization prevents overfitting by limiting the size of model parameters. It penalizes the weights during the optimization process by adding a regularization term (usually the sum of the squares of the weight parameters) to the loss function, forcing the weights to be close to 0 but not completely 0. This can prevent the weights of certain features from being too large, avoid over-adaptation of the model to the training data, and keep the complexity of the model within a reasonable range, thereby improving the generalization ability of the model. L2 regularization tends to select weight parameters with very small values, mainly used to prevent model overfitting.

[0095] From this we can see that adding data is to improve the generalization ability of the model from the data level, while L2 regularization is to prevent overfitting by adjusting model parameters from the perspective of model optimization.

[0096] To summarize, the training results are compared with the verification data, and the optimal model on the verification data is selected by judging the error size.

[0097] Specifically, in this embodiment, training was performed on the AutoDL cloud server using an RTX 4090D (24GB) GPU. The improved DeepLabV3 semantic segmentation network model was trained for a total of 200 cycles. The initial learning rate was set to 0.001, and the learning rate scheduler was used to dynamically adjust it. If the loss did not improve significantly within 10 cycles, the learning rate was reduced by 20%. This approach ensures that appropriate learning rates are used at different stages of training and improves the convergence speed of the model. In addition, L2 regularization is also applied to prevent model overfitting and improve training efficiency.

[0098] In semantic segmentation, due to imbalanced training data, some categories may be difficult to segment accurately. In order to solve this problem in model training, we further adjust the category weights during the 200-cycle training process, using the following formula:

[0099]

[0100] Among them, Prob(j) is the probability of the jth category in the training data; Weight(i) is the weighted result of Prob(i); the constant 1.02 is used for smoothing to prevent the logarithmic function from producing infinite values ​​when Prob(j) is close to zero. The purpose of the above weighted formula is to reduce the weight of frequently occurring categories and increase the weight of rare categories, so as to pay more attention to those difficult to segment categories during training.

[0101] In order to illustrate the efficacy of the present invention, the present invention has conducted the following experimental evaluation:

[0102] 1) The impact of ResNet50 feature extraction networks with different step sizes on mIOU

[0103] Experiments were conducted on the Cityscapes dataset to explore the effect of step size on segmentation accuracy. Table 1 shows the mIOU when the step size is 32, 16, and 8. It can be seen that a smaller step size can achieve higher segmentation accuracy.

[0104] Table 1 The impact of different step lengths (OS) on mIOU in the Cityscapes dataset

[0105]

[0106] 2) The impact of different expansion rates on mIOU

[0107] The step size of the improved ResNet50 feature extraction network is 8. By appropriately increasing the expansion rate, it is further found that the model's ability to extract multi-scale features is improved without adding additional parameters or computing resources. Figure 5 As shown in Table 2, with the increase of ASPP expansion rate, the segmentation accuracy of the model for boundaries and small objects is significantly improved. The numerical results are shown in Table 2:

[0108] Table 2 Effect of different expansion rates on mIOU on Cityscapes validation data

[0109]

[0110]

[0111] It can be seen from Table 2 that the improved ASPP module with expansion ratios of (6, 12, 18, 24, 30) has the best segmentation accuracy.

[0112] 3) Improve the CBAM module integrated in the DeepLabV3 semantic segmentation network model, and its influence on mIOU at different positions

[0113] (1) Inserting the CBAM module in the middle layer of the improved ResNet50 can enhance the feature representation layer by layer, making the feature representation of each layer more prominent and enhancing the overall recognition ability of the network.

[0114] (2) Adding the CBAM module after the ASPP module can enhance the features extracted through multi-scale features, making the final features more representative.

[0115] (3) Adding the CBAM module after the improved ResNet50 and before the improved ASPP module can enhance the feature representation before multi-scale feature extraction, making the features processed by the ASPP module more distinct and important. As shown in Table 3:

[0116] Table 3. The impact of inserting CBAM modules at different positions on mIOU

[0117]

[0118] As shown in Table 3, method (3) shows the best mIOU result. This is because the last layer of the improved ResNet50 feature extraction network usually extracts global context features, which may contain a lot of redundant information. Adding the CBAM module here can effectively highlight key features and suppress irrelevant information, and also ensure that the improved ASPP module can better identify and utilize important information when processing these features, thereby improving the fusion of multi-scale features.

[0119] 4) Comparison results of different semantic segmentation network models on Cityscapes test data

[0120] The improved DeepLabV3 semantic segmentation network model of the present invention is compared with other excellent models. The comparison results are shown in Table 4. It can be seen that the method of the present invention has achieved the most advanced results.

[0121] Table 4 Comparison results of different models on CityScapes test data

[0122]

[0123] Finally, the 19 categories are divided into 7 categories to represent higher-level and broader categories, and the proposed model achieves 81.6% mIOU on the GTA5 validation data, as shown in Table 5:

[0124] Table 5 Results of different categories on GTA5 validation data

[0125]

[0126] Figure 6is the segmentation result of the semantic segmentation method of the present invention on the Cityscapes test data, Figure 7 is the result of the semantic segmentation method of the present invention on the GTA5 test data, Figure 6 , Figure 7 It can be seen that this method not only enhances the model's attention to important features, but also significantly improves the segmentation accuracy of detail information.

[0127] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention.

[0128] It should be understood that the present invention is not limited to what has been described above and that various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A semantic segmentation method for urban street scenes based on improved DeepLabV3, It is characterized in that The following steps are involved: Step 1: Obtain a city street view image dataset, including training data, verification data, and test data, and perform data enhancement on the training data; Step 2: Establish and improve the DeepLabV3 semantic segmentation network model; Step 3: Use the training data obtained in step 1 to train the improved DeepLabV3 semantic segmentation network model, and use the verification data to evaluate and adjust the model during the training process to obtain the target improved DeepLabV3 semantic segmentation network model; Step 4: Input the test data obtained in step 1 into the target improved DeepLabV3 semantic segmentation network model in step 3 to obtain the semantic segmentation result.

2. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 1, It is characterized in that In step 1, the city street view image dataset uses the Cityscapes dataset and / or the GTA5 dataset, and randomly divides the city street view image dataset into training data, verification data, and test data according to a ratio of 1.9-2.1:0.9-1.1:0.9-1.

1.

3. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 1, It is characterized in that In step 1, the data enhancement is to randomly crop and randomly horizontally flip the training data using average fusion. The calculation formula of the average fusion is as follows: Among them, P final is the final prediction result, P i is the prediction result of the i-th scale or flipped image, and N is the total number of prediction results.

4. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 1, It is characterized in that In step 2, building an improved DeepLabV3 semantic segmentation network model includes the following steps: Step 2.1: Input the city street view image into the improved ResNet50 feature extraction network to obtain feature maps F of different resolutions; Step 2.2, integrate the CBAM module into the DeepLabV3 network architecture, and use the CBAM module to process the obtained feature map F to obtain the enhanced feature map F″; Step 2.3: Expand and concatenate the enhanced feature map F″ through the improved ASPP module to generate a feature map of multi-scale information.

5. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 4, It is characterized in that In step 2.1, the improved ResNet50 feature extraction network includes the first convolution layer and the maximum pooling layer, the average pooling layer and the fully connected layer are removed, and the step size of the first convolution layer and the maximum pooling layer is adjusted from 2 to 1.

6. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 4, It is characterized in that In step 2.2, the CBAM module is located between the improved ResNet50 and the improved ASPP module. The CBAM module includes a channel attention module and a spatial attention module. The specific process is as follows: Step 2.2.1: Compress the input feature map F of size H×W×C to 1×1×C in the channel attention module, and then transform the feature map F after average pooling and maximum pooling through two fully connected layers; Step 2.2.2, add the average pooling and maximum pooling results of the two fully connected layers respectively, and obtain the channel attention weight through the sigmoid activation function, and multiply the channel attention weight with the feature map F to obtain the feature map F′. The calculation formula is as follows: Mc(F)=σ(MLp(Avgpool(F)) + MLp(Maxpool(F)) (2); Where H is the height, W is the width, C is the number of channels, σ is the Sigmoid function, MLP is used to generate channel attention weights, Mc(F) is the channel attention weight, and F' is the result of multiplying the feature map F by the channel attention weight; Step 2.2.3: The spatial attention module performs average pooling and maximum pooling on the channel dimension on the input feature map F′ to generate two single-channel feature maps of size H×W×1. The two single-channel feature maps are then connected along the channel dimension to form a dual-channel feature map of size H×W×2. The dual-channel feature map is then convolved with a 7×7 convolution to fuse the spatial information. Finally, the spatial attention weight is obtained through the sigmoid activation function, and the spatial attention weight is multiplied by the feature map F′ to obtain the feature map F″. The calculation formula is as follows: Ms (F′) = σ(f 7×7 ([(Avgpool(F));(Maxpool(F))])) (4); Among them, f 7×7 represents a convolution operation with a convolution kernel of 7×7, Ms(F′) is the spatial attention weight, and F″ is the result of multiplying F′ by the spatial attention weight.

7. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 4, It is characterized in that In step 2.3, 3×3 convolutions with dilation rates of 24 and 30 are added to the 3×3 convolutions with dilation rates of (6, 12, 18) of the ASPP module, resulting in an improved ASPP module with dilation rates of (6, 12, 18, 24, 30).

8. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 1, It is characterized in that In step 3, the data-enhanced training data is input into the established improved DeepLabV3 semantic segmentation network model for multiple cycles of training, and L2 regularization is introduced to prevent model overfitting.

9. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 8, It is characterized in that In step 3, the verification data is used to evaluate the performance of the improved DeepLabV3 semantic segmentation network model during the training process, and the hyperparameters are adjusted according to the evaluation results to help obtain the target improved DeepLabV3 semantic segmentation network model.

10. The urban street scene semantic segmentation method based on improved DeepLabV3 according to claim 9, It is characterized in that The class weights are adjusted during multiple cycles of training using the following formula: Among them, Prob(j) is the probability of the jth category occurring in the training data; Weight(i) is the weighted result of Prob(i); the constant 1.02 is used for smoothing to prevent the logarithmic function from producing infinite values ​​when Prob(j) is close to zero.

Images