Real-time semantic segmentation method for urban street scene images based on attention boundary enhancement and aggregation pyramid

Abstract

This paper proposes a real-time semantic segmentation method for urban street scene images based on attention boundary enhancement and aggregation pyramids. The implementation steps are as follows: obtaining a training sample set and a test sample set; constructing a real-time semantic segmentation network model for urban street scenes based on attention boundary enhancement and aggregation pyramids and iteratively training it; and obtaining real-time semantic segmentation results for urban street scene images. This method enhances boundary sensitivity through an attention boundary enhancement subnetwork and integrates contextual information using coarse-to-fine feature perception, improving the robustness of semantic information. Simultaneously, an aggregation pyramid pooling network performs feature splicing on pooled feature maps of different scales, generating correct semantic labels. Finally, the global feature information map, detail feature information map, and boundary feature information map of each sample output by the three networks are fused to improve the recognition accuracy of semantically segmented images.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation and computer vision technology, and relates to a semantic segmentation method, specifically to a real-time semantic segmentation method for urban street scene images based on attention boundary enhancement and aggregation pyramid, which can be used in the fields of autonomous driving, smart cities, intelligent robots, etc. Background Art

[0002] Real-time semantic segmentation of visual images faces many challenges in complex scenes, especially when dealing with small or slender objects. These issues not only affect the accuracy of segmentation, but also subsequent computer vision tasks such as scene understanding and augmented reality. Real-time semantic segmentation methods for visual images are usually based on UNet networks or bilateral networks, using pixel-level classification to achieve image segmentation. However, these methods are prone to inaccurate segmentation or poor boundary processing, especially in complex scenes, where semantic label confusion is prone to occur. In recent years, semantic segmentation methods based on deep learning have widely introduced context-awareness and multi-feature fusion technologies. By processing different feature components of an image separately, deep learning models can better capture local details and global information in the image and assign semantic labels. However, when dealing with small or slender objects in complex scenes, these methods often ignore these objects or misidentify similar objects in the background.

[0003] To overcome these shortcomings, for example, in a patent application with publication number CN118334335A, entitled "A Real-time Semantic Segmentation Method for Urban Street Scene Images Based on Dual Interactive Fusion," street scene images are trained and inferred using a UNet-like network. The encoder consists of a Vanilla Conv and an STDC module, downsampling each stage through a module with a stride of 2. The highest-level contextual encoding information is then passed to the integrated perception module, where the receptive field of the model is improved through dilated convolution. The fusion module fuses adjacent, different-scale information in the encoder. The cross-attention module, CAM, cross-weightedly fuses the encoding results of the integrated perception module with the information fusion results to achieve decoding and upsampling of the contextual information. After three levels of decoding, the feature map is upsampled to the original image size and the segmentation result is predicted. The disadvantage of this method is that the model lacks sensitivity to object boundary information, resulting in inference results that are easily blurred at the segmentation boundary, affecting segmentation accuracy and visual quality. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a real-time semantic segmentation method for urban street scene images based on attention boundary enhancement and aggregation pyramid to solve the technical problem of low segmentation accuracy in the prior art.

[0005] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0006] (1) Obtain training sample set and test sample set:

[0007] Preprocess M city street scene RGB images with semantic labels, which contain S categories, randomly selected from the city street scene image dataset, and use the preprocessed M1 city street scene images as the training sample set, and use the remaining preprocessed M2 street scene images as the test sample set, where S ≥ 2, M ≥ 4000, M2=M-M1;

[0008] (2) Constructing a real-time semantic segmentation network model P for urban street scenes based on attention boundary enhancement and aggregation pyramid:

[0009] Construct a real-time semantic segmentation network model P for urban street scenes, including a pre-processing module and a multi-scale feature segmentation network and an aggregated pyramid pooling network cascaded with them as a main network, a detail enhancement sub-network and an attention boundary enhancement sub-network cascaded with the outputs of the pre-processing module and the multi-scale feature segmentation network as branch networks, and an adder cascaded with the outputs of the main network and the two branch networks;

[0010] (3) Iteratively train the real-time semantic segmentation network model for urban street scenes:

[0011] The urban street scene real-time semantic segmentation network model P is iteratively trained through the training sample set to obtain the trained semantic segmentation network model P * ;

[0012] (4) Obtaining real-time semantic segmentation results of urban street view images:

[0013] The test sample set is used as the trained semantic segmentation network model P * The input is used for forward reasoning to obtain the semantic segmentation results corresponding to M2 test samples.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] 1. In the process of iteratively training the semantic segmentation network model and obtaining the semantic segmentation results, the backbone network, detail enhancement sub-network and attention boundary enhancement sub-network respectively perform multi-scale feature extraction and feature aggregation, detail enhancement extraction and boundary enhancement extraction on each downsampled image, and fuse the global feature information map, detail feature information map and boundary feature information map of each sample output by the three networks through an adder, thereby improving the recognition accuracy of the semantic segmentation image.

[0016] 2. The present invention uses the attention boundary enhancement sub-network to perform boundary enhancement on the downsampled image output by the pre-processing module and the feature map output by the multi-scale feature segmentation network. It can integrate the complementary features of the multi-scale segmentation network, and at the same time enhance the sensitivity to boundaries through spatial integrity and channel saliency, and use coarse-to-fine feature perception to integrate contextual information, thereby improving the robustness of semantic information, and finally obtaining a clearer boundary feature information map in the boundary area, thereby improving the accuracy of the semantic segmentation image.

[0017] 3. The present invention uses the multi-scale feature segmentation network in the backbone network to extract features of different scales for each downsampled feature map, and uses the aggregated pyramid pooling network to pool the pooled feature maps of different scales at different scales, and then performs feature splicing on the pooled feature maps to obtain a global feature information map with global context information, so that the network generates correct semantic labels and enhances the robustness of the model in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flowchart for the implementation of the present invention;

[0019] Figure 2 Schematic diagram of the structure of the semantic segmentation network model of the present invention;

[0020] Figure 3 Schematic diagram of the structure of the aggregate pyramid pooling module of the present invention;

[0021] Figure 4 This is a schematic diagram of the structure of the attention enhancement perception module of the present invention;

[0022] Figure 5 This is a schematic diagram of the structure of the attention perception refinement module of the present invention;

[0023] Figure 6 This is a simulation comparison diagram of the segmentation effect of the present invention and the prior art. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Reference Figure 1 , the present invention includes the following steps.

[0026] Step 1. Build the basic dataset and test dataset:

[0027] M city street scene RGB images with semantic labels and containing S categories randomly selected from the city street scene image dataset are preprocessed. Specifically, each city street scene image with semantic labels is normalized, and the pixel values ​​of the image are normalized from the integer range of 0-255 to the floating point range of 0-1. The RandomHorizontalFlip method is applied to the normalized image. The model can learn more feature transformations, thereby reducing dependence on specific directions and avoiding bias during training. In addition, random flipping can be regarded as a regularization method to help prevent model overfitting and improve performance on unseen data. The probability of random flipping is set to This ensures that the effect of data augmentation will not be biased towards a certain type of sample distribution, and the preprocessed M1 city street view images are used as the training sample set, and the remaining preprocessed M2 street view images are used as the test sample set, where S = 19, M = 5000, M1 = 3000, and M2 = 2000.

[0028] Step 2. Build a real-time semantic segmentation network model P for urban street scenes based on attention boundary enhancement and aggregation pyramid:

[0029] Construct a real-time semantic segmentation network model P for urban street scenes, including a pre-processing module and a multi-scale feature segmentation network and an aggregated pyramid pooling network cascaded with them as a main network, a detail enhancement sub-network and an attention boundary enhancement sub-network cascaded with the outputs of the pre-processing module and the multi-scale feature segmentation network as branch networks, and an adder cascaded with the outputs of the main network and the two branch networks;

[0030] The pre-processing module includes cascaded convolutional layers and BasicBlock layers;

[0031] The backbone network includes a multi-scale feature segmentation network consisting of N cascaded feature extraction modules of different scales, N ≥ 3; the aggregate pyramid pooling network includes parallel feature extraction branches and aggregation modules;

[0032] A detail enhancement sub-network, comprising N cascaded residual modules, wherein, except for the first feature extraction module and the first residual module, an input end of each residual module is connected to an input end of a corresponding feature extraction module;

[0033] The attention boundary enhancement subnetwork includes a cascade of N-1 perceptual boundary refinement modules, and the input of the nth residual module is connected to the input of the n-1th perceptual boundary refinement module;

[0034] The segmentation head prediction network includes stacked convolutional layers, BN layers, ReLU activation functions, and convolutional layers;

[0035] In the implementation of this case, the pre-processing module is connected in sequence: the first convolutional layer → the second convolutional layer → the first BasicBlock layer → the second BasicBlock layer → the ReLU activation layer → the third BasicBlock layer → the fourth BasicBlock layer and the ReLU activation layer. The pre-processing module quickly downsamples each training sample to make the image size become the original image size.

[0036] The multi-scale feature segmentation network in the backbone network consists of three cascaded feature extraction modules of different scales, with the scales being Each feature extraction module consists of a residual block;

[0037] The aggregated pyramid pooling network includes 6 parallel feature extraction branches and aggregation modules. The first and sixth branches contain only one convolution layer, while the second to fifth branches are composed of a convolution layer and an upsampling layer respectively; the feature fusion module is divided into three submodules: in the first submodule, the output of branch one is added point by point with the outputs of branch two and branch three, and the results are added point by point again for feature splicing, and the results are processed by the sixth convolution layer. In the second submodule, the outputs of branch three and branch four are feature spliced, and then added point by point with the output of the first submodule. In the third submodule, the output of the second submodule is processed by the seventh convolution layer, and then feature spliced ​​with the second submodule, and then processed by the eighth convolution layer, and finally added point by point with the output of the sixth branch; this local parallel design solves the problem of poor real-time performance caused by high memory overhead and frequent read and write operations, so as to achieve lower inference latency;

[0038] The detail enhancement sub-network consists of three sequentially connected residual modules, where the input layer of the first residual module is the output of the pre-processing module, the input layer of the second residual module is the output of the first residual module and the output of the first feature extraction module in the backbone network, and the input layer of the third residual module is the output of the second residual module and the output of the second feature extraction module.

[0039] The attention boundary enhancement subnetwork consists of two attention-aware boundary refinement modules connected in sequence. The input of the first attention-aware boundary refinement module is the output of the pre-processing module and the output of the first feature extraction module in the backbone network. The obtained output and the second feature extraction module enter the second attention-aware boundary refinement module to complete the enhancement of boundary information.

[0040] The attention perception boundary refinement module includes two sets of input layers, which are respectively fed into the attention enhancement perception module after passing through the convolutional layer and the Bottleneck layer. The attention enhancement perception module performs weighted fusion on the two sets of features according to the attention weights (W and 1-W). The final fusion result passes through a Bottleneck layer and is output as an optimized feature map. The attention enhancement perception module includes a channel attention mechanism module and a spatial attention mechanism module connected in sequence.

[0041] Among them, the attention enhancement perception module is composed of two parallel branches and a feature fusion feature module. Branch one is composed of a global maximum pooling layer and a convolutional dimension reduction layer connected in sequence. Branch two is composed of a global average pooling layer and a convolutional dimension reduction layer connected in sequence. The fusion feature module adds the outputs of branch one and branch two point by point and then passes through a sigmoid activation function. The output is then added point by point to the original input. The spatial attention mechanism module is composed of two parallel branches connected in sequence and a feature splicing module. Branch one is composed of a global maximum pooling layer connected in sequence, and branch two is composed of a global average pooling layer. The feature splicing module performs feature splicing on the outputs of branch one and branch two and passes through the first convolution layer. Then, the result obtained by the sigmoid activation function is added point by point to the output of the channel attention mechanism module, and then passes through the sigmoid activation function to generate a gradient-based matrix to assign weights to each pixel.

[0042] The adder performs feature fusion on each global feature information map and its corresponding detail feature information map and boundary feature information map;

[0043] The segmentation head prediction network includes the first convolution layer, BN layer, ReLU activation function, and second convolution layer connected in sequence. It predicts the category probability of each pixel point of each fused feature map and obtains the category probability p of each pixel point. s (x,y) and the predicted segmentation result image O.

[0044] Step 3. Iteratively train the city street scene real-time semantic segmentation network model:

[0045] (3a) The number of initial iterations is t, the maximum number of iterations is T, and the weight of the semantic segmentation network model at the tth iteration is w t , and let t = 1, T = 188500;

[0046] (3b) Each training sample in the training sample set is used as the input of the semantic segmentation network model P. The pre-processing module downsamples each training sample to obtain M1 downsampled images. Specifically, for each input image I with a size of H×W×3, the input image is first processed by two convolutions with H×W×C of 3×3×64, a step size of 2, and a padding of 1 to obtain a size of Feature map Secondly, two consecutive BasicBlock blocks with H×W×C of 3×3×64 are used in the ResNet network to capture the feature map The features of Then use the ReLU activation function to activate the feature map Get feature map Then the feature map is processed through a BasicBlock block with H×W×C of 3×3×128 and a step size of 2. Downsample and get Then capture it through a BasicBlock block with H×W×C of 3×3×128 respectively The feature of Feature map Finally, the ReLU activation function is used to activate the feature map Get feature map That is, the size of this stage is The final processing result I stem The formula is as follows:

[0047]

[0048] In the formula, Indicates the nth stage in the stem step, Conv n×n×q,s＝m,p＝l () represents the convolution kernel with a convolution size of n×n, a number of channels of q, a step size of s of m, and zero padding p of l, and ReLU() represents the activation function ReLU;

[0049] Directly using oversized visual images for deep neural network training will result in excessively large parameters and excessive training time, hindering the real-time performance of the network model. Oversized visual images will blur object boundaries and lead to poor segmentation accuracy. This operation converts large visual images into feature maps of appropriate size to achieve a balance between parameter size and segmentation accuracy.

[0050] (3c) The backbone network, detail enhancement sub-network, and attention boundary enhancement sub-network perform multi-scale feature extraction and feature aggregation, detail enhancement extraction, and boundary enhancement extraction on each downsampled image, respectively, to obtain the global feature information map, detail feature information map, and boundary feature information map corresponding to each training sample. Specifically, stem As the input of the backbone network I s , after the first feature extraction module in the multi-scale feature segmentation network s Perform a 3×3×256 convolution with a step size of 2 and a zero padding of 1, and then perform a 3×3×256 convolution. The result is then combined with I sAdd the results of the 3×3×256 convolution with a step size of 2 and zero padding of 1 to get Then Then perform two 3×3×256 convolutions, and the convolution results are the same as Add together to get at last Then perform two 3×3×256 convolutions, and After adding, the ReLU activation function is used to obtain the size Feature map The formula is as follows:

[0051]

[0052] After the second feature extraction module Perform a 3×3×512 convolution with a step size of 2 and a zero padding of 1, and then perform a 3×3×512 convolution. The result is then combined with Add the results of the convolution of 3×3×512 with a step size of 2 and zero padding of 1 to get Afterwards Then perform two 3×3×512 convolutions, and the convolution results are the same as Add together to get Finally, two 3×3×512 convolutions are performed, and After adding, the ReLU activation function is used to obtain the size Feature map The formula is as follows:

[0053]

[0054] After the third feature extraction module Perform 1×1×512 convolution to reduce the dimension, then perform 3×3×512 convolution with a step size of 2 and zero padding of 1, then perform 1×1×1024 convolution to increase the dimension, and then Add the results of the convolution of 3×3×1024 with a step size of 2 and zero padding of 1 to get Afterwards Perform 1×1×512 convolution to reduce the dimension, then perform 3×3×512 convolution to extract features, and then perform 1×1×1024 convolution to increase the dimension. The convolution result is the same as Adding up the size is Feature map The formula is as follows:

[0055]

[0056] After three feature extractions, the output enters the efficient aggregation pyramid pooling module to pass the pooled feature maps of different scales through the convolution layer and splice the results. The size is Feature map The four pooling layers are: 5×5 size, stride 2, zero padding 2; 9×9 size, stride 4, zero padding 4; 17×17 size, stride 8, zero padding 8; full size. These four results are all reduced in dimension by their own 1×1×96 convolution, and the results are recorded as The formula is as follows:

[0057]

[0058] Among them, Avg n×n,s＝m,p＝l () represents a pooling layer with a size of n×n, a step size s of m, and a zero-padding p of l, and GAvg() represents a global pooling layer;

[0059] Respectively and The results are added point by point, and then concatenated and fed into a 3×3×96 convolutional layer to extract features. The result is recorded as and After splicing with Add them point by point, and then perform 3×3×96 convolution to extract features. The result is recorded as The formula is as follows:

[0060]

[0061] Among them, Cat() represents feature splicing, Represents the point-by-point addition of feature maps;

[0062] The calculated and The final feature concatenation is performed, and then the 1×1×256 convolution is performed to increase the dimension. The result is the same as the input feature map. The results of the 1×1×256 convolution are added point by point to obtain the global feature information map I ppm , the formula is as follows:

[0063]

[0064] The detail enhancement sub-network obtains the detail feature information map through three cascaded residual modules. The first residual module converts I s As the original input, two 3×3×128 convolutions are performed to extract features, and then combined with I s The results of the 1×1×128 convolution are added together and the result is recorded as Then two 3×3×128 convolutions are performed to extract features, and then combined with The results of the 1×1×128 convolution are added together to get a size of Detailed features The formula is as follows:

[0065]

[0066] The output of the first residual module Scale feature map The output of the first feature extraction module Scale feature map As the input of the second residual module, Perform 1×1×128 convolution dimensionality reduction and then upsample by bilinear interpolation to obtain a size of of Will and Add point by point, then activate with Sigmoid function, and get the size The weight matrix W1 and the weight matrix 1-W1 of the same size; W1 and The product of point-by-point multiplication 1-W1 and The products of the point-by-point multiplications are added point by point, resulting in a size of of The formula is as follows:

[0067]

[0068] Among them, UpSample() represents bilinear interpolation upsampling, Represents point-by-point multiplication, σ() represents the activation function Sigmoid;

[0069] The second residual block output Scale feature map With the second feature extraction module Scale feature map As the input of the third residual module, Perform two 3×3×128 convolutions to extract features, and then combine them with I s The results of the 1×1×128 convolution are added together and the result is recorded as

[0070] Then two 3×3×128 convolutions are performed to extract features, and then The results of the 1×1×128 convolution are added together to get a size of Detailed features First, perform 1×1×128 convolution to reduce the dimension, and then perform bilinear interpolation upsampling to obtain a size of of Will and Add point by point, then activate with Sigmoid function, and get the size The weight matrix W2 and the weight matrix 1-W2 of the same size;

[0071] W2 and The product of point-by-point multiplication and 1-W2 The products of the point-by-point multiplications are added point by point, resulting in a size of of Then Perform two 3×3×128 convolutions to extract features, and then The results of the 1×1×128 convolution are added together and the result is recorded as Afterwards Perform 1×1×64 convolution, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase dimension. The result is the same as The results of the 1×1×128 convolution are added point by point to get a size of Detailed feature infographic The formula is as follows:

[0072]

[0073] The attention boundary enhancement sub-network obtains the boundary feature information map by cascading two attention-aware boundary refinement modules. The input of the first attention-aware boundary refinement module is the pre-processing module I. stem The output of the first feature extraction module Among them I stem Perform 1×1×64 convolution, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase the dimension. The result is the same as I stem The results of the 1×1×128 convolution are added point by point to obtain a size of Detailed features Perform 1×1×128 convolution dimensionality reduction and then upsample by bilinear interpolation to obtain a size of of The calculated and The result of point-by-point addition fusion is I e The attention mechanism is introduced from the channel and space through the attention-enhanced perception submodule;

[0074] I eAfter global average pooling and global maximum pooling, 1×1×64 convolution is performed for dimensionality reduction, 3×3×64 convolution is performed for feature extraction, and 1×1×128 convolution is performed for dimensionality increase. Compared with the results after global maximum pooling and triple convolution Add point by point, after Sigmoid activation function, and input I e Multiply point by point to get the result After the maximum pooling and average pooling, the results are spliced, and a 3×3×1 convolution is performed. After the Sigmoid activation function, the Multiply point by point to get the result After the Sigmoid activation function, the final result W is obtained e1 , the formula is as follows:

[0075]

[0076] Among them, GMP() represents global maximum pooling, GAP() represents global average pooling, MP() represents maximum pooling, and AP() represents average pooling;

[0077] Get the size The weight feature map W e1 and 1-W e1 After that, W e1 and Point-wise product sum 1-W e1 and The products of the point-by-point multiplications are added point by point to obtain Perform 1×1×64 convolution, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase dimension. The result is the same as The results of the 1×1×128 convolution are added point by point to obtain a size of Detailed features The formula is as follows:

[0078]

[0079] The input of the second attention-aware refinement module is the output of the first attention-aware refinement module The output of the second feature extraction module Perform 1×1×64 convolution to reduce the dimension, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase the dimension. The result is the same as The results of the 1×1×128 convolution are added point by point to obtain a size of Detailed features Perform 1×1×128 convolution dimensionality reduction and then upsample by bilinear interpolation to obtain a size of of The calculated and Point-by-point addition and fusion are performed, and the AEPM module analyzes the channel and space dimensions to obtain a size of The weight feature map W e2 and 1-W e2 ;

[0080] W e2 and Point-wise product sum 1-W e2 and The products of the point-by-point multiplications are added point by point, resulting in a size of of Perform 1×1×64 convolution, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase dimension. The result is the same as The results of the 1×1×128 convolution are added point by point to get a size of Detailed feature infographic The formula is as follows:

[0081]

[0082] (3d) The adder performs feature fusion on each global feature information map and its corresponding detail feature information map and boundary feature information map; the segmentation head prediction network performs category probability prediction on each pixel point of each fused feature map to obtain the category probability p of each pixel point. s (x, y) and the predicted segmentation result map O, specifically: the global feature map information I ppm , detail feature map information and boundary feature map information Point-by-point addition and fusion yield the result I o ', and after 3×3×128 convolution, BN batch normalization and ReLU activation function, 1×1×19 convolution is performed for final classification, resulting in a size of The segmentation result O is as follows:

[0083]

[0084] O=Conv 1×1×19 (ReLU(BN(Conv 3×3×128 (Io '))))

[0085] (3e) uses the cross entropy loss function and passes the true semantic label y of each category s (x,y) and class probability p s (x,y) calculates the loss value L of P and uses L to adjust the weight w t Update, specifically: calculate the loss value L, the calculation formula is:

[0086]

[0087] Where L(x,y) represents the cross entropy of each pixel, p s (x,y) is the probability that the model predicts that the pixel coordinates (x,y) belong to category s, y s (x, y) is the actual label of the pixel label, and H×W represents the size of the input image;

[0088] And by calculating the loss gradient, the weight w is completed t The update formula is:

[0089]

[0090] in is the loss value L to p s The partial derivative of (x,y), It is p s The partial derivative of (x,y) with respect to the weight, w t+1 It is w t The updated results, is the partial derivative of the loss value with respect to the weight, and η is the learning rate;

[0091] These gradients are back-propagated to each neural network layer and used to update the weights of each layer. The process is repeated for each layer until the gradients of all weights are calculated. Finally, the Adam optimizer is used to update the weights according to the gradients to obtain the semantic segmentation network model P for this iteration. t ;

[0092] (3f) Determine whether t=T is true. If so, obtain the trained semantic segmentation network model P. * Otherwise, let t = t + 1, P = P t , and execute step (3b);

[0093] Step 4. Obtain real-time semantic segmentation results of the city street view network:

[0094] The test sample set is used as the trained semantic segmentation network model P *The input is forward inferenced to obtain the images with pixel semantic annotations corresponding to M2 test samples as the semantic segmentation results.

[0095] The following is a simulation experiment to further illustrate the technical effects of the present invention:

[0096] 1. Simulation conditions and contents:

[0097] The hardware platform used in the simulation experiment is CPU Core TM i7-9700k, Nvidia RTX 2080Ti GPU with 11GB of video memory and 32GB of RAM, respectively; Python 3.8 and PyCharm 2019.3.3 x64 as the software platform, PyTorch 1.13.0 as the deep learning framework, and Ubuntu 18.04 as the operating system;

[0098] The Cityscapes dataset is considered one of the best high-resolution datasets, containing images of various street scenes in major European cities with a resolution of up to 1024×2048. It consists of 5,000 finely annotated images and another 20,000 coarsely annotated images using polygonal labels. In this study, only the finely annotated images were used for training; the diagram of the dataset is shown below. Figure 6 As shown in (a).

[0099] 2. Analysis of simulation results:

[0100] Figure 6 (b) and 6(c) are the segmentation effect demonstration diagrams of the prior art and the present invention, respectively. It can be clearly seen from the diagram that, compared with the prior art method, the segmentation result of the present invention has sharper object edges and clearer lines, and the outline of the cyclist's body and the road marking effect diagram are more distinct; the present invention shows more detailed information, the texture of the cyclist's clothes is more obvious, and the cracks on the ground and the outline of the curb are also more clearly visible; the segmentation accuracy of the present invention is significantly improved, especially when processing complex scenes, such as the cyclist's clothes are accurately classified as the "person" category, without color inconsistency or label confusion.

[0101] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A real-time semantic segmentation method for urban street scene images based on attention boundary enhancement and aggregation pyramid, characterized by: The steps include: (1) Obtain training sample set and test sample set: Preprocess M city street scene RGB images with semantic labels, which contain S categories, randomly selected from the city street scene image dataset, and use the preprocessed M1 city street scene images as the training sample set, and use the remaining preprocessed M2 street scene images as the test sample set, where S ≥ 2, M ≥ 4000, M2=M-M1; (2) Constructing a real-time semantic segmentation network model P for urban street scenes based on attention boundary enhancement and aggregation pyramid: A real-time semantic segmentation network model P for urban street scenes is constructed, which includes a pre-processing module and a multi-scale feature segmentation network and an aggregated pyramid pooling network cascaded with them as a main network, a detail enhancement sub-network and an attention boundary enhancement sub-network cascaded with the output ends of the pre-processing module and the multi-scale feature segmentation network as branch networks, and an adder cascaded with the output ends of the main network and the two branch networks, wherein: Pre-processing module, including stacked convolutional layers and BasicBlock layers; The backbone network consists of a multi-scale feature segmentation network consisting of N cascaded feature extraction modules of different scales (N ≥ 3). The aggregated pyramid pooling network consists of parallel feature extraction branches and aggregation modules. The feature extraction branches are used to extract pooled feature maps of different scales, and the aggregation module splices the feature extraction results. A detail enhancement sub-network, comprising N cascaded residual modules, wherein, except for the first feature extraction module and the first residual module, an input end of each residual module is connected to an input end of a corresponding feature extraction module; A boundary enhancement subnetwork, comprising a cascade of N-1 attention-aware boundary refinement modules, with the input of the nth residual module connected to the input of the n-1th attention-aware boundary refinement module; The segmentation head prediction network includes stacked convolutional layers, BN layers, ReLU activation functions, and convolutional layers; (3) Iteratively train the real-time semantic segmentation network model for urban street scenes: The urban street scene real-time semantic segmentation network model P is iteratively trained through the training sample set to obtain the trained semantic segmentation network model P * ; (4) Obtaining real-time semantic segmentation results of urban street view images: The test sample set is used as the trained semantic segmentation network model P * The input is used for forward reasoning to obtain the semantic segmentation results corresponding to M2 test samples.

2. The method according to claim 1, characterized in that The preprocessing steps described in step (1) are as follows: Each city street scene RGB image with a semantic label is normalized, and the normalized M images are randomly reversed left and right to obtain M preprocessed city street scene images.

3. The method according to claim 1, characterized in that The iterative training of the real-time semantic segmentation network model for urban street scenes described in step (3) is implemented as follows: (3a) The number of initial iterations is t, the maximum number of iterations is T, T ≥ 100000, and the weight of the semantic segmentation network model at the tth iteration is w t , and let t = 1; (3b) Each training sample in the training sample set is used as the input of the semantic segmentation network model P. The pre-processing module downsamples each training sample to obtain M1 downsampled images; (3c) The backbone network, detail enhancement sub-network, and boundary enhancement sub-network perform multi-scale feature extraction and feature aggregation, detail enhancement extraction, and boundary enhancement extraction on each downsampled image, respectively, to obtain a global feature information map, a detail feature information map, and a boundary feature information map corresponding to each training sample; (3d) The adder performs feature fusion on each global feature information map and its corresponding detail feature information map and boundary feature information map; the segmentation head prediction network performs category probability prediction on each pixel point of each fused feature map to obtain the category probability p of each pixel point. s (x,y) and the predicted segmentation result map O; (3e) uses the cross entropy loss function and passes the true semantic label y of each category s (x,y) and class probability p s (x,y) calculates the loss value L of P and uses L to adjust the weight w t Update to get the semantic segmentation network model P of this iteration t ; (3f) Determine whether t=T is true. If so, obtain the trained semantic segmentation network model P. * Otherwise, let t = t + 1, P = P t , and execute step (3b).

4. The method according to claim 3, characterized in that The backbone network described in step (3c) performs multi-scale feature extraction and feature aggregation on each downsampled image. The implementation steps are as follows: The multi-scale feature segmentation network extracts features of different scales from each downsampled feature map; the aggregate pyramid pooling network concatenates the features of different pooling layers on the output of the Nth feature extraction module to obtain a global feature information map.

5. The method according to claim 3, characterized in that The loss value L described in step (3d) is calculated as follows: Where L(x,y) represents the cross entropy of each pixel, H and W are the height and width of the city street view image respectively.

6. The method according to claim 3, characterized in that The weight parameter w described in step (3d) t Update, the update formula is: in Is the loss value L versus category probability p s The partial derivative of (x,y), It is p s (x,y) weight parameter w t The partial derivative of w t+1 It is w t The updated results, Is the loss value L to the weight parameter w t is the partial derivative of , and η is the learning rate.

Images