Urban streetscape image real-time semantic segmentation method based on attention boundary enhancement and aggregation pyramid

Abstract

The invention provides a city streetscape image real-time semantic segmentation method based on attention boundary enhancement and an aggregation pyramid. The method comprises the following implementation steps: obtaining a training sample set and a test sample set; constructing an urban street scene real-time semantic segmentation network model based on attention boundary enhancement and an aggregation pyramid, and carrying out iterative training on the urban street scene real-time semantic segmentation network model; and obtaining a real-time semantic segmentation result of the city street scene image. According to the method, the sensitivity to boundaries is enhanced through the attention boundary enhancement sub-network, context information is integrated by using feature perception from coarse to fine, the robustness of semantic information is improved, meanwhile, the pyramid pooling network is aggregated to carry out feature splicing on pooling feature maps of different scales, the network generates correct semantic tags, and the accuracy of the semantic tags is improved. And finally, the global feature information graph, the detail feature information graph and the boundary feature information graph of each sample output by the three networks are fused, so that the recognition precision of the semantic segmentation image is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation and computer vision technology, and relates to a semantic segmentation method, and specifically to a real-time semantic segmentation method for urban street scene images based on attention boundary enhancement and aggregation pyramid, which can be used in the fields of autonomous driving, smart cities, intelligent robots, etc. Background Art

[0002] Real-time semantic segmentation of visual images faces many challenges in complex scenes, especially when dealing with small-scale objects or slender objects. These problems not only affect the accuracy of segmentation, but also affect subsequent computer vision tasks such as scene understanding and augmented reality. Real-time semantic segmentation methods for visual images are usually based on UNet networks or bilateral networks, using pixel-level classification to achieve image segmentation. However, these methods are prone to inaccurate segmentation or poor boundary processing, especially in complex scenes, where semantic label confusion is prone to occur. In recent years, semantic segmentation methods based on deep learning have widely introduced context-aware and multi-feature fusion technologies. By processing different feature components of an image separately, deep learning models can better capture local details and global information in the image and assign semantic labels. However, when dealing with small-scale or slender objects in complex scenes, these methods often ignore these objects or misidentify similar objects in the background.

[0003] In order to overcome these defects, for example, the patent application with the publication number CN118334335A and the name of "A real-time urban street scene image semantic segmentation method based on dual interactive fusion" uses a UNet-like network to train and infer the initialized image of the street scene image. The encoder part is composed of Vanilla Conv and STDC modules. At each stage, it is downsampled by a module with a step size of 2, and then the highest-level context encoding information is handed over to the comprehensive perception module, and the receptive field of the model is improved by the hollow convolution; the fusion module fuses the adjacent different-scale information in the encoder; the cross-attention module CAM cross-weightedly fuses the encoding result of the comprehensive perception module with the result of information fusion to achieve the decoding and upsampling of the context information; after three-level decoding, the feature map is upsampled to the original image size and the segmentation result is predicted. The disadvantage of this method is that the model lacks sensitivity to the boundary information of the object, which makes the reasoning result easy to be blurred at the segmentation boundary, affecting the segmentation accuracy and visual effect. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a real-time semantic segmentation method for urban street scene images based on attention boundary enhancement and aggregation pyramid to solve the technical problem of low segmentation accuracy in the prior art.

[0005] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0006] (1) Obtain training sample set and test sample set:

[0007] Preprocess M city street scene RGB images with semantic labels and containing S categories randomly selected from the city street scene image dataset, and use the preprocessed M1 city street scene images as the training sample set, and use the remaining preprocessed M2 street scene images as the test sample set, where S ≥ 2, M ≥ 4000, M2 = M - M1;

[0008] (2) Construct a real-time semantic segmentation network model P for urban street scenes based on attention boundary enhancement and aggregation pyramid:

[0009] Construct a city street scene real-time semantic segmentation network model P including a pre-processing module and a multi-scale feature segmentation network and an aggregate pyramid pooling network cascaded therewith as a main network, a detail enhancement sub-network and an attention boundary enhancement sub-network cascaded with the output ends of the pre-processing module and the multi-scale feature segmentation network as branch networks, and an adder cascaded with the output ends of the main network and the two branch networks;

[0010] (3) Iterative training of the real-time semantic segmentation network model for urban street scenes:

[0011] The urban street scene real-time semantic segmentation network model P is iteratively trained through the training sample set to obtain the trained semantic segmentation network model P * ;

[0012] (4) Obtaining real-time semantic segmentation results of urban street view images:

[0013] The test sample set is used as the trained semantic segmentation network model P * The input is used for forward reasoning to obtain the semantic segmentation results corresponding to M2 test samples.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] 1. In the process of iteratively training the semantic segmentation network model and obtaining the semantic segmentation results, the backbone network, the detail enhancement subnetwork and the attention boundary enhancement subnetwork respectively perform multi-scale feature extraction and feature aggregation, detail enhancement extraction and boundary enhancement extraction on each downsampled image, and fuse the global feature information map, detail feature information map and boundary feature information map of each sample output by the three networks through an adder, thereby improving the recognition accuracy of the semantic segmentation image.

[0016] 2. The present invention uses an attention boundary enhancement subnetwork to perform boundary enhancement on the downsampled image output by the pre-processing module and the feature map output by the multi-scale feature segmentation network. It can integrate the complementary features of the multi-scale segmentation network, enhance the sensitivity to boundaries through spatial integrity and channel significance, and integrate contextual information by using coarse-to-fine feature perception, thereby improving the robustness of semantic information, and finally obtaining a clearer boundary feature information map in the boundary area, thereby improving the accuracy of the semantic segmentation image.

[0017] 3. The present invention extracts features of different scales for each downsampled feature map through the multi-scale feature segmentation network in the backbone network, and pools the pooled feature maps of different scales through the aggregated pyramid pooling network, and then performs feature splicing on the pooled feature maps to obtain a global feature information map with global context information, so that the network generates correct semantic labels and enhances the robustness of the model in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart for realizing the present invention;

[0019] Figure 2 It is a structural schematic diagram of the semantic segmentation network model of the present invention;

[0020] Figure 3 It is a structural schematic diagram of the aggregate pyramid pooling module of the present invention;

[0021] Figure 4 This is a schematic diagram of the structure of the attention enhancement perception module of the present invention;

[0022] Figure 5 Schematic diagram of the structure of the attention perception refinement module of the present invention;

[0023] Figure 6 It is a simulation comparison diagram of the segmentation effect of the present invention and the prior art. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0025] Reference Figure 1 , the present invention includes the following steps.

[0026] Step 1. Build the basic data set and test data set:

[0027] M city street view RGB images with semantic labels and containing S categories randomly selected from the city street view image dataset are preprocessed. Specifically, each city street view image with semantic labels is normalized, and the pixel value of the image is normalized from the integer range of 0-255 to the floating point range of 0-1. The RandomHorizontalFlip method is applied to the normalized image. The model can learn more feature transformations, thereby reducing dependence on specific directions and avoiding bias during training. In addition, random flipping can be regarded as a regularization method to help prevent model overfitting and improve performance on unseen data. The probability of random flipping is set to This ensures that the effect of data augmentation will not be biased towards a certain type of sample distribution, and the preprocessed M1 city street view images are used as the training sample set, and the remaining preprocessed M2 street view images are used as the test sample set, where S = 19, M = 5000, M1 = 3000, and M2 = 2000.

[0028] Step 2. Build a real-time semantic segmentation network model P for urban street scenes based on attention boundary enhancement and aggregation pyramid:

[0029] Construct a city street scene real-time semantic segmentation network model P including a pre-processing module and a multi-scale feature segmentation network and an aggregate pyramid pooling network cascaded therewith as a main network, a detail enhancement sub-network and an attention boundary enhancement sub-network cascaded with the output ends of the pre-processing module and the multi-scale feature segmentation network as branch networks, and an adder cascaded with the output ends of the main network and the two branch networks;

[0030] The pre-processing module includes cascaded convolutional layers and BasicBlock layers;

[0031] The backbone network includes a multi-scale feature segmentation network that includes cascaded N feature extraction modules of different scales, N ≥ 3; the aggregate pyramid pooling network includes parallel feature extraction branches and aggregation modules;

[0032] A detail enhancement sub-network, comprising N cascaded residual modules, wherein, except for the first feature extraction module and the first residual module, an input end of each residual module is connected to an input end of a corresponding feature extraction module;

[0033] An attention boundary enhancement subnetwork, comprising a cascade of N-1 perceptual boundary refinement modules, and the input of the nth residual module is connected to the input of the n-1th perceptual boundary refinement module;

[0034] The segmentation head prediction network includes stacked convolutional layers, BN layers, ReLU activation functions, and convolutional layers;

[0035] In the implementation of this case, the pre-processing module is connected in sequence from the first convolution layer → the second convolution layer → the first BasicBlock layer → the second BasicBlock layer → the ReLU activation layer → the third BasicBlock layer → the fourth BasicBlock layer, the ReLU activation layer. The pre-processing module quickly downsamples each training sample to make the image size become the original image.

[0036] The multi-scale feature segmentation network in the backbone network consists of three cascaded feature extraction modules of different scales. Each feature extraction module consists of a residual block;

[0037] The aggregated pyramid pooling network includes 6 parallel feature extraction branches and aggregation modules. The first branch and the sixth branch only contain one convolution layer, and the second to fifth branches are composed of a convolution layer and an upsampling layer respectively; the feature fusion module is divided into three sub-modules: in the first sub-module, the output of branch one is added point by point with the outputs of branches two and three, and the results are added point by point again for feature splicing, and the results are processed by the sixth convolution layer. In the second sub-module, the outputs of branches three and four are feature spliced, and then added point by point with the output of the first sub-module. In the third sub-module, the output of the second sub-module is processed by the seventh convolution layer, and then feature spliced ​​with the second sub-module, and then processed by the eighth convolution layer, and finally added point by point with the output of the sixth branch; this local parallel design solves the problem of poor real-time performance caused by high memory overhead and frequent read and write operations, so as to achieve lower inference latency;

[0038] The detail enhancement sub-network consists of three residual modules connected in sequence, where the input layer of the first residual module is the output of the pre-processing module, the input layer of the second residual module is the output of the first residual module and the output of the first feature extraction module in the backbone network, and the input layer of the third residual module is the output of the second residual module and the output of the second feature extraction module;

[0039] The attention boundary enhancement subnetwork consists of two attention-aware boundary refinement modules connected in sequence. The input of the first attention-aware boundary refinement module is the output of the pre-processing module and the output of the first feature extraction module in the backbone network. The obtained output and the second feature extraction module enter the second attention-aware boundary refinement module to enhance the boundary information.

[0040] The attention perception boundary refinement module includes two sets of input layers that are respectively sent to the attention enhancement perception module after passing through the convolution layer and the Bottleneck layer. The attention enhancement perception module performs weighted fusion on the two sets of features according to the attention weights (W and 1-W). The final fusion result passes through a Bottleneck layer and is output as an optimized feature map. The attention enhancement perception module includes a channel attention mechanism module and a spatial attention mechanism module that are connected in sequence.

[0041] The attention enhancement perception module is composed of two parallel branches and a feature fusion feature module. Branch one is composed of a global maximum pooling layer and a convolutional dimension reduction layer connected in sequence, and branch two is composed of a global average pooling layer and a convolutional dimension reduction layer connected in sequence. The fusion feature module adds the outputs of branch one and branch two point by point and passes through a sigmoid activation function. The output is then added point by point to the original input. The spatial attention mechanism module is composed of two parallel branches connected in sequence and a feature splicing module. Branch one is composed of a global maximum pooling layer connected in sequence, and branch two is composed of a global average pooling layer. The feature splicing module performs feature splicing on the outputs of branch one and branch two and passes through the first convolution layer. Then, the result obtained by the sigmoid activation function is added point by point to the output of the channel attention mechanism module, and then passes through a sigmoid activation function to generate a gradient-based matrix to assign weights to each pixel.

[0042] The adder performs feature fusion on each global feature information map and its corresponding detail feature information map and boundary feature information map;

[0043] The segmentation head prediction network includes the first convolution layer, BN layer, ReLU activation function, and second convolution layer connected in sequence. It predicts the category probability of each pixel point of each fused feature map and obtains the category probability p of each pixel point. s (x,y) and the predicted segmentation result map O.

[0044] Step 3. Iteratively train the city street scene real-time semantic segmentation network model:

[0045] (3a) The number of initial iterations is t, the maximum number of iterations is T, and the weight of the semantic segmentation network model at the tth iteration is w t , and let t = 1, T = 188500;

[0046] (3b) Each training sample in the training sample set is used as the input of the semantic segmentation network model P. The pre-processing module downsamples each training sample to obtain M1 downsampled images. Specifically, for each input image I with a size of H×W×3, the input image is first processed by two convolutions with H×W×C being 3×3×64, a step size of 2, and a padding of 1, to obtain a size of Feature map Secondly, two consecutive BasicBlock blocks with H×W×C of 3×3×64 are used in the ResNet network to capture the feature map The features of Then use the ReLU activation function to activate the feature map Get feature map Then, the feature map is processed through a BasicBlock with a H×W×C of 3×3×128 and a step size of 2. Downsampling, we get Then capture it through a BasicBlock block with H×W×C of 3×3×128 The feature of Feature map Finally, the ReLU activation function is used to activate the feature map Get feature map That is, the size of this stage is The final processing result I stem The formula is as follows:

[0047]

[0048] In the formula, Indicates the nth stage in the stem step, Conv n×n×q,s＝m,p＝l () indicates a convolution kernel with a convolution size of n×n, a number of channels of q, a step size of s of m, and zero padding p of l, and ReLU() indicates the activation function ReLU;

[0049] Directly using visual images of too large a size for deep neural network training will result in too large parameters and too long a time consumption, which is not conducive to the real-time implementation of the network model. On the other hand, visual images of too small a size will cause blurred object boundaries and poor segmentation accuracy. This operation converts large-sized visual images into feature maps of appropriate sizes to achieve a balance between the number of parameters and segmentation accuracy.

[0050] (3c) The backbone network, detail enhancement sub-network, and attention boundary enhancement sub-network perform multi-scale feature extraction and feature aggregation, detail enhancement extraction, and boundary enhancement extraction on each downsampled image, respectively, and obtain the global feature information map, detail feature information map, and boundary feature information map corresponding to each training sample. Specifically, stem As the input of the backbone network s , after the first feature extraction module in the multi-scale feature segmentation network s Perform a 3×3×256 convolution with a step size of 2 and zero padding of 1, then perform a 3×3×256 convolution, and then add the result to I sAdd the results of the 3×3×256 convolution with a step size of 2 and zero padding of 1 to get Then Then perform two more 3×3×256 convolutions, and the convolution results are the same as Add together to get at last Then perform two more 3×3×256 convolutions, and After adding, the ReLU activation function is used to obtain a size of Feature map The formula is as follows:

[0051]

[0052] After the second feature extraction module Perform a 3×3×512 convolution with a step size of 2 and zero padding of 1, then perform a 3×3×512 convolution, and then add the result to the convolution. Add the results of the convolution of 3×3×512 with a step size of 2 and zero padding of 1 to get Afterwards Then perform two more 3×3×512 convolutions, and the convolution results are the same as Add together to get Finally, two more 3×3×512 convolutions are performed, and After adding, the ReLU activation function is used to obtain a size of Feature map The formula is as follows:

[0053]

[0054] After the third feature extraction module Perform 1×1×512 convolution to reduce the dimension, then perform 3×3×512 convolution with a step size of 2 and zero padding of 1, then perform 1×1×1024 convolution to increase the dimension, and then Add the results of the convolution of 3×3×1024 with a step size of 2 and zero padding of 1 to get Afterwards Perform 1×1×512 convolution to reduce the dimension, then perform 3×3×512 convolution to extract features, and then perform 1×1×1024 convolution to increase the dimension. The convolution result is the same as Adding together the size is Feature map The formula is as follows:

[0055]

[0056] The output after three feature extractions enters the efficient aggregation pyramid pooling module to pass the pooling feature maps of different scales through the convolution layer and splice the results to a size of Feature map The four pooling layers are: 5×5 size, stride 2, zero padding 2; 9×9 size, stride 4, zero padding 4; 17×17 size, stride 8, zero padding 8; full size. These four results are all reduced in dimension by 1×1×96 convolution, and the results are recorded as The formula is as follows:

[0057]

[0058] Among them, Avg n×n,s＝m,p＝l () represents a pooling layer with size n×n, step size s is m, and zero padding p is l, and GAvg() represents a global pooling layer;

[0059] Respectively and The results are added point by point, and then concatenated and sent to the 3×3×96 convolutional layer to extract features. The result is recorded as and After splicing with Add them point by point, and then perform 3×3×96 convolution to extract features. The result is recorded as The formula is as follows:

[0060]

[0061] Among them, Cat() represents feature splicing, Represents the point-by-point addition of feature maps;

[0062] The calculated and The final feature concatenation is performed, and then a 1×1×256 convolution is performed to increase the dimension. The result is the same as the input feature map. The results of the 1×1×256 convolution are added point by point to obtain the global feature information map I ppm , the formula is as follows:

[0063]

[0064] The detail enhancement sub-network obtains the detail feature information map through three cascaded residual modules. The first residual module converts I s As the original input, two 3×3×128 convolutions are performed to extract features, and then combined with I s The results of the 1×1×128 convolution are added together and the result is recorded as Then two 3×3×128 convolutions are performed to extract features, and then The results of the 1×1×128 convolution are added together to get a size of Detailed features The formula is as follows:

[0065]

[0066] The output of the first residual module is Scale feature map The output of the first feature extraction module Scale feature map As the input of the second residual module, Perform 1×1×128 convolution to reduce the dimension, and then upsample by bilinear interpolation to obtain a size of of Will and Add them point by point, and then activate them with Sigmoid function to get the size of The weight matrix W1 and the weight matrix 1-W1 of the same size; W1 and The product of point-wise multiplication 1-W1 and The products of the point-wise multiplications are added point by point, resulting in a size of of The formula is as follows:

[0067]

[0068] Among them, UpSample() represents bilinear interpolation upsampling, Represents point-by-point multiplication, σ() represents the activation function Sigmoid;

[0069] The second residual block output Scale feature map With the second feature extraction module Scale feature map As the input of the third residual module, Perform two 3×3×128 convolutions to extract features and then combine them with I s The results of the 1×1×128 convolution are added together and the result is recorded as

[0070] Then two 3×3×128 convolutions are performed to extract features, and then The results of the 1×1×128 convolution are added together to get a size of Detailed features First, a 1×1×128 convolution is performed to reduce the dimension, and then upsampled by bilinear interpolation to obtain a size of of Will and Add them point by point, and then activate them with Sigmoid function to get the size of The weight matrix W2 of and the weight matrix 1-W2 of the same size;

[0071] W2 and The product of point-wise multiplication and 1-W2 The products of the point-wise multiplications are added point by point, resulting in a size of of Then Perform two 3×3×128 convolutions to extract features and then The results of the 1×1×128 convolution are added together and the result is recorded as Afterwards Perform 1×1×64 convolution, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase the dimension. The result is the same as The results of the 1×1×128 convolution are added point by point to get a size of Detailed feature infographic The formula is as follows:

[0072]

[0073] The attention boundary enhancement sub-network obtains the boundary feature information map through two cascaded attention-aware boundary refinement modules. The input of the first attention-aware boundary refinement module is the pre-processing module I. stem The output of the first feature extraction module Among them I stem Perform 1×1×64 convolution, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase the dimension. The result is the same as I stem The results of the 1×1×128 convolution are added point by point to obtain a size of Detailed features Perform 1×1×128 convolution to reduce the dimension, and then upsample by bilinear interpolation to obtain a size of of The calculated and The result obtained by point-by-point addition fusion I e The attention mechanism is introduced from the channel and space through the attention-enhanced perception submodule;

[0074] I eAfter global average pooling and global maximum pooling, 1×1×64 convolution is performed to reduce the dimension, 3×3×64 convolution is performed to extract features, and 1×1×128 convolution is performed to increase the dimension. The result after global average pooling and triple convolution Compared with the result after global maximum pooling and triple convolution Add them point by point, and after Sigmoid activation function, add them to the input I e Multiply point by point to get the result After the maximum pooling and average pooling, the results are concatenated, and a 3×3×1 convolution is performed. After the Sigmoid activation function, Multiply point by point to get the result After the Sigmoid activation function, the final result W is obtained e1 , the formula is as follows:

[0075]

[0076] Among them, GMP() represents global maximum pooling, GAP() represents global average pooling, MP() represents maximum pooling, and AP() represents average pooling;

[0077] Get the size The weight feature map W e1 and 1-W e1 After that, W e1 and Point-wise multiplication and sum of products 1-W e1 and The products of the point-by-point multiplications are added point by point, giving Perform 1×1×64 convolution, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase the dimension. The result is the same as The results of the 1×1×128 convolution are added point by point to obtain a size of Detailed features The formula is as follows:

[0078]

[0079] The input of the second attention-aware refinement module is the output of the first attention-aware refinement module. The output of the second feature extraction module Perform 1×1×64 convolution to reduce the dimension, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase the dimension. The result is the same as The results of the 1×1×128 convolution are added point by point to obtain a size of Detailed features Perform 1×1×128 convolution to reduce the dimension, and then upsample by bilinear interpolation to obtain a size of of The calculated and The point-by-point addition and fusion are analyzed from the two dimensions of channel and space through the AEPM module to obtain a size of The weight feature map W e2 and 1-W e2 ;

[0080] W e2 and Point-wise product sum 1-W e2 and The products of the point-wise multiplications are added point by point, resulting in a size of of Perform 1×1×64 convolution, then perform 3×3×64 convolution to extract features, and then perform 1×1×128 convolution to increase the dimension. The result is the same as The results of the 1×1×128 convolution are added point by point to get a size of Detailed feature infographic The formula is as follows:

[0081]

[0082] (3d) The adder performs feature fusion on each global feature information map and its corresponding detail feature information map and boundary feature information map; the segmentation head prediction network performs category probability prediction on each pixel point of each fused feature map to obtain the category probability p of each pixel point. s (x, y) and the predicted segmentation result map O, specifically: the global feature map information I ppm , Detailed feature map information and boundary feature map information Add and fuse point by point to get the result I o ', and after 3×3×128 convolution, BN batch normalization and ReLU activation function, 1×1×19 convolution is performed for final classification, resulting in a size of The segmentation result O is as follows:

[0083]

[0084] O=Conv 1×1×19 (ReLU(BN(Conv 3×3×128 (Io '))))

[0085] (3e) uses the cross entropy loss function and passes the true semantic label y of each category s (x,y) and class probability p s (x,y) calculates the loss value L of P and uses L to adjust the weight w t Update, specifically: calculate the loss value L, the calculation formula is:

[0086]

[0087] Where L(x,y) represents the cross entropy of each pixel, p s (x, y) is the probability that the model predicts that the pixel coordinates (x, y) belong to category s, y s (x, y) is the actual label of the pixel label, and H×W represents the size of the input image;

[0088] And by calculating the loss gradient, the weight w is completed t The update formula is:

[0089]

[0090] in is the loss value L to p s The partial derivative of (x,y), Yes s The partial derivative of (x,y) with respect to the weight, w t+1 Yes t The updated results, is the partial derivative of the loss value with respect to the weight, and η is the learning rate;

[0091] These gradients are back-propagated to each neural network layer and used to update the weights of each layer. The process is repeated for each layer until the gradients of all weights are calculated. Finally, the Adam optimizer is used to update the weights according to the gradients to obtain the semantic segmentation network model P for this iteration. t ;

[0092] (3f) Determine whether t = T. If so, obtain the trained semantic segmentation network model P. * Otherwise, let t = t + 1, P = P t , and execute step (3b);

[0093] Step 4. Obtain real-time semantic segmentation results of the city street view network:

[0094] The test sample set is used as the trained semantic segmentation network model P *The input is forward inferenced to obtain the images with pixel semantic annotations corresponding to M2 test samples as the semantic segmentation results.

[0095] The following is a simulation experiment to further illustrate the technical effect of the present invention:

[0096] 1. Simulation conditions and contents:

[0097] The hardware platform used in the simulation experiment is CPU Core TM i7-9700k, GPU Nvidia RTX 2080Ti with 11GB video memory and 32GB memory respectively; software platform is Python3.8 and PyCharm 2019.3.3x64, deep learning framework is PyTorch 1.13.0, operating system is Ubuntu 18.04;

[0098] The public dataset Cityscapes is selected. This dataset is considered to be one of the best high-resolution datasets. It contains images of various street scenes in major European cities with a resolution of up to 1024×2048. It consists of 5,000 finely annotated images and another 20,000 coarsely annotated images using polygonal labels. In this study, only the finely annotated images are used for training; the schematic diagram of the dataset is shown in the figure below. Figure 6 (a) shown.

[0099] 2. Analysis of simulation results:

[0100] Figure 6 (b) and 6(c) are the segmentation effect demonstration diagrams of the prior art and the present invention, respectively. It can be clearly seen from the diagram that, compared with the prior art method, the segmentation result of the present invention has sharper object edges and clearer lines, and the contour of the cyclist's body and the road marking effect diagram are more distinct; the present invention shows more detailed information, the texture of the cyclist's clothes is more obvious, and the cracks on the ground and the contours of the curb are also more clearly visible; the segmentation accuracy of the present invention is significantly improved, especially when processing complex scenes, such as the cyclist's clothes are accurately classified as "person" without color inconsistency or label confusion.

[0101] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A real-time semantic segmentation method for urban street scene images based on attention boundary enhancement and aggregation pyramid, characterized in that: The steps include: (1) Obtain training sample set and test sample set: Preprocess M city street scene RGB images with semantic labels and containing S categories randomly selected from the city street scene image dataset, and use the preprocessed M1 city street scene images as the training sample set, and use the remaining preprocessed M2 street scene images as the test sample set, where S ≥ 2, M ≥ 4000, (2) Construct a real-time semantic segmentation network model P for urban street scenes based on attention boundary enhancement and aggregation pyramid: Construct a city street scene real-time semantic segmentation network model P including a pre-processing module and a multi-scale feature segmentation network and an aggregate pyramid pooling network cascaded therewith as a main network, a detail enhancement sub-network and an attention boundary enhancement sub-network cascaded with the output ends of the pre-processing module and the multi-scale feature segmentation network as branch networks, and an adder cascaded with the output ends of the main network and the two branch networks; (3) Iterative training of the real-time semantic segmentation network model for urban street scenes: The urban street scene real-time semantic segmentation network model P is iteratively trained through the training sample set to obtain the trained semantic segmentation network model P * ; (4) Obtaining real-time semantic segmentation results of urban street view images: The test sample set is used as the trained semantic segmentation network model P * The input is used for forward reasoning to obtain the semantic segmentation results corresponding to M2 test samples.

2. The method according to claim 1, characterized in that The preprocessing of M randomly selected city street scene RGB images with semantic labels from the city street scene image dataset described in step (1) is implemented by: Each city street scene RGB image with a semantic label is normalized, and the normalized M images are randomly reversed left and right to obtain the preprocessed M city street scene images.

3. The method according to claim 1, characterized in that The urban street scene real-time semantic segmentation network model P described in step (2), wherein: Pre-processing module, including stacked convolutional layers and BasicBlock layers; The backbone network includes a multi-scale feature segmentation network that includes cascaded N feature extraction modules of different scales, N ≥ 3; the aggregate pyramid pooling network includes parallel feature extraction branches and aggregation modules, the feature extraction branches are used to extract pooling feature maps of different scales, and the aggregation module splices the results of feature extraction; A detail enhancement sub-network, comprising N cascaded residual modules, wherein, except for the first feature extraction module and the first residual module, an input end of each residual module is connected to an input end of a corresponding feature extraction module; A boundary enhancement subnetwork, comprising a cascade of N-1 attention-aware boundary refinement modules, and the input of the nth residual module is connected to the input of the n-1th attention-aware boundary refinement module; The segmentation head prediction network includes stacked convolutional layers, BN layers, ReLU activation functions and convolutional layers.

4. The method according to claim 3, characterized in that The iterative training of the urban street scene real-time semantic segmentation network model described in step (3) is implemented by: (3a) The number of initial iterations is t, the maximum number of iterations is T, T ≥ 100000, and the weight of the semantic segmentation network model at the tth iteration is w t , and let t = 1; (3b) Each training sample in the training sample set is used as the input of the semantic segmentation network model P, and the pre-processing module downsamples each training sample to obtain M1 downsampled images; (3c) The backbone network, detail enhancement sub-network, and boundary enhancement sub-network perform multi-scale feature extraction and feature aggregation, detail enhancement extraction, and boundary enhancement extraction on each downsampled image, respectively, to obtain a global feature information map, a detail feature information map, and a boundary feature information map corresponding to each training sample; (3d) The adder performs feature fusion on each global feature information map and its corresponding detail feature information map and boundary feature information map; the segmentation head prediction network performs category probability prediction on each pixel point of each fused feature map to obtain the category probability p of each pixel point. s (x, y) and the predicted segmentation result map O; (3e) uses the cross entropy loss function and passes the true semantic label y of each category s (x,y) and class probability p s (x,y) calculates the loss value L of P and uses L to adjust the weight w t Update to get the semantic segmentation network model P of this iteration t ; (3f) Determine whether t = T. If so, obtain the trained semantic segmentation network model P. * Otherwise, let t = t + 1, P = P t , and execute step (3b).

5. The method according to claim 4, characterized in that The backbone network described in step (3c) performs multi-scale feature extraction and feature aggregation on each downsampled image, and the implementation steps are as follows: The multi-scale feature segmentation network extracts features of different scales for each downsampled feature map; the aggregate pyramid pooling network concatenates the features of different pooling layers on the output of the Nth feature extraction module to obtain a global feature information map.

6. The method according to claim 4, characterized in that The loss value L described in step (3d) is calculated as follows: Where L(x,y) represents the cross entropy of each pixel, H and W are the height and width of the city street view image respectively.

7. The method according to claim 4, characterized in that The weight parameter w described in step (3d) t Update, the update formula is: in Is the loss value L for the class probability p s The partial derivative of (x,y), Yes s (x,y) for the weight parameter w t The partial derivative of t+1 Yes t The updated results, is the loss value L for the weight parameter w t is the partial derivative of , and η is the learning rate.

Images