Semantic segmentation method based on attention-guided multi-scale contextual information interaction
By adopting a multi-scale context information interaction method based on attention guidance in semantic segmentation, the problems of multi-scale segmentation restriction and in-class segmentation in street scenes are solved, and higher segmentation accuracy and consistency are achieved.
Patent Information
- Application Number
- CN202210600487.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-30
- Publication Date
- 2025-03-28
- Estimated Expiration
- 2042-05-30
AI Technical Summary
The prior art has problems in the semantic segmentation of street scenes caused by multi-scale segmentation limitation and single serial connection.
The semantic segmentation method based on attention guidance is adopted for multi-scale context information interaction, and pixel features are encoded on different scales through the attention guidance module, combined with the ResNet50 network model and depth separation convolution, to realize the interaction and splicing of multi-scale features.
It effectively solves the problem that multi-scale objects are difficult to segment and partially divided within the same category in street scenes, and improves the in-class consistency and segmentation accuracy of multi-scale objects.
Smart Images
Figure CN115115831B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of digital image processing, and in particular relates to a semantic segmentation method based on attention-guided multi-scale context information interaction. Background Art
[0002] The research goal of image semantic segmentation is to obtain the category label of each pixel in the image. It is a priori task of scene understanding, and scene understanding is based on segmentation to obtain the category labels of all pixels in the image. Image semantic segmentation for street scenes has shown extremely important application value in the fields of intelligent driving, intelligent traffic management, etc. Semantic segmentation based on deep learning only needs to input data, and it can automatically extract the features of each layer in the image through the nonlinear mapping and back propagation mechanism of input and output. The features learned through the network have stronger expression ability. All features in the image from low-level features to high-level features are conducive to semantic segmentation, but deep learning methods are limited by the depth of the model. Therefore, it is difficult to extract edge information and strong structural information at the same time. It is inevitable to lose one thing while extracting another when extracting features, especially in the semantic segmentation of street scenes, small-scale targets are lost and similar targets are difficult to accurately segment. Summary of the invention
[0003] The purpose of the present invention is to provide a semantic segmentation method based on attention-guided multi-scale contextual information interaction, which solves the problems of multi-scale segmentation limitation and intra-class segmentation inconsistency caused by a single serial connection in the street scene semantic segmentation in the prior art.
[0004] The technical solution adopted by the present invention is a semantic segmentation method based on attention-guided multi-scale context information interaction, which is specifically implemented according to the following steps:
[0005] Step 1: Divide all image data in the public data set CamVid into training set, validation set and test set, and then perform data enhancement and preprocessing operations on all image data in the training set;
[0006] Step 2: Use the residual structure to construct the ResNet50 network model; use the attention mechanism and depth-wise separable convolution to construct the attention guidance module, and use global average pooling to construct the pooling unit;
[0007] Step 3: First, upsample and downsample the urban scene image S1 by two times to obtain images S2 and S3. Then, use the ResNet50 model to extract the deep features of S1, S2, and S3, denoted as S1′, S2′, and S3′. The third layer feature of S1 is denoted as S l Finally, the multi-scale interactive encoding network based on attention guidance is used to extract the multi-scale features F of the image S ;
[0008] Step 4: Multi-scale feature F S The third layer of S1 l Perform concatenation, convolution, and quadruple upsampling operations in sequence to gradually restore the image resolution and finally output the semantic segmentation result;
[0009] Step 5: Use the training set and validation set to train the model, and verify the segmentation effect of the model on the test set.
[0010] The present invention is also characterized in that:
[0011] Step 1 is implemented according to the following steps:
[0012] Step 1.1, divide the image data of the CamVid dataset into a training set, a validation set, and a test set in a ratio of 4:1:3;
[0013] Step 1.2: For all image data in the validation set and the test set, data augmentation is first performed using random cropping and random rotation methods, and then the images are preprocessed using bilinear interpolation to scale the images to 513×513 pixels.
[0014] Step 2 is implemented as follows:
[0015] Step 2.1, construct the residual module: first use 64 1×1 convolutions and ReLU functions to reduce the dimension, then use 3×3 convolutions and ReLU functions to extract features, and finally use 256 1×1 convolutions and ReLU functions to restore the dimension;
[0016] Step 2.2, ResNet50 structure table, first use 64 7×7 convolutions with a stride of 2 to extract features, and then use 16 residual modules, where the residual structure adds the feature matrix every other layer;
[0017] Step 2.3, use the deep convolution unit and the attention head unit to construct the attention guidance module. The deep convolution unit first uses the global average pooling layer to extract the average value x' of the channel dimension of the ResNet50 output feature map x, and then uses two groups of deep separable convolution units to extract the deep feature x'. The deep separable convolution unit is composed of 3×3 depth-wise convolution layer, batch normalization layer, ReLU activation layer, 3×3 point-wise convolution layer, batch normalization layer, ReLU activation layer, and finally uses double upsampling on x" to obtain the feature x'.
[0018] Step 2.4: The attention head unit in the attention guidance module first uses two groups of attention modules to calculate the channel attention map y of x'. The attention module consists of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation layer. Then, 1×1 convolution and Sigmoid activation are used to obtain the feature y' on the channel attention map y. Finally, the feature y' is multiplied element-by-element with the feature x" in step 2.3 to obtain the output feature of the attention guidance module.
[0019] Step 2.5: Construct a pooling unit, and use global average pooling, 1×1 convolution, batch normalization, and ReLU activation on the input feature map x to obtain the output feature map x. g .
[0020] In steps 2.1, 2.3, 2.4, and 2.5, the definition of the ReLU activation function is shown in formula (1):
[0021] ReLU(x)=max(0,x) (1)
[0022] Among them, x represents the input data, and the max() function returns the maximum value of the input data;
[0023] The Sigmoid activation function in step 2.4 is defined as formula (2)
[0024]
[0025] Here, x represents the input data.
[0026] Step 3 is as follows:
[0027] Step 3.1: First, upsample and downsample the urban scene image S1 by two times to obtain images S2 and S3; then use the ResNet50 model to extract the deep features of S1, S2, and S3, denoted as S'1, S'2, and S'3, and extract the third layer features of S1, denoted as S l ;Finally, the pooling unit is used to obtain the feature S g ;
[0028] Step 3.2: Use the attention guidance module in step 2 to calculate the channel attention weights of the deep features S'1, S'2, and S'3 respectively to obtain features S'1, S'2, and S'3; then g Add element by element to feature S'3 to get feature S 3_1 , the feature S 3_1 Add element by element to feature S'1 to get feature S 1_1 , and finally S 1_1 Add element-by-element with S'2 to get the multi-scale feature F S .
[0029] Step 4 is as follows:
[0030] First, the multi-scale feature F S The third layer feature S of the urban scene image S1 l Concatenate according to the channel dimension to get the feature F s ', then for the feature F s 'Use 3×3 convolution and four times upsampling to obtain semantic segmentation results.
[0031] Step 5 is as follows:
[0032] The model was trained using the image data of the training set of the CamVid dataset for a total of 110,100 training runs. The average intersection over Union (mIoU) was used to evaluate the accuracy of the validation set for 367 training runs. The training model used the cross entropy loss function L ce , the initial learning rate is set to 0.0005, and a polynomial decay strategy is adopted. After training, the average intersection-over-union ratio of this method on the Cityscapes test set is 74.02%.
[0033] In step 5, the cross entropy loss function L ce The definition of is as follows:
[0034]
[0035] Where N represents the total number of categories 12; n represents the current category, n∈{0,1,...,11}; p is the probability value of the model output for each category of the current pixel; Represents the true label value of the current pixel;
[0036] The definition of the average intersection over union (mIoU) used is as follows (4):
[0037]
[0038] Among them, k is the number of categories other than background, p ij The number of pixels that represent class i but are inferred to be class j, k=11, i,j∈{0,1,...,k}.
[0039] The beneficial effect of the present invention is that the semantic segmentation method based on the interaction of multi-scale context information guided by attention makes the network focus on the relative attention of adjacent scale features through a chain connection structure, rather than calculating the attention of all scales at the same time, which can reduce the burden during training and facilitate model convergence; the method of the present invention uses an attention guidance module to encode pixel features at different scales, which is conducive to multi-scale object segmentation, and improves the intra-class consistency by capturing long-distance dependencies. The module can be regarded as a small encoding and decoding structure to increase the network depth and improve the model learning ability. Therefore, the method of the present invention can effectively solve the problem of multi-scale objects being difficult to segment in street scene images and inconsistent segmentation within the same category. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 is a flow chart of the attention-guided asymmetric semantic segmentation method of the present invention;
[0041] Figure 2 is a schematic diagram of the structure of an attention guidance module used in the attention-guided asymmetric semantic segmentation method of the present invention;
[0042] Figure 3 is the implementation details of the attention head unit used in the attention-guided asymmetric semantic segmentation method of the present invention;
[0043] Figure 4 is a schematic diagram of a pooling unit used in the attention-guided asymmetric semantic segmentation method of the present invention;
[0044] Figure 5 A comparison diagram of the first original image and the second original image randomly obtained from the test set and the segmentation result in an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0046] The semantic segmentation method of the present invention is based on the interaction of multi-scale context information guided by attention, and the flowchart is as follows: Figure 1 As shown, the specific implementation steps are as follows:
[0047] Step 1: Divide all image data in the public data set CamVid into training set, validation set and test set, and then perform data enhancement and preprocessing operations on all image data in the training set;
[0048] Step 1 is implemented according to the following steps:
[0049] Step 1.1, divide the image data of the CamVid dataset into a training set, a validation set, and a test set in a ratio of 4:1:3;
[0050] Step 1.2: For all image data in the validation set and the test set, data augmentation is first performed using random cropping and random rotation methods, and then the images are preprocessed using bilinear interpolation to scale the images to 513×513 pixels.
[0051] Step 2: Use the residual structure to construct the ResNet50 network model; use the attention mechanism and depth-wise separable convolution to construct the attention guidance module, and use global average pooling to construct the pooling unit;
[0052] Combination Figure 2 to Figure 4 , step 2 is implemented according to the following steps:
[0053] Step 2.1, construct the residual module: first use 64 1×1 convolutions and ReLU functions to reduce the dimension, then use 3×3 convolutions and ReLU functions to extract features, and finally use 256 1×1 convolutions and ReLU functions to restore the dimension;
[0054] Step 2.2, the structure table of ResNet50, as shown in Table 1, first uses 64 7×7 convolutions with a stride of 2 to extract features, and then uses 16 residual modules, where the residual structure adds the feature matrix layer by layer; it is a short-circuit connection.
[0055] Table 1 ResNet50 structure information table
[0056]
[0057] Step 2.3, use the deep convolution unit and the attention head unit to construct the attention guidance module. The deep convolution unit first uses the global average pooling layer to extract the average value x' of the channel dimension of the ResNet50 output feature map x, and then uses two groups of deep separable convolution units to extract the deep feature x'. The deep separable convolution unit is composed of 3×3 depth-wise convolution layer, batch normalization layer, ReLU activation layer, 3×3 point-wise convolution layer, batch normalization layer, ReLU activation layer, and finally uses double upsampling on x" to obtain the feature x'.
[0058] Step 2.4: The attention head unit in the attention guidance module first uses two groups of attention modules to calculate the channel attention map y of x'. The attention module consists of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation layer. Then, 1×1 convolution and Sigmoid activation are used to obtain the feature y' on the channel attention map y. Finally, the feature y' is multiplied element-by-element with the feature x" in step 2.3 to obtain the output feature of the attention guidance module.
[0059] Step 2.5: Construct a pooling unit, and use global average pooling, 1×1 convolution, batch normalization, and ReLU activation on the input feature map x to obtain the output feature map x.g .
[0060] In steps 2.1, 2.3, 2.4, and 2.5, the definition of the ReLU activation function is shown in formula (1):
[0061] ReLU(x)=max(0,x) (5)
[0062] Among them, x represents the input data, and the max() function returns the maximum value of the input data;
[0063] The Sigmoid activation function in step 2.4 is defined as formula (2)
[0064]
[0065] Here, x represents the input data.
[0066] Step 3: First, upsample and downsample the city scene image S1 by two times to obtain images S2 and S3. Then, use the ResNet50 model to extract the deep features of S1, S2, and S3, denoted as S1', S2', and S3'. The third layer feature of S1 is denoted as S l Finally, the multi-scale interactive encoding network based on attention guidance is used to extract the multi-scale features F of the image S ;
[0067] Step 3 is as follows:
[0068] Step 3.1: First, upsample and downsample the urban scene image S1 by two times to obtain images S2 and S3; then use the ResNet50 model to extract the deep features of S1, S2, and S3, denoted as S'1, S'2, and S'3, and extract the third layer features of S1, denoted as S l ;Finally, the pooling unit is used to obtain the feature S g ;
[0069] Step 3.2: Use the attention guidance module in step 2 to calculate the channel attention weights of the deep features S'1, S'2, and S'3 respectively to obtain features S'1, S'2, and S'3; then g Add element by element to feature S'3 to get feature S 3_1 , the feature S 3_1 Add element by element to feature S'1 to get feature S 1_1 , and finally S 1_1 Add element-by-element with S'2 to get the multi-scale feature F S .
[0070] Step 4: Multi-scale feature F S The third layer of S1 lPerform concatenation, convolution, and quadruple upsampling operations in sequence to gradually restore the image resolution and finally output the semantic segmentation result;
[0071] Step 4 is as follows:
[0072] First, the multi-scale feature F S The third layer feature S of the urban scene image S1 l Concatenate according to the channel dimension to get the feature F s ', then for the feature F s 'Use 3×3 convolution and four times upsampling to obtain semantic segmentation results.
[0073] Step 5: Use the training set and validation set to train the model, and verify the segmentation effect of the model on the test set.
[0074] Step 5 is as follows:
[0075] The model was trained using the image data of the training set of the CamVid dataset for a total of 110,100 training runs. The average intersection over Union (mIoU) was used to evaluate the accuracy of the validation set for 367 training runs. The training model used the cross entropy loss function L ce , the initial learning rate is set to 0.0005, and a polynomial decay strategy is adopted. After training, the average intersection-over-union ratio of this method on the Cityscapes test set is 74.02%.
[0076] In step 5, the cross entropy loss function L ce The definition of is as follows:
[0077]
[0078] Where N represents the total number of categories 12; n represents the current category, n∈{0,1,...,11}; p is the probability value of the model output for each category of the current pixel; Represents the true label value of the current pixel;
[0079] The definition of the average intersection over union (mIoU) used is as follows (4):
[0080]
[0081] Among them, k is the number of categories other than background, p ij The number of pixels that represent class i but are inferred to be class j, k=11, i,j∈{0,1,...,k}.
[0082] Figure 5This is the semantic segmentation result of this method on the test set of the public dataset CamVid. As can be seen from scene 1, this method achieves fine segmentation of the target, for example, the body contour edge and tire edge of the segmentation result image are segmented more accurately; in scene 2, it can be seen that this method achieves accurate prediction of the details of small targets, for example, the details of the street light pole and the structure of the distant vehicle are completely segmented.
Claims
1. A semantic segmentation method based on attention-guided multi-scale contextual information interaction, characterized in that: Follow the steps below to implement it: Step 1: Divide all image data in the public data set CamVid into training set, validation set and test set, and then perform data enhancement and preprocessing operations on all image data in the training set; Step 2: Use the residual structure to construct the ResNet50 network model; use the attention mechanism and depth-wise separable convolution to construct the attention guidance module, and use global average pooling to construct the pooling unit; The step 2 is specifically implemented according to the following steps: Step 2.1, construct the residual module: first use 64 1×1 convolutions and ReLU functions to reduce the dimension, then use 3×3 convolutions and ReLU functions to extract features, and finally use 256 1×1 convolutions and ReLU functions to restore the dimension; Step 2.2, ResNet50 structure table, first use 64 7×7 convolutions with a stride of 2 to extract features, and then use 16 residual modules, where the residual structure adds the feature matrix every other layer; Step 2.3, use the deep convolution unit and the attention head unit to construct the attention guidance module. The deep convolution unit first uses the global average pooling layer to extract the average value x' of the channel dimension of the ResNet50 output feature map x, and then uses two groups of deep separable convolution units to extract the deep feature x'. The deep separable convolution unit is composed of 3×3 depth-wise convolution layer, batch normalization layer, ReLU activation layer, 3×3 point-wise convolution layer, batch normalization layer, ReLU activation layer, and finally uses double upsampling on x" to obtain the feature x'. Step 2.4: The attention head unit in the attention guidance module first uses two groups of attention modules to calculate the channel attention map y of x'. The attention module consists of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation layer. Then, 1×1 convolution and Sigmoid activation are used to obtain the feature y' on the channel attention map y. Finally, the feature y' is multiplied element-by-element with the feature x" in step 2.3 to obtain the output feature of the attention guidance module. Step 2.5: Construct a pooling unit, and use global average pooling, 1×1 convolution, batch normalization, and ReLU activation on the input feature map x to obtain the output feature map x. g ; Step 3: First, upsample and downsample the city scene image S1 by two times to obtain images S2 and S3. Then, use the ResNet50 model to extract the deep features of S1, S2, and S3, denoted as S1', S2', and S3'. The third layer feature of S1 is denoted as S l Finally, the multi-scale interactive encoding network based on attention guidance is used to extract the multi-scale features F of the image S ; Step 4: Multi-scale feature F S The third layer of S1 l Perform concatenation, convolution, and quadruple upsampling operations in sequence to gradually restore the image resolution and finally output the semantic segmentation result; Step 5: Use the training set and validation set to train the model, and verify the segmentation effect of the model on the test set.
2. The semantic segmentation method based on attention-guided multi-scale context information interaction according to claim 1 is characterized in that: The step 1 is specifically implemented according to the following steps: Step 1.1, divide the image data of the CamVid dataset into training set, validation set and test set in a ratio of 4:1:3; Step 1.2: For all image data in the validation set and the test set, data augmentation is first performed using random cropping and random rotation methods, and then the images are preprocessed using bilinear interpolation to scale the images to 513×513 pixels.
3. The semantic segmentation method based on attention-guided multi-scale contextual information interaction according to claim 2 is characterized in that: In steps 2.1, 2.3, 2.4, and 2.5, the definition of the ReLU activation function is as shown in formula (1): ReLU(x)=max(0,x) (1) Among them, x represents the input data, and the max() function returns the maximum value of the input data; The Sigmoid activation function in step 2.4 is defined as formula (2) Here, x represents the input data.
4. The semantic segmentation method based on attention-guided multi-scale contextual information interaction according to claim 3 is characterized in that: The step 3 is as follows: Step 3.1: First, upsample and downsample the urban scene image S1 by two times to obtain images S2 and S3; then use the ResNet50 model to extract the deep features of S1, S2, and S3, denoted as S1', S'2, and S3', and extract the third layer features of S1 as S l ; Finally, the pooling unit is used to obtain the feature S1' g ; Step 3.2: Use the attention guidance module in step 2 to calculate the channel attention weights of the deep features S1', S'2', and S3' respectively to obtain features S1", S'2', and S3"; then use the feature S g Add element by element to feature S3" to get feature S 3_1 , the feature S 3_1 Add element by element to feature S1" to get feature S 1_1 , and finally S 1_1 Add element-by-element with S2" to get the multi-scale feature F S .
5. The semantic segmentation method based on attention-guided multi-scale context information interaction according to claim 4 is characterized in that: The step 4 is specifically as follows: First, the multi-scale feature F S The third layer feature S of the urban scene image S1 l Concatenate according to the channel dimension to get the feature F s ', then for the feature F s 'Use 3×3 convolution and four times upsampling to obtain semantic segmentation results.
6. The semantic segmentation method based on attention-guided multi-scale context information interaction according to claim 5, characterized in that: The step 5 is specifically as follows: The model was trained using the image data of the training set of the CamVid dataset for a total of 110,100 training runs. The average intersection over Union (mIoU) was used to evaluate the accuracy of the validation set for 367 training runs. The training model used the cross entropy loss function L ce The initial learning rate is set to 0.0005, and the polynomial decay strategy is adopted. After training, the average intersection-over-union ratio of this method on the Cityscapes test set is 74.02%.
7. The semantic segmentation method based on attention-guided multi-scale context information interaction according to claim 6 is characterized in that: In step 5, the cross entropy loss function L ce The definition of is as follows: Where N represents the total number of categories 12; n represents the current category, n∈{0,1,...,11}; p is the probability value of the model output for each category of the current pixel; Represents the true label value of the current pixel; The definition of the average intersection over union (mIoU) used is as follows (4): Among them, k is the number of categories other than background, p ij The number of pixels that represent class i but are inferred to be class j, k=11, i,j∈{0,1,...,k}.