A semantic segmentation method for street scene images based on cross-aggregation model

Abstract

The present invention discloses a method for semantic segmentation of street scene images based on a cross-aggregation model. First, all image data in the Cityscapes dataset are divided into a training set, a validation set and a test set. Then, data enhancement and preprocessing operations are performed on all image data in the training set. A ResNet50 network model is constructed using a residual structure to obtain four feature maps. Two cross-aggregation modules are used to fuse two sets of non-adjacent feature maps. Then, an integration module is used to realize the re-fusion of feature maps. Finally, a feature selection unit is used to select features that are conducive to segmentation, and a feature map is output. Finally, the result of semantic segmentation is output. The present invention solves the problem that multi-scale objects and local area categories are mixed and difficult to segment in semantic segmentation in the prior art.

Description

Technical Field

[0001] The invention belongs to the technical field of digital image processing, and in particular relates to a semantic segmentation method for street scene images based on a cross-aggregation model. Background Art

[0002] Humans have a well-developed nervous system and can rely on their eyes to obtain external image information. After processing by the brain, they can accurately abstract low-level features such as color, texture, and edges in the image into high-level features such as categories and shapes, and then feed the processed signals back to the brain to complete complex tasks such as image scene understanding and target recognition. However, this process is difficult for computers to achieve. Humans are born with the ability to understand and perceive scenes, but this ability is a very challenging task for machines. The research goal of image semantic segmentation is to obtain the category label of each pixel in the image. It is a priori task for scene understanding, and scene understanding is based on segmentation to obtain the category labels of all pixels in the image.

[0003] Image semantic segmentation is a pixel-by-pixel classification task that has shown extremely important application value in fields such as intelligent driving, robotics, and medical diagnosis. In street scenes, the task of semantic segmentation technology is to understand the labels of different categories such as vehicles, pedestrians, roads, traffic lights, and traffic signs in the scene. Not only do multi-scale targets in street scenes affect segmentation, but the mixing of target categories in small areas also makes it impossible to distinguish similar target objects. There are still certain research difficulties in semantic segmentation for street scenes. Summary of the invention

[0004] The purpose of the present invention is to provide a street scene image semantic segmentation method based on a cross-aggregation model, which solves the problem in the prior art that multi-scale objects and local area categories are mixed and difficult to segment in semantic segmentation.

[0005] The technical solution adopted by the present invention is a method for semantic segmentation of street scene images based on a cross-aggregation model, which is specifically implemented according to the following steps:

[0006] Step 1: First, divide all image data in the Cityscapes dataset into training set, validation set and test set, and then perform data augmentation and preprocessing operations on all image data in the training set;

[0007] Step 2: Use the residual structure to construct a ResNet50 network model, input the images of the training set into the constructed ResNet50 model to extract the deep features of the image, and output the underlying features F l and high-level features F h , then for F h Four convolution operations are used to obtain four feature maps;

[0008] Step 3: First, use two cross-aggregation modules to fuse two sets of non-adjacent feature maps, then use the integration module to realize the re-fusion of feature maps, and finally use the feature selection unit to select features that are conducive to segmentation and output the feature map F. S ;

[0009] Step 4: The feature map F S Compared with the underlying feature map F in step 2 l The splicing, convolution and pixel convolution operations are performed in sequence to gradually restore the image resolution and finally output the semantic segmentation result.

[0010] The present invention is also characterized in that:

[0011] Step 1 is implemented according to the following steps:

[0012] Step 1.1, divide the image data of the public dataset Cityscapes into training set, validation set and test set in a ratio of 6:3:1;

[0013] Step 1.2: For all the image data in the training set, use the random flipping and random cropping methods to enhance the data, and finally normalize the image data in the training set;

[0014] Step 1.3: For the validation set and test set, first use the bilinear interpolation method to scale the image size to 513×1026 pixels for all image data; then crop them to 528×528 images; finally, normalize all image data for the validation set and test set.

[0015] Step 2 is implemented as follows:

[0016] Step 2.1, construct the residual module: first use 64 1×1 convolutions and ReLU functions to reduce the dimension, then use 3×3 convolutions and ReLU functions to extract features, and finally use 256 1×1 convolutions and ReLU functions to restore the dimension;

[0017] Step 2.2: First, use 64 7×7 convolutions with a stride of 2 to extract features, then use 1 3×3 global maximum pooling with a stride of 2 to operate on the data, and finally use 16 residual modules to construct the ResNet50 network. The seventh output is the underlying feature F. l , the last layer output is the high-level feature F h ;

[0018] Step 2.3: high-level features F h Using three different dilation rates d = 0, 1, 2, 4, the dilated convolutions obtain four feature maps: F1, F2, F3, and F4.

[0019] In step 2.1, the definition of the ReLU activation function is shown in formula (1):

[0020] ReLU(x)=max(0,x) (1)

[0021] Among them, x represents the input data, and the max() function returns the maximum value of the input data;

[0022] In step 2.3, the calculation method of the equivalent convolution kernel size k′ of the dilated convolution is defined as shown in formula (2):

[0023] k′=k+(k-1)×(d-1) (2)

[0024] Among them, k is the size of the traditional convolution kernel, and d is the expansion rate of the void convolution.

[0025] Step 3 is as follows:

[0026] Step 3.1, construct a cross-aggregation module: The two input feature maps of the cross-aggregation module are denoted as s and b. First, concatenate feature maps a and b to obtain feature M. Then, use 1×1 convolution, global pooling, batch normalization and ReLU activation on feature M to obtain feature M′. Multiply feature M′ and feature M to obtain feature M″. Finally, add feature M″ and feature map a to obtain the output of the cross-aggregation module.

[0027] Step 3.2: Input feature maps F1 and F3 into the cross-aggregation module to obtain feature C1, and input feature maps F2 and F4 into the cross-aggregation module to obtain feature C2;

[0028] Step 3.3: Construct an integration module, and obtain the output features of the integration module by subjecting the input feature map to global pooling, 3×3 convolution, batch normalization, and ReLU activation;

[0029] Step 3.4: First, concatenate C1 and C2 to obtain the feature map S, and then use the integration module to obtain the feature map S′;

[0030] Step 3.5, construct a feature selection unit: first concatenate the feature maps F1, F2, F3, and F4 in step 2.3, and concatenate them with the feature map S′ in step 3.4 to obtain the feature map M, use 3×3 convolution to obtain the feature map N, then use average pooling, batch normalization, and Sigmoid activation to obtain the feature N′, and finally multiply the feature maps N and N′ element by element to obtain the feature N″, and add N″ and the output features of the integrated model element by element to obtain the output features of the feature selection unit;

[0031] Step 3.6, the feature maps F1, F2, F3, F4 in step 2.3 are concatenated and the output feature map S′ of the integrated model is input into the feature selection unit obtained in step 3.5 to obtain the feature map FS .

[0032] The Sigmoid activation function in step 3.5 is defined as formula (3):

[0033]

[0034] Here, x represents the input value.

[0035] Step 4 is as follows:

[0036] First, F S With the underlying feature map F l Splicing to get F s ', then the feature map F s 'Use 3×3 convolution to get feature F s ", and finally for feature F s "Use pixel convolution to output the result of semantic segmentation.

[0037] The beneficial effects of the present invention are that a method for semantic segmentation of street scene images based on a cross-aggregation model (1) implements a two-stage fusion mechanism of multi-scale contexts, wherein the cross-aggregation module skips layers and guides the aggregation of information flows of different scales in pairs, realizes pre-fusion of non-adjacent scales, and the integration model re-integrates the two sets of multi-scale contexts to generate richer spatial context information; (2) the method of the present invention realizes global information weighted fusion of multi-scale features through a feature selection module, uses splicing features to guide the multi-scale context of the integration model to select features that are conducive to segmentation in different sub-regions, balances the influence of the context of different sub-regions on the segmentation results, and thus weakens the gridding effect. Therefore, the method of the present invention can effectively improve the problem that the limited expression of multi-scale features makes it difficult to distinguish similar targets, thereby causing confusion in the segmentation of street scene sub-regions. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the semantic segmentation method of the cross-aggregation model of the present invention;

[0039] Figure 2 is a schematic diagram of the cross-aggregation module structure used in the semantic segmentation method of the cross-aggregation model of the present invention;

[0040] Figure 3 is a schematic diagram of an integration module used in the semantic segmentation method of the cross-aggregation model of the present invention;

[0041] Figure 4 is a schematic diagram of a feature selection module used in the semantic segmentation method of the cross-aggregation model of the present invention;

[0042] Figure 5is a schematic diagram of a single pixel convolution diagram used in the semantic segmentation method of the cross-aggregation model of the present invention;

[0043] Figure 6 A comparison diagram of the first original image randomly obtained from the test set and the segmentation result in the embodiment of the present invention;

[0044] Figure 7 A comparison diagram of the second original image randomly obtained from the test set and the segmentation result in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] The present invention provides a method for semantic segmentation of street scene images based on a cross-aggregation model, the flow chart of which is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0047] Step 1: First, divide all image data in the Cityscapes dataset into training set, validation set and test set, and then perform data augmentation and preprocessing operations on all image data in the training set;

[0048] Step 1 is implemented according to the following steps:

[0049] Step 1.1, divide the image data of the public dataset Cityscapes into training set, validation set and test set in a ratio of 6:3:1;

[0050] Step 1.2: For all the image data in the training set, use the random flipping and random cropping methods to enhance the data, and finally normalize the image data in the training set;

[0051] Step 1.3: For the validation set and test set, first use the bilinear interpolation method to scale the image size to 513×1026 pixels for all image data; then crop them to 528×528 images; finally, normalize all image data for the validation set and test set.

[0052] Step 2: Use the residual structure to construct a ResNet50 network model, input the images of the training set into the constructed ResNet50 model to extract the deep features of the image, and output the underlying features F l and high-level features F h , then for F h Four convolution operations are used to obtain four feature maps;

[0053] Combination Figure 2 to Figure 5 , step 2 is implemented according to the following steps:

[0054] Step 2.1, construct the residual module: first use 64 1×1 convolutions and ReLU functions to reduce the dimension, then use 3×3 convolutions and ReLU functions to extract features, and finally use 256 1×1 convolutions and ReLU functions to restore the dimension;

[0055] Step 2.2: First, use 64 7×7 convolutions with a stride of 2 to extract features, then use 1 3×3 global maximum pooling with a stride of 2 to operate on the data, and finally use 16 residual modules to construct the ResNet50 network. The seventh output is the underlying feature F. l , the last layer output is the high-level feature F h ;

[0056] Step 2.3: high-level features F h Using three different dilation rates d = 0, 1, 2, 4, the dilated convolutions obtain four feature maps: F1, F2, F3, and F4.

[0057] In step 2.1, the definition of the ReLU activation function is shown in formula (1):

[0058] ReLU(x)=max(0,x) (4)

[0059] Among them, x represents the input data, and the max() function returns the maximum value of the input data;

[0060] In step 2.3, the calculation method of the equivalent convolution kernel size k′ of the dilated convolution is defined as shown in formula (2):

[0061] k′=k+(k-1)×(d-1) (5)

[0062] Among them, k is the size of the traditional convolution kernel, and d is the expansion rate of the void convolution.

[0063] Step 3: First, two cross-aggregation modules are used to fuse two sets of non-adjacent feature maps. The purpose is to pre-fuse features of different scales. Then, an integration module is used to realize the re-fusion of feature maps. The purpose is to eliminate semantic inconsistency. Finally, a feature selection unit is used to select features that are conducive to segmentation and output feature map F. S ;

[0064] Step 3 is as follows:

[0065] Step 3.1, construct a cross-aggregation module: The two input feature maps of the cross-aggregation module are denoted as a and b. First, feature maps a and b are concatenated to obtain feature M. Then, 1×1 convolution, global pooling, batch normalization and ReLU activation are applied to feature M to obtain feature M′. Feature M′ is multiplied by feature M to obtain feature M″. Finally, feature M″ is added to feature map a to obtain the output of the cross-aggregation module.

[0066] Step 3.2: Input feature maps F1 and F3 into the cross-aggregation module to obtain feature C1, and input feature maps F2 and F4 into the cross-aggregation module to obtain feature C2;

[0067] Step 3.3: Construct an integration module, and obtain the output features of the integration module by subjecting the input feature map to global pooling, 3×3 convolution, batch normalization, and ReLU activation;

[0068] Step 3.4: First, concatenate C1 and C2 to obtain the feature map S, and then use the integration module to obtain the feature map S′;

[0069] Step 3.5, construct a feature selection unit: first concatenate the feature maps F1, F2, F3, and F4 in step 2.3, and concatenate them with the feature map S′ in step 3.4 to obtain the feature map M, use 3×3 convolution to obtain the feature map N, then use average pooling, batch normalization, and Sigmoid activation to obtain the feature N′, and finally multiply the feature maps N and N′ element by element to obtain the feature N″, and add N″ and the output features of the integrated model element by element to obtain the output features of the feature selection unit;

[0070] Step 3.6, the feature maps F1, F2, F3, F4 in step 2.3 are concatenated and the output feature map S' of the integrated model is input into the feature selection unit obtained in step 3.5 to obtain the feature map F S .

[0071] The Sigmoid activation function in step 3.5 is defined as formula (3):

[0072]

[0073] Here, x represents the input value.

[0074] Step 4: The feature map F S Compared with the underlying feature map F in step 2 l The splicing, convolution and pixel convolution operations are performed in sequence to gradually restore the image resolution and finally output the semantic segmentation result.

[0075] Step 4 is as follows:

[0076] First, F S With the underlying feature map F l Splicing to get F s ', then the feature map F s 'Use 3×3 convolution to get feature F s ", and finally for feature F s "Use pixel convolution to output the result of semantic segmentation.

[0077] Figure 6 and Figure 7 is the semantic segmentation result of this method in two scenes on the Cityscapes test set. Figure 6 From the segmentation results, we can see that this method has a relatively accurate segmentation effect on small-scale objects (vehicles and street lights in the distance) and a relatively accurate segmentation effect on the edge contours of large-scale objects. Figure 7 It can be seen from the segmentation results that this method can segment the crowd scene with dense targets more clearly without any category mixing phenomenon.

[0078] Experimental verification:

[0079] The model is trained using the preprocessed training set image data. The validation set is used to evaluate the segmentation accuracy of the model during model training. The validation set does not participate in the training process. The training model uses the cross entropy loss function L ce , the initial learning rate is set to 0.0005, and the polynomial decay strategy is adopted;

[0080] Cross entropy loss function L ce The definition of is as follows:

[0081]

[0082] Where N represents the total number of categories; n is the current category, n∈{0, 1, …, N-1}; p is the probability value of the model output for each category of the current pixel; represents the true label value of the current pixel, where N = 19, n∈{0, 1, ..., 19};

[0083] The image data in the test set is input into the model one by one. The output of the model is the semantic segmentation result of each image. Then the average intersection-over-union ratio in the semantic segmentation method is used to measure the segmentation accuracy of the model.

[0084] The definition of the average intersection over union (mIoU) used is as follows (5):

[0085]

[0086] Among them, k is the number of categories other than background, among which, P ij The number of pixels that represent class i but are inferred to be class j, where k=19, i,j∈{0,1,...,19}.

Claims

1. A method for semantic segmentation of street scene images based on a cross-aggregation model, characterized in that: Follow the steps below to implement it: Step 1: First, divide all image data in the Cityscapes dataset into training set, validation set and test set, and then perform data augmentation and preprocessing operations on all image data in the training set; Step 2: Use the residual structure to construct the ResNet50 network model, input the images of the training set into the constructed ResNet50 model to extract the deep features of the image, and output the underlying features and high-level features , for high-level features Four different expansion rates were used The dilated convolutions of get four feature maps respectively: ; Step 3: First, use two cross-aggregation modules to fuse two sets of non-adjacent feature maps, then use the integration module to realize the re-fusion of feature maps, and finally use the feature selection unit to select features that are conducive to segmentation and output feature maps. ; The construction of the cross-aggregation module is as follows: The two input feature maps of the cross aggregation module are denoted as and , firstly transform the feature map and Features obtained by splicing , and then use the feature M Convolution, global pooling, batch normalization and ReLU activation get features , the feature Multiply it by feature M to get feature , and the final feature will be and feature map Add together to get the output of the cross-aggregation module; The construction feature selection unit is as follows: First, the feature map of step 2 Concatenate it with the feature map Concatenate to get feature map ,use Convolution to get feature map , and then use average pooling, batch normalization, and Activate the feature Finally, the feature map and Multiply element by element to get features ,Will The output features of the feature selection unit are obtained by adding the output features of the integrated model element by element; Step 4: Feature map Compared with the underlying feature map in step 2 The splicing, convolution and pixel convolution operations are performed in sequence to gradually restore the image resolution and finally output the semantic segmentation result.

2. The method for semantic segmentation of street scene images based on a cross-aggregation model according to claim 1, characterized in that: The step 1 is specifically implemented according to the following steps: Step 1.1, divide the image data of the public dataset Cityscapes into training set, validation set and test set in a ratio of 6:3:1; Step 1.2: For all the image data in the training set, use the random flipping and random cropping methods to enhance the data, and finally normalize the image data in the training set; Step 1.3: For the validation set and test set, first use the bilinear interpolation method to scale the image size to pixels; then cropped to Finally, all image data of the validation set and the test set are normalized.

3. The method for semantic segmentation of street scene images based on a cross-aggregation model according to claim 2, characterized in that: The Sigmoid activation function in step 1 is defined as formula (3): (3) in, Represents the input value.

4. The method for semantic segmentation of street scene images based on a cross-aggregation model according to claim 3, characterized in that: The step 2 is specifically implemented according to the following steps: Step 2.1: Construct residual module: First, use 64 The convolution and ReLU functions are used to reduce the dimension, and then 3×3 convolution and ReLU functions are used to extract features. Finally, 256 The convolution and ReLU functions restore the dimensions; Step 2.2, first use 64 , a convolution with a stride of 2 to extract features, and then use 1 , the global maximum pooling with a step size of 2 is operated on the data, and finally 16 residual modules are used to construct the ResNet50 network. The seventh output is the underlying feature , the last layer output is high-level features .

5. The method for semantic segmentation of street scene images based on a cross-aggregation model according to claim 4, characterized in that: In step 2, the definition of the ReLU activation function is shown in formula (1): (1) in, Represents input data, The function returns the maximum value in the input data; Equivalent convolution kernel size of dilated convolution The calculation method of is defined as shown in formula (2): (2) in, is the traditional convolution kernel size, is the dilation rate of the dilated convolution.

6. The method for semantic segmentation of street scene images based on a cross-aggregation model according to claim 5, characterized in that: The step 3 is as follows: Step 3.1: Feature map , Input into the cross aggregation module to get features , the feature map , Input into the cross aggregation module to get features ; Step 3.2: Construct an integration module to globally pool the input feature map. Convolution, batch normalization and ReLU activation are used to obtain the output features of the integration module; Step 3.3: First, and Concatenate to get feature map , and then use the integration module to get the feature map ; Step 3.4: The feature map in step 2 Concatenate and combine the output feature maps of the model Input into the feature selection unit to obtain the feature map .

7. The method for semantic segmentation of street scene images based on a cross-aggregation model according to claim 6, characterized in that: The step 4 is specifically as follows: First, With the underlying feature map Splicing , and then the feature map use Convolution gets features , and finally the feature Use pixel convolution to output the result of semantic segmentation.

Images