A Weakly Supervised Semantic Segmentation Method Based on Deep Learning

Abstract

The present invention discloses a weakly supervised semantic segmentation method based on deep learning; firstly, the existing data set is used to fine-tune the pre-trained Resnet50, then the trained Resnet50 is used to obtain the corresponding class activation map, the pseudo-label of the segmentation is obtained by using the set threshold, and the fully connected conditional random fields (Dense conditional random fields, Dense CRF) are used to optimize the label. Finally, the segmentation network is trained using the optimized pseudo-label. The method described in the present invention can complete the classification and semantic segmentation tasks of the target by using only image-level labels, which greatly reduces the large amount of manpower and material resources consumed by manual labeling. Compared with the existing weakly supervised method, this method is more efficient and the positioning result is more accurate.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and relates to semantic segmentation of images, and specifically to a deep learning method for performing semantic segmentation on images using image-level labels. Background Art

[0002] Image segmentation is one of the basic and key technologies for image understanding. Traditional image segmentation methods mainly include threshold-based image segmentation methods, region-based image segmentation methods, edge detection-based image segmentation methods, wavelet analysis and wavelet transform-based image segmentation methods, Markov random field model-based image segmentation methods, genetic algorithm-based image segmentation methods, clustering-based segmentation methods, etc. Due to their inherent limitations, these methods are not effective for the segmentation of complex images, such as natural image segmentation. With the development of deep learning, convolutional neural networks have been increasingly applied to image segmentation. From FCN, UNet, DilatedNet to DeepLab, PSPNet, the segmentation accuracy has been further improved. Since semantic segmentation training requires annotating every pixel in the image, it is very time-consuming to annotate data, and its complexity far exceeds that of image classification and target detection. Training a segmentation model often requires a lot of manpower on mask annotation. In order to solve this problem, people explore using image-level labels, Bounding Box, or Scribble, points, which are relatively easy annotations to train semantic segmentation models. This is weakly supervised semantic segmentation. Among them, image-level labels are the easiest to obtain, the cheapest, and the most widely used.

[0003] Since the image category label does not contain any position information, additional methods must be used to locate the target object in the image during segmentation. Class Activation Mapping (CAM) is one of the most commonly used positioning methods. By globally averaging the features extracted from the last convolutional layer of networks such as VGGNet and GoogLeNet and then inputting them into the fully connected layer for classification, CAM can project the category scores output by the fully connected layer back to the feature map of the last layer of the convolutional neural network to complete the rough positioning of the object to be segmented. The mainstream image-level annotation weakly supervised semantic segmentation methods all use CAM to locate the target to be segmented, and SEC is one of the more representative ones. SEC proposes the three principles of seed, expand, and constraint, and uses CAMs to calculate the contribution of local image areas to the scores of each category in the final image classification. SEC roughly estimates the area where each category of objects appears in the image. However, this positioning method can often only locate the most prominent area in the target area, and sometimes even positioning errors occur, so the obtained class activation map needs to be corrected. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the positioning area obtained by using the conventional class activation map is often not accurate enough and needs to be corrected. Therefore, the accuracy of the training result obtained by using the segmentation label obtained by the class activation map is not high enough.

[0005] In view of the above actual situation, the present invention provides a weakly supervised semantic segmentation method based on deep learning; establishes a unified framework that integrates classification and segmentation networks, and proposes a multi-scale reasoning class activation map method to improve the accuracy of weakly supervised positioning. At the same time, a new UNet-based segmentation network, Mixed-UNet, is proposed to improve the accuracy of semantic segmentation.

[0006] The method of the present invention is to first use the existing data set to fine-tune the pre-trained Resnet50, then use the trained Resnet50 to obtain the corresponding class activation map, use the set threshold to obtain the segmentation pseudo-label, and use the fully connected conditional random field (Dense CRF) to optimize the label. Finally, the segmentation network is trained using the optimized pseudo-label. Specifically, the following steps are included:

[0007] Step (1), training the classification network;

[0008] Using Resnet50 pre-trained on the ImageNet dataset as the classification framework, the convolutions of the fourth and fifth convolutional blocks are replaced with dilated convolutions with a dilated rate of 2. This way, while keeping the image spatial resolution unchanged, a larger receptive field can be obtained, a denser feature response can be obtained, and the amount of computation can be kept unchanged. First, the original data is collected and labeled by professionals. The labeled dataset is divided into a training set, a validation set, and a test set. According to the actual situation, the dataset has a total of K categories, and c represents any of the categories. Then, the classification framework is fine-tuned through the training set with classification labels to complete the training of the classification network.

[0009] Step (2), after the classification network training is completed, each image in the training set is sampled at multiple scales by bilinear interpolation, with sampling rates of 0.5, 1, 1.5, and 2.0 respectively. The sampled images of the four scales are respectively input into the classification network to obtain class activation maps of the four scales, and then the class activation maps of the four scales are sampled to the same size as the original input image, and they are fused and averaged to obtain the fused class activation map, that is, the corrected class activation map.

[0010] Step (3), normalize the corrected class activation map, and use the threshold segmentation method to obtain the segmentation map of each class according to the actual number of classifications.

[0011] Step (4): Take the training set as input, and use the segmentation map obtained in (3) as the label of semantic segmentation. At the same time, input the original input image and the segmentation map obtained in (3) into the fully connected conditional random field (Dense CRF) to optimize the label. The optimized label and the label obtained in (3) are used to train the UNet-based semantic segmentation network - Mixed-UNet. The overall framework of the semantic segmentation network is composed of two UNets. In the feature extraction stage, the two UNets share a feature extractor and are divided into two branches in the upsampling stage. The first branch adds the pyramid pooling module (PPM), and the second branch adds the attention mechanism.

[0012] Step (5), training phase, using the training set as input, calculating the output loss function and adjusting the network parameters through the back propagation algorithm, and verifying the model using the validation set. The model with the best effect on the validation set is taken as the final model. After the training is completed, the trained model is tested on the test set.

[0013] Beneficial effects of the present invention:

[0014] The method of the present invention can complete the classification and semantic segmentation of targets by using only image-level labels, which greatly reduces the manpower and material resources consumed by manual labeling. Compared with the existing weak supervision method, this method is more efficient and the positioning result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0016] Figure 2 Schematic diagram of Mixed-UNet according to an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of a PPM module according to an embodiment of the present invention;

[0018] Figure 4 Schematic diagram of the SE module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0020] This paper proposes a unified image classification and semantic segmentation framework, which can complete image classification and weakly supervised semantic segmentation tasks using only image-level labels. Figure 1 A weakly supervised semantic segmentation method based on deep learning includes the following steps:

[0021] Step (1) Train the classification network.

[0022] First, collect the original data and have professionals mark it. Divide the marked data set into training set, validation set, and test set. According to the actual situation, the data set has a total of K categories, and c represents any of the categories. Taking Resnet50 pre-trained on the ImageNet data set as the classification framework, replace the convolution of the fourth and fifth convolution blocks of Resnet50 with dilated convolutions with a dilated rate of 2. Under the same feature map, the dilated convolution can obtain a larger receptive field, thereby obtaining a denser feature response. Then remove the fully connected layer of Resnet50, and apply global average pooling (GAP) to the final feature map, and use the features obtained after global average pooling as the features of the fully connected layer that produces the required output (classification or other). Through such a connection structure, the weights of the output layer can be projected back to the convolution feature map to identify the importance of the image area. The detailed calculation process is as follows:

[0023] For any given input image, f k(x, y) represents the activation unit k of the last convolutional layer at the spatial position (x, y). Then, for the activation unit k, the result of global average pooling F is performed k =∑ x，y f k (x, y). Therefore, for a given category c, the output S after softmax is c for in is the response weight of the kth unit to category c. Reflects F k For the importance of category c. By substituting F k To S c You can get:

[0024]

[0025] Finally, the training set is used to train the classification network in order to fine-tune the parameters so that it has good performance on a specific data set.

[0026] Step (2) After the classification network training is completed, each image in the training set is sampled four times, with sampling rates of 0.5, 1.0, 1.5, and 2.0 respectively. The images of the four scales are input into the classification network Resnet50 to obtain the class activation map corresponding to each input image. Then, the class activation maps of the four different scales are sampled to the same size as the original input image, and they are fused and averaged. The specific process is as follows:

[0027] For the i-th input image, Represents the class activation map of scale j corresponding to the i-th image category c, where j = 0.5, 1.0, 1.5, 0.2. The class activation map after fusion of the four scales is:

[0028]

[0029] On the one hand, the fusion reasoning of multiple scales can enhance the accuracy of positioning and avoid target positioning errors. On the other hand, fusing the reasoning results of different scales can solve the detail loss of single-scale reasoning to a certain extent.

[0030] Step (3) normalizes the fused class activation map. c For any position in the c-th class activation map, use the following normalization formula to convert the value of the class activation map to a number between [0, 1].

[0031]

[0032] After normalization, the class activation map is threshold segmented. When the class activation map is greater than or equal to the threshold, it is set as the foreground, and when it is less than the threshold, it is set as the background. Let f(x, y) be the index of the class activation map at the spatial position (x, y), then the segmentation process is as follows:

[0033]

[0034] Wherein, T is the set threshold.

[0035] Step (4) uses the segmentation map obtained in step 3 as the label for training the semantic segmentation network. Figure 2 As shown in the figure, the overall framework of the semantic segmentation network is formed by the fusion of two UNets. In the feature extraction stage, the two UNets share a feature extractor, and in the upsampling stage, they are divided into two branches. The first branch adds the pyramid pooling module (PPM module), and the second branch adds the attention mechanism (SE module). Figure 3 It is a schematic diagram of the PPM module of an embodiment of the present invention; wherein, the pyramid pooling module can reduce the loss of contextual information between different sub-regions. The module integrates features of four different pyramid scales. The first row is the coarsest feature - global pooling generates a single bin output, and the following three rows are pooling features of different scales. In order to ensure the weight of the global feature, if the pyramid has a total of N levels, a 1×1 convolution is used after each level to reduce the number of channels to 1 / N of the original. Then, bilinear interpolation is used to obtain a feature map of the same size as before pooling, and finally concat them together. As shown Figure 4 As shown in Figure 2, the second branch adds a Squeeze-and-Excitation (SE) block, hoping to improve the network's representation ability by modeling the dependencies of each channel, and to adjust the features channel by channel, so that the network can learn to selectively enhance features containing useful information and suppress useless features through global information. The basic structure of the SE block is shown in Figure 1 The first step is the squeeze operation, which uses the global spatial features of each channel in the feature map as the representation of each channel in the feature map to form a channel descriptor. The second step is the activation operation, which learns the degree of dependence of the segmentation network on each channel and adjusts the weight of each channel in the feature map according to the degree of dependence. The adjusted feature map is the output of the SE block. The multi-scale information extracted by the first branch and the attention information extracted by the second branch are concat and convolved to obtain the final segmentation result.

[0036] The labels are optimized by using Dense CRF. Specifically, the training set and labels are input into DenseCRF at the same time. The Dense CRF model will optimize the labels according to the pixel correlation of the training set to obtain a more refined output. The loss function of the semantic segmentation network consists of two parts. One part is the loss Loss1 between the segmentation result and the label obtained by threshold segmentation in step (3), and the other part is the loss Loss1 between the segmentation result and the label after Dense CRF optimization. The total loss function Loss = Loss1 + Loss2. The loss function is in the form of weighted cross entropy loss function, and the formula is as follows:

[0037]

[0038] Among them, p represents the true label, Represents the predicted label. β is a hyperparameter that can be set manually. All positive samples are weighted according to β (0-1), which can effectively solve the problem of class imbalance.

[0039] Step (5) During the training process, the model is verified with the validation set every 100 iterations, and the verified model is saved. After the training is completed, the model with the best performance on the validation set is selected as the final model. In the testing phase, the image data of the test set is input to obtain the classification and segmentation results of the image.

Claims

1. A weakly supervised semantic segmentation method based on deep learning, characterized in that: Here are the steps: Step (1), training the classification network; Using Resnet50 pre-trained on the ImageNet dataset as the classification framework, the convolutions of the fourth and fifth convolutional blocks are replaced with dilated convolutions with a dilation rate of 2. First, the original data is collected and labeled by professionals. The labeled dataset is divided into a training set, a validation set, and a test set. According to the actual situation, the dataset has a total of K categories, and c represents any one of them. Then, the classification framework is fine-tuned through the training set with classification labels to complete the training of the classification network; Step (2), after the classification network training is completed, each image of the training set is sampled at multiple scales by bilinear interpolation, with sampling rates of 0.5, 1, 1.5, and 2.0 respectively; the sampled images of the four scales are respectively input into the classification network to obtain class activation maps of the four scales, and then the class activation maps of the four scales are sampled to be consistent with the size of the original input image, and they are fused and averaged to obtain the fused class activation map, that is, the corrected class activation map; Step (3), normalizing the corrected class activation map, and using the threshold segmentation method to obtain the segmentation map of each class according to the actual number of classifications; Step (4), taking the training set as input, taking the segmentation map obtained in (3) as the label of semantic segmentation, and inputting the original input image and the segmentation map obtained in (3) into the fully connected conditional random field to optimize the label; using the optimized label and the label obtained in (3) to train the UNet-based semantic segmentation network Mixed-UNet, the overall framework of the semantic segmentation network is formed by the fusion of two UNets, in the feature extraction stage, the two UNets share a feature extractor, and in the upsampling stage, they are divided into two branches; the first branch adds a pyramid pooling module, and the second branch adds an attention mechanism; Step (5), training phase, using the training set as input, calculating the output loss function and adjusting the network parameters through the back propagation algorithm, and verifying the model using the validation set; taking the model with the best effect on the validation set as the final model; after the training is completed, testing the trained model on the test set; The specific method of step (1) is as follows: First, the original data is collected and labeled by professionals; the labeled data set is divided into training set, validation set, and test set; according to the actual situation, the data set has K categories, and c represents any category; Taking Resnet50 pre-trained on the ImageNet dataset as the classification framework, the convolutions of the fourth and fifth convolutional blocks of Resnet50 are replaced with dilated convolutions with a dilation rate of 2; then the fully connected layer of Resnet50 is removed, and global average pooling is applied to the final feature map, and the features obtained after global average pooling are used as the features of the fully connected layer that produces the required output; The detailed calculation process is as follows: For any given input image, f k (x, y) represents the activation unit k of the last convolutional layer at the spatial position (x, y); then, for the activation unit k, the result F after performing global average pooling is k =∑ x,y f k (x, y); Therefore, for a given category c, the output S after softmax c for in is the response weight of the kth unit to category c; Reflects F k Importance of category c; by bringing in F k To S c You can get: Finally, the training set is used to train the classification network in order to fine-tune the parameters so that it has good performance on the data set; The specific method of step (2) is as follows: After the classification network training is completed, each image in the training set is sampled four times with sampling rates of 0.5, 1.0, 1.5, and 2.0 respectively. The images of the four scales are input into the classification network Resnet50 respectively to obtain the class activation map corresponding to each input image. Then, the class activation maps of these four different scales are sampled to the same size as the original input image, and they are fused and averaged. The specific process is as follows: For the i-th input image, Represents the class activation map of scale j corresponding to the i-th image category c, where j = 0.5, 1.0, 1.5, 0.

2. The class activation map after fusion of the four scales is: The specific method of step (3) is as follows: Normalize the fused class activation map; let x c For any position in the c-th class activation map, use the following normalization formula to convert the value of the class activation map into a number between [0,1]; After normalization, the class activation map is segmented by threshold. When the class activation map is greater than or equal to the threshold, it is set as the foreground, and when it is less than the threshold, it is set as the background. Let f(x, y) be the value of the class activation map at the spatial position (x, y), then the segmentation process is as follows: Among them, T is the set threshold; The specific method of step (4) is as follows: The segmentation map obtained in step 3 is used as the label for training the semantic segmentation network. The overall framework of the semantic segmentation network is formed by the fusion of two UNets. In the feature extraction stage, the two UNets share a feature extractor and are divided into two branches in the upsampling stage. The first branch adds a pyramid pooling module, and the second branch adds an attention mechanism. Among them, the pyramid pooling module can reduce the loss of contextual information between different sub-regions. The module integrates features of 4 different pyramid scales. The first row is the coarsest feature. Global pooling generates a single bin output, and the following three rows are pooling features of different scales. In order to ensure the weight of the global feature, if the pyramid has N levels, a 1×1 convolution is used after each level to reduce the number of channels to the original 1 / N. Then, a feature map of the same size as before pooling is obtained through bilinear interpolation, and finally concat them together. The second branch adds Squeeze-and-Excitation. block, the first step is the squeezing operation, which takes the global spatial features of each channel in the feature map as the representation of each channel in the feature map to form a channel descriptor; the second step is the activation operation, which learns the dependence of the segmentation network on each channel and adjusts the weight of each channel in the feature map according to the degree of dependence. The adjusted feature map is the output of the SE block; the multi-scale information extracted by the first branch and the attention information extracted by the second branch are concat and convolved to obtain the final segmentation result; The labels are optimized by using Dense CRF. Specifically, the training set and labels are input into Dense CRF at the same time. The Dense CRF model will optimize the labels according to the pixel correlation of the training set to obtain a more refined output. The loss function of the semantic segmentation network consists of two parts. One part is the loss Loss1 between the segmentation result and the label obtained by threshold segmentation in step (3). The other part is the loss Loss2 between the segmentation result and the label optimized by Dense CRF. The total loss function Loss = Loss1 + Loss2. The loss function is in the form of weighted cross entropy loss function, and the formula is as follows: Among them, p represents the true label, Represents the predicted label; β is a hyperparameter set by an artificial person. All positive samples are weighted according to β∈0~1, which can effectively solve the problem of class imbalance.

2. The method for weakly supervised semantic segmentation based on deep learning according to claim 1, characterized in that: Step (5) During the training process, the model is verified with the validation set every 100 iterations, and the verified model is saved; after the training is completed, the model with the best performance on the validation set is selected as the final model; During the testing phase, the image data of the test set is input to obtain the classification and segmentation results of the image.

Images