A semi-supervised semantic segmentation method based on generative adversarial network framework

Abstract

The present invention discloses a semi-supervised semantic segmentation method based on a generative adversarial network framework, which belongs to the field of computer semantic segmentation and can be used for semantic segmentation of conventional images. The method comprises the following steps: Step 1: Obtaining and preprocessing the VOC2012 dataset; Step 2: Constructing a generative adversarial network framework, using DeepLabv2 as the basic generative network and a fully convolutional network as the discriminant network; Step 3: Improving the generative network based on DeepLabv2, optimizing the backbone segmentation network Resnet101 in DeepLabv2, introducing the CBAM attention mechanism and depthwise separable convolution, so that the entire model distributes weight parameters more reasonably, improves the model's representation ability and makes the calculation more efficient; Step 4: Optimizing the discriminant network based on the fully convolutional network, changing the standard convolution to a dilated convolution, increasing the receptive field and strengthening the training effect of the entire network; Step 5: Training the model; Step 6: Using the trained model to perform a segmentation test on a test sample and outputting the segmentation result.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation technology, and specifically relates to a semi-supervised semantic segmentation method based on a generative adversarial network framework. Background Art

[0002] Deep learning-based image semantic segmentation models have achieved tremendous success, but data labeling is time-consuming and labor-intensive, pixel-level labeling is challenging, and requires extensive expertise, making it difficult to obtain robust supervisory information. Weakly supervised learning (WSL) leverages manually labeled information to reduce the workload. By integrating human experience with learning rules, it effectively addresses the data dependency inherent in fully supervised learning and has significant practical significance in challenging fields such as medicine and meteorology. Summary of the Invention

[0003] In view of the development of semantic segmentation, the present invention aims to provide a semi-supervised semantic segmentation method based on a generative adversarial network framework, which has the characteristics of good segmentation effect and high precision and can be competent for semi-supervised semantic segmentation tasks.

[0004] The object of the present invention is to achieve this by providing a semi-supervised semantic segmentation method based on a generative adversarial network framework, comprising the following steps:

[0005] Step 1: Obtain the daily segmentation dataset VOC2012 and preprocess it;

[0006] Step 2: Based on the generative adversarial network framework, DeepLabv2 is used as the basic generative network and the fully convolutional network is used as the discriminative network;

[0007] Step 3: Improve the generative network based on DeepLabv2, optimize the backbone segmentation network Resnet101 in DeepLabv2, introduce the CBAM attention mechanism and depthwise separable convolution, so that the weight parameters of the entire model are more reasonably distributed, the representation ability of the model is improved, and the calculation is more efficient;

[0008] Step 4: Optimize the identification network based on the fully convolutional network, change the standard convolution to a dilated convolution, increase the receptive field, and enhance the training effect of the entire network;

[0009] Step 5: After the optimization is completed, the model is trained, and the generator and discriminator are trained adversarially to improve the network segmentation ability;

[0010] Step 6: Use the trained model to perform segmentation test on the test sample and output the segmentation result. The present invention has high accuracy and precision in the field of semi-supervised segmentation for image segmentation and can be effectively applied to semi-supervised semantic segmentation tasks.

[0011] A further improvement of the present invention is that in the above step 1 of obtaining the VOC2012 dataset and preprocessing it, the specific steps are: first importing the samples in the dataset into the network, and then using random scaling and cropping operations with a size of 321×321.

[0012] A further improvement of the present invention is that in step 2 above, the generative adversarial network framework uses the Deeplabv2 framework and ResNet-101 as the generative network. The ResNet-101 network consists of 100 convolutional layers and one fully connected layer. The image scale is changed by the convolution step size. Its network structure is divided into five stages, and the image is downsampled by a factor of 2 after each stage. The discriminator network adopts a fully convolutional structure similar to FCN.

[0013] A further improvement of the present invention is that in the above step 3, the specific measures for optimizing the generated network are as follows:

[0014] The CBAM attention mechanism is introduced, and the CBAM self-attention mechanism is added before the last convolutional layer to more reasonably distribute the weight parameters of the model; the three standard convolutions of the Resnet residual block in the DeepLabv2 network are replaced with depthwise separable convolutions to speed up computing efficiency and improve training effects.

[0015] A further improvement of the present invention is to improve the discriminator fully convolutional network in step 4 as follows: introducing dilated convolution, replacing the original convolutional layers with a backbone network consisting of five convolutional layers, with a 4×4 kernel, the number of channels {64, 128, 256, 512, 1}, and dilated convolution parameters {1, 2, 5, 1, 2}. Except for the last layer, each convolutional layer is followed by a Leaky-ReLU with a parameter of 0.2. An upsampling layer is added to the last layer.

[0016] A further improvement of the present invention is that the training in the above step 5 includes the following steps:

[0017] 5.1 Network Parameter Initialization

[0018] 5.2 Setting training parameters

[0019] 5.3 Loading training data

[0020] 5.4 Iterative Adversarial Training

[0021] A further improvement of the present invention is that the training parameters in the above step 5.2 are: the generator network uses the stochastic gradient descent (SGD) optimization method, with a momentum of 0.9 and a weight decay of 10 −4 The initial learning rate is set to 2.5×10 −4The discriminator network uses the Adam optimization method with a learning rate of 10 −4 .

[0022] A further improvement of the present invention is that in the iterative adversarial training of step 5.4 above, the generator network in the generative adversarial network framework generates semantic segmentation results for the image, which are input into the discriminator network. The discriminator network determines whether the image is generated by the generator network or from a pre-labeled segmentation label. The network is iteratively trained through backpropagation, ultimately achieving a state where it is difficult for the discriminator to determine whether the generated image is a segmentation prediction map generated by the generator or a pre-labeled segmentation label. A better segmentation effect is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flow chart of the method of the present invention.

[0024] Figure 2 Generate adversarial network framework

[0025] Figure 3 To generate the improved structure diagram of the network

[0026] Figure 4 Improved structural diagram for identification network DETAILED DESCRIPTION

[0027] The present invention is further summarized below with reference to the accompanying drawings.

[0028] Example: A semi-supervised semantic segmentation method based on a generative adversarial network framework, such as Figure 1 As shown, the steps are as follows:

[0029] Step 1: Obtain the VOC2012 dataset, a dataset of everyday segmentation images. This dataset contains 21 categories of images, covering common object categories such as people, cars, airplanes, and animals. Each image is annotated with precise pixel-level annotations, indicating the location and category of each object. Import the image into the network and use random scaling and cropping to a size of 321×321.

[0030] Step 2: Based on the generative adversarial network framework, DeepLabv2 is used as the basic generative network, and the fully convolutional network is used as the discriminant network. The generative network in the generative adversarial network framework uses the Deeplabv2 framework and ResNet-101 as the generative network. The ResNet101 network consists of 100 convolutional layers and one fully connected layer. The image scale is changed by the convolution step. Its network structure is divided into five stages, and the image is downsampled by 2 times after each stage. The discriminator network uses a fully convolutional structure similar to FCN. Figure 2As shown in the figure, the generative network and the discriminative network are trained against each other, and the network is continuously optimized through back propagation, so that the segmentation result generated by the generative network is finally such that the discriminative network cannot clearly distinguish whether it is a manually labeled result or a segmentation result.

[0031] Step 3: Improve the generative network based on DeepLabv2, as follows Figure 3 As shown in the figure, the CBAM attention mechanism is introduced, and the CBAM self-attention mechanism is added before the last convolutional layer to more reasonably distribute the weight parameters of the model; the three standard convolutions of the Resnet residual block in the DeepLabv2 network are replaced with depthwise separable convolutions to speed up the calculation efficiency and improve the training effect.

[0032] Step 4: Optimize the identification network based on the fully convolutional network, such as Figure 4 As shown in the figure, dilated convolution is introduced, replacing the original convolutional layers with a backbone network consisting of five convolutional layers. The convolution kernel is 4×4, the number of channels is {64, 128, 256, 512, 1}, and the dilated convolution parameters are {1, 2, 5, 1, 2}. Except for the last layer, each convolutional layer is followed by a Leaky-ReLU with a parameter of 0.2. An upsampling layer is added to the last layer.

[0033] Step 5: The optimized model was trained. This paper used a generative network framework implemented in pyTorch and trained on a single NVIDIA GeForce RTX 2080 GPU. Training parameters were set as follows: The generator network used stochastic gradient descent (SGD) optimization with a momentum of 0.9 and a weight decay of 10−4. The initial learning rate was set to 2.5×10−4. The discriminator network used Adam optimization with a learning rate of 10−4. 20,000 iterations were performed with a batch size of 10.

[0034] Step 6: Use the trained model to perform segmentation tests on the test sample and output the segmentation results. This invention achieves high image segmentation accuracy and precision in the field of semi-supervised segmentation and can be effectively applied to semi-supervised semantic segmentation tasks. The results of the ablation experiment are shown in the table.

[0035] Group number Self-attention mechanism Separable convolution Dilated convolution MIoU (%) PA (%) Training duration (s) 1 × × × 59.2 86.1 19602 2 √ × × 62.3 87.2 20152 3 × √ × 59.5 86.5 17985 4 × × √ 63.5 87.7 21023 5 √ √ × 63.6 87.5 18502 6 √ × √ 65.5 88.9 20922 7 × √ √ 64.6 87.2 19012 8 √ √ √ 66.7 89.6 18625

[0036] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A semi-supervised semantic segmentation method based on a generative adversarial network framework, characterized by: The main steps include: Step 1: Obtain the daily segmentation dataset VOC2012 and preprocess it; Step 2: Based on the generative adversarial network framework, DeepLabv2 is used as the basic generative network and the fully convolutional network is used as the discriminative network; Step 3: Improve the generative network based on DeepLabv2, optimize the backbone segmentation network Resnet101 in DeepLabv2, introduce the CBAM attention mechanism and depthwise separable convolution, so that the weight parameters of the entire model are more reasonably distributed, the representation ability of the model is improved, and the calculation is more efficient; Step 4: Optimize the identification network based on the fully convolutional network, change the standard convolution to a dilated convolution, increase the receptive field, and enhance the training effect of the entire network; Step 5: After the optimization is completed, the model is trained, and the generator and discriminator are trained adversarially to improve the network segmentation ability; Step 6: Use the trained model to perform segmentation test on the test sample and output the segmentation result. The present invention has high accuracy and precision in the field of semi-supervised segmentation for image segmentation and can be effectively applied to semi-supervised semantic segmentation tasks. In the generative adversarial network framework in step 2, the generative network uses the Deeplabv2 framework and ResNet-101 as the generative network. The ResNet101 network consists of 100 convolutional layers and one fully connected layer. The image scale is changed by the convolution step size. Its network structure is divided into five stages. The image is downsampled by 2 times in each stage. The discriminator network adopts a fully convolutional structure based on FCN. The improvement of step 3 for generating the network is as follows: The CBAM attention mechanism is introduced, and the CBAM self-attention mechanism is added before the last convolutional layer to more reasonably distribute the model weight parameters; the three standard convolutions in the Resnet residual block in the DeepLabv2 network are replaced with depthwise separable convolutions to speed up computational efficiency and improve training results; The improvements to the discriminator full convolutional network in step 4 are as follows: introducing dilated convolution, replacing the original convolutional layer with a backbone network consisting of five convolutional layers, with a 4×4 convolution kernel, the number of channels {64, 128, 256, 512, 1}, and dilated convolution parameters {1, 2, 5, 1, 2}. Except for the last layer, each convolutional layer is followed by a Leaky-ReLU with a parameter of 0.2, and an upsampling layer is added to the last layer. The training in step 5 includes the following steps: 5.1 Network parameter initialization; 5.2 Set training parameters; 5.3 Loading training data; 5.4 Iterative Adversarial Training 2. The semi-supervised semantic segmentation method based on the generative adversarial network framework according to claim 1 is characterized by: The preprocessing method in step 1 is as follows: first, the samples in the dataset are imported into the network, and then random scaling and cropping operations with a size of 321×321 are performed.

3. The semi-supervised semantic segmentation method based on the generative adversarial network framework according to claim 2 is characterized in that: The training parameters for step 5.2 are: the generator network uses the stochastic gradient descent (SGD) optimization method, with a momentum of 0.9 and a weight decay of 10 −4 , the initial learning rate is set to 2.5×10 -4 , the discriminator network uses the Adam optimization method with a learning rate of 10 −4 .

4. The semi-supervised semantic segmentation method based on the generative adversarial network framework according to claim 3 is characterized in that: In the iterative adversarial training of step 5.4, the generative network in the generative adversarial network framework generates a semantic segmentation result for the image and inputs it into the discriminator network. The discriminator network determines whether this is an image generated by the generative network or from an already labeled segmentation label. The network is iteratively trained through backpropagation, and eventually the discriminator finds it difficult to determine whether the generated image is a segmentation prediction map generated by the generator or an already manually labeled segmentation label.

Images