A method for image semantic segmentation under difficult dataset preparation

Abstract

The invention discloses an image semantic segmentation method under the difficulty of data set preparation, which belongs to the field of image processing. Compared with existing data augmentation methods, such as flipping, rotation, translation, scaling, etc., the ACGAN designed by the present invention does not destroy the contextual information in the target image, and can generate data that is extremely similar to the real scene. When used for semantic segmentation network training, other data augmentation methods may change the image semantic information, but the samples generated by the present invention are extremely similar to the real scene and will not lose semantic information. Compared with other semantic segmentation methods, the AC‑Net designed by the present invention designs two-way convolution in the convolution layer and integrates multi-scale feature information, which can extract richer feature information, thereby improving the segmentation effect.

Description

Technical Field

[0001] This technology involves the field of image semantic segmentation, especially an image semantic segmentation method for some real-world scenarios where data sets are difficult to mine. Background Art

[0002] Although the scene objects used in semantic segmentation are currently drawn from real-world scenes, their complexity and variability make it difficult to collect datasets for specific practical tasks. Furthermore, because semantic segmentation is pixel-level, its labels require a large number of densely packed pixels, making the labeling of semantic segmentation relatively expensive and time-consuming. Therefore, this paper aims to propose a method that can effectively segment objects even in difficult dataset mining situations.

[0003] To address the difficulty of dataset mining, data augmentation methods are often used to expand datasets. Commonly used data augmentation methods include geometric transformations, such as flipping, rotation, translation, and scaling; and color transformations, such as contrast, color perturbation, and noise. However, due to the complexity and variability of real-world scenarios, the data augmented by these methods may not be effective for network training. Only data augmented by appropriate data augmentation methods will be effective. Therefore, it is often necessary to choose the appropriate data augmentation method for each dataset. For example, horizontally flipping an image is an effective data augmentation method for CIFAR-10, but not for MNIST, because the digit "6" becomes "9" after horizontal flipping. As real-world scenarios become more complex and volatile, existing data augmentation methods are increasingly unable to meet real-world needs. Summary of the Invention

[0004] Therefore, this paper aims to propose a method that can still effectively segment under the condition of difficult dataset mining, with the purpose of generating more effective data from existing datasets through data augmentation methods, thereby solving the problem of difficult dataset mining and production. A conditional generative adversarial network ACGAN is designed, and the conditional generative adversarial network is used to amplify data. A semantic segmentation network is designed to serve as the generator structure of ACGAN, and the generator structure is composed of a convolutional layer structure and a dual attention mechanism designed by the present invention. First, the convolution layer is designed with a multi-scale splicing idea. Then, considering that a simple two-convolution cascade may not be enough to extract enough feature information, the present invention designs a two-way convolution structure to extract sufficiently rich feature information. At the same time, the idea of ​​residual structure is also used in the design of the convolution layer, thereby designing a two-way convolution structure, and using this structure to design a new semantic segmentation network structure, thereby improving the effect of semantic segmentation under difficult dataset mining conditions in terms of both network structure and dataset.

[0005] This paper builds on the conditional generative adversarial network (ACGAN) and the semantic segmentation model (AC-Net). By effectively improving and enhancing the segmentation model and generating more valid data using existing datasets, efficient segmentation is achieved. The proposed method includes four parts: sample preprocessing and data augmentation, overall semantic segmentation network structure design, prediction result evaluation, and model testing. This paper proposes a method for image semantic segmentation under difficult dataset production conditions. The method includes:

[0006] Step 1: Sample preprocessing and data augmentation; such as Figure 3

[0007] Step 1.1: Obtain a sample image, normalize its resolution, and then concatenate the sample image and its corresponding semantic label visualization image into a new image.

[0008] Step 1.2: Use the ACGAN model to perform data augmentation on the new image obtained in step 1;

[0009] The ACGAN model includes: a generator and a discriminator. The generator has a total of 18 layers, including an encoding part and a decoding part. The encoding part includes: layers 1 to 8 connected in sequence, where the first layer is a two-way convolution structure. The two-way convolution structure is as follows: Figure 1 As shown, the structure includes 3 paths, and the input is directly divided into 3 paths, of which 2 paths have the same structure. These two paths pass through 2 3x3 convolution layers in sequence, and the input and output of the second 3x3 convolution layer are spliced ​​as the output of the path. The other path is a 1x1 convolution layer. The outputs of the three paths are fused together to form the output of the dual-path convolution structure; the second layer is a maximum pooling structure with a kernel of 2, and the first and second layers form a group of convolution pooling structures. The subsequent third and fourth layers, fifth and sixth layers, seventh and eighth layers are also of this convolution pooling structure; the decoding part includes: the 9th to 18th layers connected in sequence, the structure of the 9th layer is the same as that of the first layer, and the 10th layer is an upsampling structure, which is realized by bilinear interpolation. , the 11th and 12th layers have the same structure as the 9th and 10th layers, and the 13th layer has the same structure as the 1st layer; the 14th layer is a dual attention mechanism structure, which is composed of the position attention mechanism and the channel attention mechanism in DANet, the 15th layer is an upsampling structure, the 16th layer is a two-way convolution structure, the 17th layer is an upsampling structure, and the 18th layer has the same structure as the 1st layer; and, the output of the 1st layer is concatenated with the output of the 18th layer as the output of the 18th layer, the output of the 3rd layer is concatenated with the output of the 16th layer as the output of the 16th layer, the output of the 5th layer is concatenated with the output of the 13th layer as the output of the 13th layer, and the output of the 7th layer is concatenated with the output of the 11th layer as the output of the 11th layer;

[0010] After the data is input into the generator structure, a generated image is output, which then enters the discriminator;

[0011] The discriminator is a fully convolutional structure with a total of 5 layers. The first three layers are 3 4×4 convolutions with a stride of 2, and the last two layers are 2 4×4 convolutions with a stride of 1. The generator generates an image and enters the discriminator, which outputs a scalar value in the range of [0,1]. The generator and discriminator are continuously trained by inputting training data. The training ends when the discriminator output stabilizes at 0.5. At this time, inputting samples to the trained generator can generate new data, which is the augmented sample.

[0012] Step 2: Build a semantic segmentation network; Figure 2 As shown in the figure, the semantic segmentation network has the same structure as the generator in step 1; however, the training process is different from step 1. When training the generator in step 1, the input is the semantic label, while in the training of the semantic segmentation network, the input is the original image of the Cityscapes training set. In addition, there is a discriminator constraint during the training of the generator in step 1. The image generated by the generator in step 1 will continuously generate data close to the real image after being constrained by the discriminator, while in this part of the semantic segmentation network, there is no discriminator constraint, so this part of the semantic segmentation network will only generate segmented pictures, which is specifically reflected in the loss function of the two. The loss function in step 1 uses the conditional generative adversarial network cGAN-Loss, and the semantic segmentation network loss function is the cross entropy loss function CrossEntropyLoss.

[0013] Step 3: Use the data preprocessed in step 1 to train the semantic segmentation network obtained in step 2, and use the trained semantic segmentation network to perform actual image semantic segmentation.

[0014] The present invention designs ACGAN to provide a solution to the difficulties in collecting and producing real-world scene datasets. At the same time, the present invention designs a new semantic segmentation network AC-Net, which has better segmentation effect than U-Net.

[0015] Compared with existing data augmentation methods such as flipping, rotation, translation, and scaling, the ACGAN designed in the present invention does not destroy the contextual information in the target image and can generate data that is extremely similar to the real scene. When used for semantic segmentation network training, other data augmentation methods may change the semantic information of the image, but the samples generated by the present invention are extremely similar to the real scene and do not lose semantic information.

[0016] Compared with other semantic segmentation methods, the AC-Net designed in this paper designs two convolutions in the convolutional layer and integrates multi-scale feature information, which can extract richer feature information and thus improve the segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a two-way convolution structure;

[0018] Figure 2 The overall structure of the semantic segmentation network AC-Net;

[0019] Figure 3 ACGAN's network principle structure;

[0020] Figure 4 This is the overall system flow chart;

[0021] Figure 5 Data samples augmented by ACGAN;

[0022] Figure 6 A visual comparison chart of semantic segmentation results. DETAILED DESCRIPTION

[0023] The dataset selected for this article is the Cityscapes urban street scene dataset. The Cityscapes dataset includes street scene data from 50 different cities in France, Germany, and Switzerland. The dataset provides a total of 34 categories, but generally, so many categories are not needed. Generally, 19 categories plus a background class are used. The dataset provides two types of data: finely labeled data and coarsely labeled data. The finely labeled dataset has 5,000 finely labeled samples, while the coarsely labeled dataset provides 5,000 finely labeled samples and 20,000 coarsely labeled samples. In the experiment, only the finely labeled dataset was used. Of the 5,000 finely labeled samples, 2,975 are training sets, 500 are validation sets, and 1,525 are test sets.

[0024] Conventional data augmentation methods include image cropping, scaling, flipping, shifting, brightness adjustment, and noise addition. However, these conventional methods are not effective in some real-world scenarios. Therefore, this paper designs a conditional generative adversarial network model (ACGAN). First, the training set data in the Cityscapes dataset is preprocessed by resizing it to a uniform resolution of 1024×512. Then, the training set data and its corresponding semantic label visualization image are spliced ​​together to form a new image with a resolution of 2048×512. The processed dataset is input into the ACGAN model to generate more sample data.

[0025] 1. Overall structure of semantic segmentation network

[0026] After obtaining the new sample data generated by preprocessing, the generated new sample data and the original sample data are input into the semantic segmentation network model. The overall structure of the semantic segmentation network includes five parts: convolutional layer structure design, encoding and decoding parts, attention mechanism module, and skip connection.

[0027] 1.1 Convolutional Layer Structure

[0028] The U-Net convolutional layer structure extracts feature information through two connected 3×3 convolutions. However, with such a simple convolution design, the feature information extracted by the convolution layer is often limited. Therefore, this paper considers setting up multiple such 3×3 convolution combinations to extract feature information in the image separately, and then fusing the extracted feature information to improve the segmentation effect of the semantic segmentation network. At the same time, considering the advantages of the residual structure and multi-scale feature splicing analyzed above, these technologies are also applied to the convolutional layer design, and a dual-path convolution structure is proposed.

[0029] This dual-path convolutional architecture utilizes a residual structure and multi-scale feature splicing technology. To extract more and richer feature information in the convolutional layer, a multi-path convolutional structure is designed, and feature fusion is performed at the end to enable the network to obtain richer feature information. The input first enters the left and right convolutional structures of the dual-path convolutional architecture, and both convolutional structures use feature splicing to obtain multi-scale feature information. Then, with reference to the residual structure, the middle input undergoes a 1x1 convolution and is finally fused with the feature response maps on the left and right.

[0030] 1.2 Coding

[0031] The encoding part mainly performs feature extraction and compression, and mainly includes five dual-way convolutional structures proposed in this paper and four max pooling layers. Each dual-way convolutional structure is followed by a max pooling downsampling layer with a kernel of 2.

[0032] 1.3 Decoding part

[0033] The decoding part primarily performs feature reconstruction. This paper also designs an attention mechanism module within the decoding part. This attention mechanism improves the network's segmentation performance by capturing dependencies between pixels and channels. The decoding part primarily consists of four upsampling modules, four dual-path convolutional structures proposed in this paper, and an attention mechanism module. The upsampling in the decoding part uses bilinear interpolation, and each upsampling step is followed by a dual-path convolutional structure. However, due to the presence of skip connections, the input now consists of two parts.

[0034] 1.4 Attention Mechanism Module

[0035] This paper introduces a dual attention mechanism module, a type of attention mechanism proposed in DANet. It consists of two submodules: a position attention module and a channel attention module. The position attention module leverages contextual information by establishing dependencies between any two positions in the feature map and then updating the features of all positions via a weighted sum. The channel attention mechanism captures the dependencies between channels, obtains the channel weights by taking the weighted sum of all channels, and then updates each channel feature map, thereby improving the representation of semantic features.

[0036] 1.5 Skip Connections

[0037] This paper uses skip connections to fuse the feature information of the encoding and decoding parts. As the network deepens, some feature information is often lost during the process of large-scale convolution and pooling, affecting the final segmentation effect. Skip connections fuse the information of the shallow encoding part with the semantic information of the deep decoding part. The network can relearn some previously lost details, thereby improving the segmentation effect of the semantic segmentation network.

[0038] 2. Evaluation of prediction results

[0039] Semantic segmentation algorithms are evaluated using three main metrics: accuracy, memory usage, and time complexity. Accuracy is the key metric used to evaluate the performance of semantic segmentation models. Commonly used metrics include pixel accuracy (PA), mean pixel accuracy (MPA), mean intersection over union (MIoU), and frequency weighted intersection over union (FWIoU). To facilitate understanding and rationally assess the performance of semantic segmentation network models, this paper uses two metrics to evaluate the final segmentation results: PA and FWIoU.

[0040] 3. Model Testing

[0041] First, we train the semantic segmentation network model proposed in this paper using new data generated by the improved Pix2pix algorithm and the original data. After the trained model is trained, it is necessary to conduct actual tests to determine whether it has achieved the desired results. Therefore, we need to conduct model testing. The dataset used for this test is the test data provided by Cityscapes.

[0042] Figure 4 This is an overall flow chart, which is used to specifically illustrate the technical solution of the present invention.

[0043] 1) Since the ACGAN method designed in this paper requires paired images as input and the original dataset has a large resolution, the training set data in the Cityscapes dataset is preprocessed and resized to a uniform resolution of 1024×512. The training set data and its corresponding semantic label visualization image are then spliced ​​left and right to form a new image with a resolution of 2048×512.

[0044] 2) The processed training set data of the original data set is input into the improved Pix2pix network designed by the present invention, thereby augmenting a new data set.

[0045] 3) Integrate the augmented new dataset and the original dataset to obtain an integrated new dataset.

[0046] 4) The training set data in the original data set are respectively input into the U-Net network and the semantic segmentation network designed by the present invention. At the same time, the new data set obtained after augmentation and integration are respectively input into the U-Net network and the semantic segmentation network AC-Net designed by the present invention. After obtaining the trained semantic segmentation network model, the test set data in the original data are used to test the segmentation effects of these four network models.

[0047] 5) Finally, a comparative analysis of the experimental results was conducted. This experiment used evaluation metrics including pixel accuracy (PA) and weighted frequency intersection over (FWIoU). The PA and FWIoU of the trained U-Net network and the semantic segmentation network designed by the present invention were obtained through testing. A comparative analysis of the experimental results demonstrated the superiority of the semantic segmentation network designed by the present invention. Furthermore, networks were trained using the original dataset and a new dataset augmented and integrated by the present invention's method, and the test results were compared and analyzed. This demonstrates that the present invention's method can still accurately and effectively segment targets even when dataset acquisition and production are difficult.

[0048] This paper analyzes the principles, advantages, and disadvantages of the classic semantic segmentation network model U-Net, the advantages of the residual structure in the residual network, and the multi-scale feature splicing technology commonly used in semantic segmentation. Considering that a simple cascade of two convolutions may not be sufficient to extract sufficient feature information, this paper designs a dual-path convolution structure to extract sufficiently rich feature information. At the same time, the idea of ​​the residual structure and the multi-scale feature splicing technology are also applied to the design of the convolutional layer, thus designing a dual-path convolution structure. This dual-path convolution structure is then used to design a new semantic segmentation network structure, AC-Net. Next, from the perspective of the attention mechanism, this paper introduces a dual-attention mechanism module from DANet to improve the segmentation effect from both channel attention and position attention. Finally, to verify the effectiveness of the semantic segmentation network structure proposed in this paper, a comparative experiment was conducted with the U-Net network on the Cityscapes dataset. The final result is a 1.99% improvement in PA and a 2.09% improvement in FWIoU compared to the U-Net.

[0049] Furthermore, to address the difficulty of mining datasets in real-world scenarios, this paper designed the ACGAN method to augment the dataset. Using the augmented data along with the original sample data to train the network, a comparative experiment showed a 0.27% improvement in FWIoU compared to using only the original data. Therefore, the proposed method can achieve good segmentation results even with insufficient datasets, and holds great promise in the field of image semantic segmentation.

Claims

1. A method for image semantic segmentation when dataset preparation is difficult, the method comprising: Step 1: Sample preprocessing and data augmentation; Step 1.1: Obtain a sample image, normalize its resolution, and then concatenate the sample image and its corresponding semantic label visualization image into a new image. Step 1.2: Use the ACGAN model to perform data augmentation on the new image obtained in step 1; The ACGAN model includes: a generator and a discriminator. The generator has a total of 18 layers, including an encoding part and a decoding part. The encoding part includes: layers 1 to 8 connected in sequence, wherein the first layer is a dual-path convolution structure, which includes 3 paths. The input is directly divided into 3 paths, of which 2 paths have the same structure. These two paths pass through two 3x3 convolution layers in sequence, and the input and output of the second 3x3 convolution layer are spliced ​​as the output of the path. The other path is a 1x1 convolution layer. The outputs of the three paths are fused together to form the output of the dual-path convolution structure; the second layer is a maximum pooling structure with a kernel of 2. The first and second layers form a group of convolutional pooling structures. The subsequent layers 3, 4, 5, 6, 7, and 8 are also of this convolutional pooling structure; the decoding part includes: layers 9 to 10 connected in sequence. 18 layers, the 9th layer structure is the same as the 1st layer structure, the 10th layer is an upsampling structure, which is implemented by bilinear interpolation, the 11th and 12th layers are the same as the 9th and 10th layers, and the 13th layer has the same structure as the 1st layer; the 14th layer is a dual attention mechanism structure, which is composed of the position attention mechanism and the channel attention mechanism in DANet, the 15th layer is an upsampling structure, the 16th layer is a two-way convolution structure, the 17th layer is an upsampling structure, and the 18th layer has the same structure as the 1st layer; in addition, the output of the 1st layer is concatenated with the output of the 18th layer as the output of the 18th layer, the output of the 3rd layer is concatenated with the output of the 16th layer as the output of the 16th layer, the output of the 5th layer is concatenated with the output of the 13th layer as the output of the 13th layer, and the output of the 7th layer is concatenated with the output of the 11th layer as the output of the 11th layer; After the data is input into the generator structure, a generated image is output, which then enters the discriminator; The discriminator is a fully convolutional structure with a total of 5 layers. The first three layers are 3 4×4 convolutions with a stride of 2, and the last two layers are 2 4×4 convolutions with a stride of 1. The generator generates an image and enters the discriminator, which outputs a scalar value in the range of [0,1]. The generator and discriminator are continuously trained by inputting training data. The training ends when the discriminator output stabilizes at 0.

5. At this time, inputting samples to the trained generator can generate new data, which is the augmented sample. Step 2: Build a semantic segmentation network. This semantic segmentation network has the same structure as the generator in step 1. However, the training process is different from step 1. In step 1, the input for training the generator is the semantic label, while in step 1, the input for training the semantic segmentation network is the original image of the Cityscapes training set. In step 1, the loss function used is the conditional generative adversarial network cGAN-Loss, while the loss function of the semantic segmentation network is the cross-entropy loss function CrossEntropyLoss. Step 3: Use the data preprocessed in step 1 to train the semantic segmentation network obtained in step 2, and use the trained semantic segmentation network to perform actual image semantic segmentation.

Images