Semantic segmentation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and discloses a semantic segmentation method, device and equipment and a storage medium, and the method comprises the steps: improving a generative adversarial network based on a DeepLabv3 + network architecture in combination with a cavity space pyramid pooling module, a space attention module and a feature alignment module, obtaining an improved generator network, and obtaining a semantic segmentation result; the method comprises the following steps: inputting a source image into an improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation graph and a discriminator network based on a complete convolution structure, replacing standard convolution in the discriminator network with cavity convolution to obtain an improved discriminator network, and inputting the semantic segmentation graph into the improved discriminator network for discrimination. And obtaining a semantic segmentation result in combination with unlabeled data of the source image. The generative adversarial network is improved based on a cavity space pyramid pooling module, a space attention module, a feature alignment module and complete convolution, and the model generalization ability and semantic segmentation precision are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a semantic segmentation method, apparatus, device and storage medium. Background Art

[0002] Semantic segmentation is an important task in the field of computer vision. Its goal is to assign each pixel in an image to a specific category. It has a wide range of applications in many fields, such as autonomous driving and medical image analysis.

[0003] At present, the mainstream semantic segmentation method is the fully supervised learning method. The training effect of the semantic segmentation network depends too much on the quality and quantity of data, and the fully supervised method requires accurate pixel-by-pixel annotation of each training image, which consumes a lot of time. In addition, in the fully supervised learning method, although the fully supervised small sample image segmentation technology has developed in recent years, its dependence on a small amount of labeled data can easily lead to model overfitting, limiting the generalization ability and the complexity of feature capture. Summary of the invention

[0004] The main purpose of this application is to provide a semantic segmentation method, device, equipment and storage medium, aiming to solve the technical problems that the existing semantic segmentation technology is low in efficiency and easily leads to overfitting of the training model.

[0005] To achieve the above objectives, the present application proposes a semantic segmentation method, which includes:

[0006] Based on the DeepLabv3+ network architecture, the generative adversarial network is improved by combining the dilated spatial pyramid pooling module, spatial attention module and feature alignment module to obtain an improved generator network.

[0007] Inputting the source image into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map;

[0008] Based on a discriminator network with a fully convolutional structure, the standard convolution in the discriminator network is replaced with a hole convolution to obtain an improved discriminator network;

[0009] The semantic segmentation map is input into the improved discriminator network for discrimination, and the semantic segmentation result is obtained by combining the unlabeled data of the source image.

[0010] In one embodiment, the step of inputting the source image into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map includes:

[0011] Inputting the labeled data of the source image into the improved generator network, and extracting features from the source image through the residual network;

[0012] Processing the features based on depthwise separable convolution to obtain secondary features, and concatenating multiple secondary features in the channel dimension to obtain fused features;

[0013] Processing the source image using the atrous spatial pyramid pooling module, the spatial attention module, and the feature alignment module to obtain global features;

[0014] The fusion feature and the global feature are combined to obtain a semantic segmentation map.

[0015] In one embodiment, the step of using the atrous spatial pyramid pooling module, the spatial attention module, and the feature alignment module to process the source image to obtain global features includes:

[0016] Based on the atrous spatial pyramid pooling module, a global average pooling operation is performed on the source image to obtain a global feature vector;

[0017] Based on the spatial attention module, perform an average pooling operation and a maximum pooling operation on the source image to obtain an average pooling feature and a maximum pooling feature, and splice the average pooling feature and the maximum pooling feature according to the channel dimension to obtain a spliced ​​feature;

[0018] After performing a convolution operation on the concatenated features, a Sigmoid activation function is used to normalize the concatenated feature values ​​to obtain an attention weight map;

[0019] Based on the global feature vector and the attention weight map, combined with the feature alignment module, a global feature is obtained.

[0020] In one embodiment, the step of obtaining the global feature based on the global feature vector and the attention weight map in combination with the feature alignment module includes:

[0021] Based on the feature alignment module, low-level features and high-level features of the source image are acquired, and the high-level features are up-sampled;

[0022] After the high-level features are upsampled, the high-level features are processed using the spatial attention module to obtain processed high-level features;

[0023] Splicing the processed high-level features with the low-level features to obtain fused features;

[0024] Performing branch processing on the fused features to obtain branch features, and predicting a two-dimensional offset of the branch features;

[0025] Aligning the processed high-level features and the low-level features through the two-dimensional offset to obtain an aligned feature map;

[0026] The global feature vector, the attention weight map and the aligned feature map are combined to obtain a global feature.

[0027] In one embodiment, the step of improving the generative adversarial network based on the DeepLabv3+ network architecture, combining the dilated spatial pyramid pooling module, the spatial attention module and the feature alignment module to obtain an improved generator network includes:

[0028] The DeepLabv3+ network architecture is used as the basic framework of the generative adversarial network, and the residual network is used as the backbone network of the basic framework to obtain the generator network;

[0029] Applying a dilated spatial pyramid pooling module to the generator network, and improving the parallel branches of the dilated spatial pyramid pooling module into a multi-branch series connection form;

[0030] A spatial attention module and a feature alignment module are applied to the generator network to obtain an improved generator network.

[0031] In one embodiment, the step of inputting the semantic segmentation map into the improved discriminator network for discrimination, and obtaining the semantic segmentation result in combination with the unlabeled data of the source image, comprises:

[0032] Inputting the semantic segmentation map into the improved discriminator network for discrimination to obtain a discrimination result;

[0033] Combining the unlabeled data of the source image and the discrimination result, training the improved generator network to obtain a trained generator network;

[0034] Based on the trained generator network, iteratively optimize the improved discriminator network to obtain a trained discriminator network;

[0035] The trained generator network and the trained discriminator network are combined to obtain a semantic segmentation result.

[0036] In one embodiment, the step of inputting the semantic segmentation map into the improved discriminator network for discrimination to obtain a discrimination result includes:

[0037] Inputting the semantic segmentation map into the improved discriminator network;

[0038] Based on the linear rectification function and the multi-level convolution layer parameters, the semantic segmentation map is subjected to a hole convolution process to obtain a processed semantic segmentation map;

[0039] Based on the upsampling layer of the improved discriminator network, the processed semantic segmentation map is rescaled to the source image size to obtain a discrimination result.

[0040] In addition, to achieve the above purpose, the present application also proposes a semantic segmentation device, a generator improvement module, which is used to improve the generative adversarial network based on the DeepLabv3+ network architecture, combined with a hollow spatial pyramid pooling module, a spatial attention module and a feature alignment module to obtain an improved generator network;

[0041] A generator processing module, used for inputting the source image into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map;

[0042] A discriminator improvement module, which is used for a discriminator network based on a fully convolutional structure, replacing the standard convolution in the discriminator network with a hole convolution to obtain an improved discriminator network;

[0043] The result acquisition module is used to input the semantic segmentation map into the improved discriminator network for discrimination, and obtain the semantic segmentation result in combination with the unlabeled data of the source image.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes a semantic segmentation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the semantic segmentation method described above.

[0045] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the semantic segmentation method described above are implemented.

[0046] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the semantic segmentation method described above are implemented.

[0047] The technical solution proposed in this application is based on the DeepLabv3+ network architecture, and combines the dilated spatial pyramid pooling module, the spatial attention module, and the feature alignment module to improve the generative adversarial network to obtain an improved generator network. The source image is input into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map. The discriminator network based on the fully convolutional structure replaces the standard convolution in the discriminator network with a dilated convolution to obtain an improved discriminator network. The semantic segmentation map is input into the improved discriminator network for discrimination, and the semantic segmentation result is obtained in combination with the unlabeled data of the source image. This application improves the generative adversarial network based on the dilated spatial pyramid pooling module, the spatial attention module, the feature alignment module, and the full convolution, which improves the generalization ability of the model and the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 A flowchart of the first embodiment of the semantic segmentation method of the present application is provided;

[0051] Figure 2 Schematic diagram of the semantic segmentation network structure;

[0052] Figure 3 Schematic diagram of the semantic segmentation process;

[0053] Figure 4 A flowchart of the second embodiment of the semantic segmentation method of the present application is provided;

[0054] Figure 5 This is a schematic diagram of the spatial attention module of this application;

[0055] Figure 6 This is a schematic diagram of the feature alignment module of this application;

[0056] Figure 7 A flowchart of the third embodiment of the semantic segmentation method of the present application is provided;

[0057] Figure 8 This is a schematic diagram of the improved discriminator network structure of this application;

[0058] Fig. 9This is a schematic diagram of the module structure of the semantic segmentation device according to an embodiment of the present application;

[0059] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the semantic segmentation method in the embodiment of the present application.

[0060] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0062] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0063] At present, the mainstream semantic segmentation method is the fully supervised learning method. The training effect of the semantic segmentation network depends too much on the quality and quantity of data, and the fully supervised method requires accurate pixel-by-pixel annotation of each training image, which consumes a lot of time. In addition, in the fully supervised learning method, although the fully supervised small sample image segmentation technology has developed in recent years, its dependence on a small amount of labeled data can easily lead to model overfitting, limiting the generalization ability and the complexity of feature capture.

[0064] Therefore, in order to overcome the above defects, the present application provides a solution to improve the generative adversarial network based on the void spatial pyramid pooling module, spatial attention module, feature alignment module and full convolution, thereby improving the model generalization ability and semantic segmentation accuracy.

[0065] It should be noted that the execution subject of each embodiment of the present application may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a semantic segmentation device, etc. The following embodiments are described below taking a semantic segmentation device as an example.

[0066] Based on this, the present application embodiment provides a semantic segmentation method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the semantic segmentation method of the present application.

[0067] In this embodiment, the semantic segmentation method includes steps S10 to S40:

[0068] Step S10, based on the DeepLabv3+ network architecture, the generative adversarial network is improved by combining the dilated spatial pyramid pooling module, the spatial attention module and the feature alignment module to obtain an improved generator network.

[0069] It should be noted that semantic segmentation refers to the classification of each pixel in the image so that each pixel is assigned a specific category label. This process is to divide the entire image into several regions without leaving any gaps. Each region represents a category, and the region without a category is usually assumed to be the background. The purpose of semantic segmentation is to enable computers to understand the objects and scenes in the image, thereby achieving a higher level of image analysis and understanding. In semantic segmentation, the classification result of each pixel is based on the contextual information around it and the global information of the entire image.

[0070] Semi-supervised semantic segmentation is a machine learning method between supervised learning and unsupervised learning. It uses a small amount of labeled data and a large amount of unlabeled data for training, aiming to improve the generalization ability and segmentation accuracy of the model. In traditional supervised semantic segmentation, a large amount of labeled data is required to train the model, but labeling data is often very time-consuming and expensive. Semi-supervised semantic segmentation improves the performance of the model by introducing unlabeled data and utilizing the inherent structure and features of this data.

[0071] Generative Adversarial Network (GAN) is a deep learning model consisting of two neural networks: Generator Network and Discriminator Network. These two networks are trained and optimized together through competition and cooperation to achieve the purpose of generating realistic data samples.

[0072] Typically, the task of the generator network is to generate realistic data samples from random noise, with the goal of deceiving the discriminator network into thinking that the generated data samples are real; the task of the discriminator network is to distinguish whether the input data samples are real or generated by the generator network, with the goal of accurately distinguishing between real data and generated data, that is, maximizing the probability of correctly classifying real data and the probability of incorrectly classifying generated data.

[0073] The "improved generator network" in this application is designed for semi-supervised semantic segmentation tasks. In the scenario of semi-supervised learning, the improved generator network first extracts features from the input source image (the labeled data of the source image), and uses deep separable convolution, dilated convolution, and multiple modules to process these features to generate the final semantic segmentation map. The "improved discriminator network" in this application is more focused on optimizing semantic segmentation tasks in semi-supervision, receiving the semantic segmentation map generated by the improved generator network as input, combining it with the unlabeled data of the source image, and training the improved generator network. Based on the trained generator network, the improved discriminator network is iteratively optimized.

[0074] In addition, it should be noted that the DeepLabv3+ network architecture is an advanced semantic segmentation network architecture, which mainly consists of two parts: encoder and decoder. The Atrous Spatial Pyramid Pooling (ASPP) module is used to capture multi-scale contextual information and enhance the network's ability to segment objects of different scales; the Spatial Attention (SA) module is used to enhance the network's ability to focus on important spatial locations in the image and improve segmentation accuracy; the Feature Alignment (FAM) module is used to solve the possible misalignment problem between feature maps and ensure the spatial and semantic consistency of feature maps.

[0075] The step S10 may include:

[0076] The DeepLabv3+ network architecture is used as the basic framework of the generative adversarial network, and the residual network is used as the backbone network of the basic framework to obtain the generator network;

[0077] Applying a dilated spatial pyramid pooling module to the generator network, and improving the parallel branches of the dilated spatial pyramid pooling module into a multi-branch series connection form;

[0078] A spatial attention module and a feature alignment module are applied to the generator network to obtain an improved generator network.

[0079] It should be noted that the residual network (Residual Network, ResNet) is a deep neural network architecture. Choosing the residual network as the backbone network of the basic framework can further improve the feature extraction capability.

[0080] In addition, it should be noted that the receptive field is an important concept in semi-supervised semantic segmentation. It refers to the size of the area where the pixels on the output feature map of a certain layer are mapped on the input image, that is, the size of the visual receptive area. It determines which parts of the input image the neurons of this layer can "see". The semantic segmentation task requires the model to classify each pixel, so the model needs to have a large enough receptive field to capture the contextual information around each pixel. In order to increase the receptive field range and effectively utilize the semantic information of each branch, the parallel branches of the atrous spatial pyramid pooling module are improved to a multi-branch series form.

[0081] For ease of understanding, refer to Figure 2 This invention is provided for illustration, but is not intended to limit the present application. Figure 2This is a schematic diagram of the semantic segmentation network structure. The multi-branch series-connected atrous spatial pyramid pooling module retains the 1x1 convolution branch and global average pooling branch of the original atrous spatial pyramid pooling module, and changes the three atrous convolution branches to serial connections to increase the receptive field range and enhance the correlation between each branch.

[0082] Step S20, inputting the source image into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map.

[0083] Step S30, based on the discriminator network with a fully convolutional structure, the standard convolution in the discriminator network is replaced with a dilated convolution to obtain an improved discriminator network.

[0084] To be clear, convolution is a mathematical operation that is used to combine arrays and functions. Fully convolutional usually refers to a neural network consisting of only convolutional layers, without any fully connected layers or multi-layer perceptrons that are usually found at the end of the network.

[0085] It should be understood that the use of dilated convolution in the discriminator network of a fully convolutional structure can not only reduce the amount of calculation and improve the training efficiency, but also has a good effect for scenes that need to maintain high-resolution information. By determining the standard convolution layer in the discriminator network, the identified standard convolution layer is replaced with a dilated convolution layer with the same number of input / output channels and convolution kernel size to obtain an improved discriminator network.

[0086] Step S40: input the semantic segmentation map into the improved discriminator network for discrimination, and obtain the semantic segmentation result in combination with the unlabeled data of the source image.

[0087] For ease of understanding, refer to Figure 3 This invention is provided for illustration, but is not intended to limit the present application. Figure 3 This is a schematic diagram of the semantic segmentation process. The input image first passes through the improved generator network (semantic segmentation network) for semi-supervised semantic segmentation to obtain a semantic segmentation map. Then, the semantic segmentation map is input into the improved discriminator network for discrimination to obtain a confidence map. Finally, the unlabeled data of the source image and the confidence map are combined to obtain the final true segmentation result.

[0088] It should be noted that the input image represents the original image to be semantically segmented; the semantic segmentation network is built based on the DeepLabv3+ network architecture, and is improved by combining the dilated spatial pyramid pooling module, the spatial attention module and the feature alignment module to form an improved generator network; the improved discriminator network is built based on the fully convolutional structure, and the standard convolution is replaced by the dilated convolution to form an improved discriminator network; the confidence map represents the confidence level of the discriminator network in the semantic segmentation map; the semantic segmentation result is obtained by combining the unlabeled data of the source image and the output of the improved discriminator network.

[0089] This embodiment is based on the DeepLabv3+ network architecture, and combines the atrous spatial pyramid pooling module, the spatial attention module, and the feature alignment module to improve the generative adversarial network to obtain an improved generator network. The source image is input into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map. Based on the discriminator network with a fully convolutional structure, the standard convolution in the discriminator network is replaced with a atrous convolution to obtain an improved discriminator network. The semantic segmentation map is input into the improved discriminator network for discrimination, and the semantic segmentation result is obtained in combination with the unlabeled data of the source image, thereby improving the generalization ability of the model and the accuracy of semantic segmentation.

[0090] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , the step S20 may include steps S201 to S204:

[0091] Step S201, inputting the labeled data of the source image into the improved generator network, and extracting features from the source image through the residual network.

[0092] It should be noted that in the field of semantic segmentation, source image data is usually divided into two categories: labeled data and unlabeled data. Labeled data refers to images that not only contain image content but also have precise pixel-level annotations. These annotations usually exist in the form of semantic segmentation masks, which clearly indicate the category to which each pixel in the image belongs (such as roads, buildings, trees, etc.). Unlabeled data refers to images that only contain image content without any annotation information.

[0093] It can be understood that the labeled data of the source image is input into the improved generator network, and at the same time, the source image is input into a pre-trained residual network, and the residual network will extract the features in the source image layer by layer.

[0094] Step S202: Process the features based on depthwise separable convolution to obtain secondary features, and concatenate multiple secondary features in the channel dimension to obtain fused features.

[0095] It should be noted that depthwise separable convolution (DS Conv) is an algorithm obtained by improving the standard convolution calculation in convolutional neural networks. It reduces the number of parameters required for convolution calculation by splitting the correlation between spatial dimensions and channel (depth) dimensions.

[0096] It should be understood that after the depth-wise separable convolution, the secondary features are obtained, which contain richer information and more fine-grained feature representation. Then, multiple secondary feature maps are concatenated in the channel dimension. This concatenation operation allows the information from different secondary feature maps to be used simultaneously, and this information can be further processed and fused through subsequent network layers.

[0097] Step S203, using the atrous spatial pyramid pooling module, the spatial attention module and the feature alignment module to process the source image to obtain global features.

[0098] It can be understood that the combination of the atrous spatial pyramid pooling module, the spatial attention module, and the feature alignment module can significantly enhance the processing capability of the source image and extract global features rich in contextual information and fine details.

[0099] Step S204: combining the fusion feature and the global feature to obtain a semantic segmentation map.

[0100] This embodiment inputs the labeled data of the source image into the improved generator network, extracts features from the source image through the residual network, processes the features based on deep separable convolution to obtain secondary features, splices multiple secondary features in the channel dimension to obtain fused features, uses a dilated spatial pyramid pooling module, a spatial attention module and a feature alignment module to process the source image to obtain global features, and combines the fused features and the global features to obtain a semantic segmentation map, thereby significantly enhancing the feature expression capability, improving the accuracy and robustness of semantic segmentation, and achieving efficient and accurate image segmentation.

[0101] In the second embodiment, step S203 may include:

[0102] Based on the atrous spatial pyramid pooling module, a global average pooling operation is performed on the source image to obtain a global feature vector;

[0103] Based on the spatial attention module, perform an average pooling operation and a maximum pooling operation on the source image to obtain an average pooling feature and a maximum pooling feature, and splice the average pooling feature and the maximum pooling feature according to the channel dimension to obtain a spliced ​​feature;

[0104] After performing a convolution operation on the concatenated features, a Sigmoid activation function is used to normalize the concatenated feature values ​​to obtain an attention weight map;

[0105] Based on the global feature vector and the attention weight map, combined with the feature alignment module, a global feature is obtained.

[0106] It can be understood that the dilated spatial pyramid pooling module uses dilated convolutions with different dilation rates to capture multi-scale contextual information. Then, a global average pooling operation is performed on these multi-scale features, that is, all pixel values ​​of each feature map are averaged to obtain a global feature vector of fixed length.

[0107] It should be understood that the average pooling and maximum pooling operations are performed on the source image to capture the average information and the most significant information in the image, namely the average pooling features and the maximum pooling features, respectively. The average pooling features and the maximum pooling features are spliced ​​in the channel dimension to form spliced ​​features. The spliced ​​features are convolved to fuse the information of different channels. Then, the sigmoid activation function is used to normalize the convolved feature values ​​so that their values ​​are between 0 and 1. The feature map obtained in this way is the attention weight map, which can reflect the importance of each spatial position in the source image.

[0108] After obtaining the global feature vector and attention weight map, they are combined. At the same time, in order to ensure that these features are consistent in the same semantic space, the features are further processed using a feature alignment module.

[0109] For ease of understanding, refer to Figure 5 This invention is provided for illustration, but is not intended to limit the present application. Figure 5 This is a schematic diagram of the spatial attention module of this application. The spatial attention module uses average pooling and maximum pooling to aggregate channel information, focusing on the global features and salient features of the feature map respectively. The input features first pass through two pooling modules to obtain the average pooling features and maximum pooling features on the channel, and then the average pooling features and maximum pooling features are spliced ​​in the channel dimension. Then, a 3×3 convolution fusion feature is used to reduce the channel dimension to 1. Finally, a Sigmoid activation function is used to normalize the feature attention weights and encode the positions to be emphasized or suppressed.

[0110] The step of obtaining the global feature based on the global feature vector and the attention weight map in combination with the feature alignment module includes:

[0111] Based on the feature alignment module, low-level features and high-level features of the source image are acquired, and the high-level features are up-sampled;

[0112] After the high-level features are upsampled, the high-level features are processed using the spatial attention module to obtain processed high-level features;

[0113] Splicing the processed high-level features with the low-level features to obtain fused features;

[0114] Performing branch processing on the fused features to obtain branch features, and predicting a two-dimensional offset of the branch features;

[0115] Aligning the processed high-level features and the low-level features through the two-dimensional offset to obtain an aligned feature map;

[0116] The global feature vector, the attention weight map and the aligned feature map are combined to obtain a global feature.

[0117] It should be understood that the feature alignment module can be used to extract low-level features and high-level features from the source image and upsample the high-level features, wherein the low-level features usually contain details and edge information of the image, while the high-level features contain more abstract semantic information.

[0118] The processed high-level features and low-level features are concatenated in the channel dimension to obtain a fused feature that combines low-level details and high-level semantic information. The fused features are then branched, that is, they are input into different network branches to obtain different feature representations. The two-dimensional offsets of these branch features are then predicted, and these offsets are used in subsequent feature alignment operations to ensure the spatial consistency of low-level features and high-level features.

[0119] It can be understood that aligning the processed high-level features with the low-level features using a two-dimensional offset usually involves spatially transforming the high-level features to match the spatial layout of the low-level features, and the aligned feature map contains both the detail information of the low-level features and the semantic information of the high-level features. Finally, the global feature vector, the attention weight map, and the aligned feature map are combined to obtain the final global feature.

[0120] For ease of understanding, refer to Figure 6 This invention is provided for illustration, but is not intended to limit the present application. Figure 6This is a schematic diagram of the feature alignment module of this application. The high-level features in the feature alignment module are upsampled and processed by the spatial attention module to obtain the high-level features F h , F h With low-level features F l Concat cascade is performed to establish the correlation between the two features. After that, it is divided into two branches after 1×1 convolution and batch normalization layer. Each branch uses 1×1 convolution to reduce the channel dimension to 2 to predict the two-dimensional offset Δ∈R of the branch feature. 2×H×W , the two dimensions of Δ represent the lateral and longitudinal offsets of the feature. The two-dimensional offset Δ output by the two branches h and Δ l It will be used to align high-level features and low-level features respectively, and the feature information after alignment (aligned feature map) is obtained through the alignment function U: out =U(F h ,Δh)+U(F l , Δ l ).

[0121] This embodiment performs a global average pooling operation on the source image based on a dilated spatial pyramid pooling module to obtain a global feature vector, performs an average pooling operation and a maximum pooling operation on the source image based on a spatial attention module to obtain average pooling features and maximum pooling features, and splices the average pooling features and the maximum pooling features according to the channel dimension to obtain spliced ​​features, performs a convolution operation on the spliced ​​features, and then uses a Sigmoid activation function to normalize the spliced ​​feature values ​​to obtain an attention weight map, and obtains global features based on the global feature vector and the attention weight map in combination with a feature alignment module, thereby enhancing the accuracy and robustness of the global features.

[0122] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 7 , the step S40 may include steps S401 to S404:

[0123] Step S401, inputting the semantic segmentation map into the improved discriminator network for discrimination to obtain a discrimination result.

[0124] The step S401 includes:

[0125] Inputting the semantic segmentation map into the improved discriminator network;

[0126] Based on the linear rectification function and the multi-level convolution layer parameters, the semantic segmentation map is subjected to a hole convolution process to obtain a processed semantic segmentation map;

[0127] Based on the upsampling layer of the improved discriminator network, the processed semantic segmentation map is rescaled to the source image size to obtain a discrimination result.

[0128] It should be noted that the rectified linear unit (ReLU) is a commonly used activation function in artificial neural networks, usually referring to a nonlinear function represented by a ramp function and its variants.

[0129] It can be understood that in the improved discriminator network, the input semantic segmentation map is firstly processed with dilated convolution using linear rectification function and multi-level convolution layer parameters, which can increase the receptive field of the convolution kernel without increasing the computational complexity. Through dilated convolution, richer and more abstract features can be extracted from the semantic segmentation map.

[0130] Subsequently, the upsampling layer in the improved discriminator network is used to rescale the semantic segmentation map after the dilated convolution process to the same size as the source image. The purpose is to generate a discriminant result map with the same spatial resolution as the source image, which is convenient for subsequent discrimination and visualization operations.

[0131] For ease of understanding, refer to Figure 8 This invention is provided for illustration, but is not intended to limit the present application. Figure 8 This is a schematic diagram of the improved discriminator network structure of this application. The improved discriminator network uses a fully convolutional structure, and changes the standard convolution in the convolution structure to a dilated convolution to expand the receptive field of the discriminator convolution layer. It consists of 5 convolutional layers, and the linear rectification function is combined in the process. The convolution kernel is 4×4, the number of channels is {64, 128, 256, 512, 1}, and the dilated convolution rate is {1, 2, 5, 1, 2}. The last layer adds an upsampling layer to rescale the output to the size of the input image.

[0132] Step S402, combining the unlabeled data of the source image and the discrimination result, training the improved generator network to obtain a trained generator network.

[0133] It should be understood that the improved generator network is trained using the unlabeled data of the source image and the discrimination results of the improved discriminator. The core of this training process is to generate semantic segmentation maps through the generator network and use the discriminator network to evaluate the quality of these segmentation maps, thereby guiding the parameter update of the generator network.

[0134] Step S403: Based on the trained generator network, iteratively optimize the improved discriminator network to obtain a trained discriminator network.

[0135] It should be understood that the trained generator network parameters are fixed, and then the data for training the discriminator network is prepared. These data include two parts: one is the real annotated semantic segmentation map, and the other is the semantic segmentation map generated by the generator network (i.e., "fake" data). The real data is used to teach the discriminator how to identify the real semantic segmentation, while the generated data is used to let the discriminator learn to distinguish the difference between the generated data and the real data.

[0136] After that, the discriminator network is iteratively trained. In each iteration, the real data and the generated data are alternately input into the discriminator network, and the authenticity score or probability of the discriminator for these data is calculated. Based on these scores and the true label (i.e., whether the data is real or generated), the discriminator's loss function is calculated, and the discriminator's parameters are updated through the back-propagation algorithm. During the training process, the performance of the discriminator needs to be evaluated regularly. If the performance of the discriminator reaches a preset threshold or stopping condition (such as loss convergence, the number of training rounds reaches the upper limit, etc.), the training process can be ended early. Finally, when the training process of the discriminator network ends, a trained discriminator network is obtained, which can more accurately identify the difference between the real and generated semantic segmentation maps.

[0137] Step S404, combining the trained generator network and the trained discriminator network to obtain a semantic segmentation result.

[0138] This embodiment obtains a discrimination result by inputting the semantic segmentation map into the improved discriminator network for discrimination, and trains the improved generator network in combination with the unlabeled data of the source image and the discrimination result to obtain a trained generator network. Based on the trained generator network, the improved discriminator network is iteratively optimized to obtain a trained discriminator network. The trained generator network and the trained discriminator network are combined to obtain a semantic segmentation result, thereby iteratively optimizing the generator and discriminator networks, effectively improving the generalization ability of the model, and further ensuring the accuracy and robustness of the semantic segmentation results.

[0139] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the semantic segmentation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0140] This application also provides a semantic segmentation device, please refer to Fig. 9 , the semantic segmentation device comprises:

[0141] The generator improvement module 10 is used to improve the generative adversarial network based on the DeepLabv3+ network architecture by combining the dilated spatial pyramid pooling module, the spatial attention module and the feature alignment module to obtain an improved generator network;

[0142] A generator processing module 20 is used to input the source image into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map;

[0143] A discriminator improvement module 30, which is used to replace the standard convolution in the discriminator network with a hole convolution based on the discriminator network of the fully convolution structure to obtain an improved discriminator network;

[0144] The result acquisition module 40 is used to input the semantic segmentation map into the improved discriminator network for discrimination, and obtain the semantic segmentation result in combination with the unlabeled data of the source image.

[0145] The semantic segmentation device provided by the present application adopts the semantic segmentation method in the above embodiment, which can solve the technical problems that the existing semantic segmentation technology is inefficient and easily leads to overfitting of the training model. Compared with the prior art, the beneficial effects of the semantic segmentation device provided by the present application are the same as the beneficial effects of the semantic segmentation method provided by the above embodiment, and the other technical features in the semantic segmentation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0146] The present application provides a semantic segmentation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the semantic segmentation method in the above-mentioned embodiment 1.

[0147] Reference below Fig.10 , which shows a schematic diagram of the structure of a semantic segmentation device suitable for implementing the embodiments of the present application. The semantic segmentation device in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Desctions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The semantic segmentation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0148] like Fig.10As shown, the semantic segmentation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the semantic segmentation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the semantic segmentation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a semantic segmentation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0149] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0150] The semantic segmentation device provided by the present application adopts the semantic segmentation method in the above embodiment, which can solve the technical problems that the existing semantic segmentation technology is inefficient and easily leads to overfitting of the training model. Compared with the prior art, the beneficial effects of the semantic segmentation device provided by the present application are the same as the beneficial effects of the semantic segmentation method provided by the above embodiment, and the other technical features in the semantic segmentation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0151] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0152] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0153] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the semantic segmentation method in the above-mentioned embodiment.

[0154] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0155] The computer-readable storage medium may be included in the semantic segmentation device; or may exist independently without being assembled into the semantic segmentation device.

[0156] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a semantic segmentation device, the semantic segmentation device: based on the DeepLabv3+ network architecture, improves the generative adversarial network in combination with the hole spatial pyramid pooling module, the spatial attention module and the feature alignment module to obtain an improved generator network, inputs the source image into the improved generator network for semi-supervised semantic segmentation, obtains a semantic segmentation map, replaces the standard convolution in the discriminator network with a hole convolution based on a discriminator network with a fully convolutional structure, obtains an improved discriminator network, inputs the semantic segmentation map into the improved discriminator network for discrimination, and obtains a semantic segmentation result in combination with the unlabeled data of the source image.

[0157] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0159] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0160] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned semantic segmentation method, and can solve the technical problems that the existing semantic segmentation technology is inefficient and easily leads to overfitting of the training model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the semantic segmentation method provided in the above-mentioned embodiment, and will not be repeated here.

[0161] The present application also provides a computer program product, including a computer program, which implements the steps of the semantic segmentation method as described above when executed by a processor.

[0162] The computer program product provided by this application can solve the technical problems that the existing semantic segmentation technology is inefficient and easily leads to overfitting of the training model. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the semantic segmentation method provided by the above embodiment, which will not be repeated here.

[0163] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present application.

Claims

1. A semantic segmentation method, characterized in that: The method comprises the following steps: Based on the DeepLabv3+ network architecture, the generative adversarial network is improved by combining the dilated spatial pyramid pooling module, spatial attention module and feature alignment module to obtain an improved generator network. Inputting the source image into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map; Based on a discriminator network with a fully convolutional structure, the standard convolution in the discriminator network is replaced with a hole convolution to obtain an improved discriminator network; The semantic segmentation map is input into the improved discriminator network for discrimination, and the semantic segmentation result is obtained by combining the unlabeled data of the source image.

2. The semantic segmentation method according to claim 1, characterized in that: The step of inputting the source image into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map comprises: Inputting the labeled data of the source image into the improved generator network, and extracting features from the source image through the residual network; Processing the features based on depthwise separable convolution to obtain secondary features, and concatenating multiple secondary features in the channel dimension to obtain fused features; Processing the source image using the atrous spatial pyramid pooling module, the spatial attention module, and the feature alignment module to obtain global features; The fusion feature and the global feature are combined to obtain a semantic segmentation map.

3. The semantic segmentation method according to claim 2, characterized in that: The step of using the atrous spatial pyramid pooling module, the spatial attention module and the feature alignment module to process the source image to obtain global features includes: Based on the atrous spatial pyramid pooling module, a global average pooling operation is performed on the source image to obtain a global feature vector; Based on the spatial attention module, perform an average pooling operation and a maximum pooling operation on the source image to obtain an average pooling feature and a maximum pooling feature, and splice the average pooling feature and the maximum pooling feature according to the channel dimension to obtain a spliced ​​feature; After performing a convolution operation on the concatenated features, a Sigmoid activation function is used to normalize the concatenated feature values ​​to obtain an attention weight map; Based on the global feature vector and the attention weight map, combined with the feature alignment module, a global feature is obtained.

4. The semantic segmentation method according to claim 3, characterized in that: The step of obtaining the global feature based on the global feature vector and the attention weight map in combination with the feature alignment module includes: Based on the feature alignment module, low-level features and high-level features of the source image are acquired, and the high-level features are up-sampled; After the high-level features are upsampled, the high-level features are processed using the spatial attention module to obtain processed high-level features; Splicing the processed high-level features with the low-level features to obtain fused features; Performing branch processing on the fused features to obtain branch features, and predicting a two-dimensional offset of the branch features; Aligning the processed high-level features and the low-level features through the two-dimensional offset to obtain an aligned feature map; The global feature vector, the attention weight map and the aligned feature map are combined to obtain a global feature.

5. The semantic segmentation method according to any one of claims 1 to 4, characterized in that: The step of improving the generative adversarial network based on the DeepLabv3+ network architecture and combining the dilated spatial pyramid pooling module, the spatial attention module and the feature alignment module to obtain an improved generator network includes: The DeepLabv3+ network architecture is used as the basic framework of the generative adversarial network, and the residual network is used as the backbone network of the basic framework to obtain the generator network; Applying a dilated spatial pyramid pooling module to the generator network, and improving the parallel branches of the dilated spatial pyramid pooling module into a multi-branch series connection form; A spatial attention module and a feature alignment module are applied to the generator network to obtain an improved generator network.

6. The semantic segmentation method according to any one of claims 1 to 4, characterized in that: The step of inputting the semantic segmentation map into the improved discriminator network for discrimination, and obtaining the semantic segmentation result in combination with the unlabeled data of the source image, comprises: Inputting the semantic segmentation map into the improved discriminator network for discrimination to obtain a discrimination result; Combining the unlabeled data of the source image and the discrimination result, training the improved generator network to obtain a trained generator network; Based on the trained generator network, iteratively optimize the improved discriminator network to obtain a trained discriminator network; The trained generator network and the trained discriminator network are combined to obtain a semantic segmentation result.

7. The semantic segmentation method according to claim 6, characterized in that: The step of inputting the semantic segmentation map into the improved discriminator network for discrimination to obtain a discrimination result comprises: Inputting the semantic segmentation map into the improved discriminator network; Based on the linear rectification function and the multi-level convolution layer parameters, the semantic segmentation map is subjected to a hole convolution process to obtain a processed semantic segmentation map; Based on the upsampling layer of the improved discriminator network, the processed semantic segmentation map is rescaled to the source image size to obtain a discrimination result.

8. A semantic segmentation device, characterized in that: The semantic segmentation device comprises: The generator improvement module is used to improve the generative adversarial network based on the DeepLabv3+ network architecture by combining the dilated spatial pyramid pooling module, the spatial attention module, and the feature alignment module to obtain an improved generator network; A generator processing module, used for inputting the source image into the improved generator network for semi-supervised semantic segmentation to obtain a semantic segmentation map; A discriminator improvement module, which is used for a discriminator network based on a fully convolutional structure, replacing standard convolutions in the discriminator network with hole convolutions to obtain an improved discriminator network; The result acquisition module is used to input the semantic segmentation map into the improved discriminator network for discrimination, and obtain the semantic segmentation result in combination with the unlabeled data of the source image.

9. A semantic segmentation device, characterized in that: The semantic segmentation device includes: a memory, a processor, and a semantic segmentation program stored in the memory and executable on the processor. When the semantic segmentation program is executed by the processor, the semantic segmentation method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a semantic segmentation program, which, when executed by a processor, implements the semantic segmentation method according to any one of claims 1 to 7.

Images