Remote sensing image semantic segmentation method and device based on semi-supervised adversarial learning

Abstract

The invention provides a remote sensing image semantic segmentation method and device based on semi-supervised adversarial learning, and the method comprises the steps: obtaining a reference data set of a remote sensing image, dividing the reference data set into a training set with a label and a training set without a label, constructing a segmentation network and a discriminator network, training the segmentation network through the training set with the label, and carrying out the recognition of the segmentation network. Obtaining a prediction map of the data in the training set without labels based on the trained segmentation network; training a discriminator network by adopting a training set without labels, and selecting a pseudo label based on a confidence score output by the discriminator network; performing self-training on the segmentation network based on the pseudo labels and the corresponding non-label images, and updating segmentation network parameters; through continuous iterative training, storing optimal network parameters as a remote sensing image semantic segmentation model; and performing remote sensing image semantic segmentation prediction by adopting the remote sensing image semantic segmentation model. The method can effectively improve the segmentation precision and reduce the data annotation cost under the condition of a small amount of annotation data.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image semantic segmentation method and device based on semi-supervised adversarial learning. Background Art

[0002] With the rapid development of remote sensing technology, remote sensing images are increasingly used in urban planning, environmental monitoring, agricultural management and other fields. In these applications, remote sensing image semantic segmentation technology plays a vital role, which can assign each pixel in the image to the corresponding object category, thereby realizing accurate recognition and management of objects in the image.

[0003] Traditional remote sensing image semantic segmentation methods mainly rely on manual features and classic machine learning algorithms such as support vector machines (SVM) and random forests (RF). However, these methods have limitations when processing high-resolution and multispectral remote sensing images, which are mainly manifested in the following aspects: (1) Feature engineering dependence: Traditional methods rely on features manually designed by experts, which often have difficulty capturing the complex and high-dimensional information of remote sensing images. (2) Limited generalization ability: Due to the lack of sufficient training samples, the generalization ability of traditional methods is limited when facing remote sensing images from different regions and under different conditions. (3) High cost of obtaining labeled data: High-quality labeled data is crucial for training deep learning models, but in the field of remote sensing, obtaining large-scale labeled data is costly and time-consuming.

[0004] In recent years, deep learning methods, especially convolutional neural networks (CNNs), have made significant progress in the field of image semantic segmentation. However, these methods usually require a large amount of labeled data to train the model, which remains a challenge in the field of remote sensing images. In addition, deep learning models are prone to overfitting during training, especially when there is limited labeled data.

[0005] Although semantic segmentation has made rapid progress, there are still many problems to be solved, such as high resource consumption, rough segmentation boundaries, context processing methods that need to be optimized, and difficulty in segmenting small objects. Therefore, how to use deep learning technology to make full use of the rich internal information of images to achieve more effective semantic segmentation still has important practical significance and application value. Summary of the invention

[0006] The purpose of the present invention is to provide a remote sensing image semantic segmentation method and device based on semi-supervised adversarial learning, which can use unlabeled data to generate pseudo labels to realize the semi-supervised learning process. These pseudo labels serve as additional supervision, effectively guiding the semi-supervised training of the network and enhancing the generalization ability of the model.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a remote sensing image semantic segmentation method based on semi-supervised adversarial learning, comprising:

[0009] Acquire a remote sensing image to be detected;

[0010] Inputting the remote sensing image to be detected into a pre-built remote sensing image semantic segmentation model to obtain a remote sensing image semantic segmentation result, wherein the remote sensing semantic segmentation result refers to ground object recognition and classification for each pixel in the remote sensing image;

[0011] The remote sensing image semantic segmentation model is constructed as follows:

[0012] Obtain a remote sensing image benchmark data set and divide it into a labeled training set and an unlabeled training set, wherein the label refers to the identification and classification of objects for each pixel in the remote sensing image;

[0013] Constructing a segmentation network and a discriminator network, wherein the segmentation network is used to extract the semantic information feature map of the input image and perform feature enhancement, and output the semantic segmentation result of the remote sensing image; the discriminator network is used to distinguish whether the input data comes from the real data annotation or the data annotation generated by the segmentation network;

[0014] Using the labeled training set to train the segmentation network, and obtaining a prediction graph of the data in the unlabeled training set based on the trained segmentation network;

[0015] Training the discriminator network based on the prediction graph of the data in the unlabeled training set and the labeled training set, and selecting a pseudo label based on a confidence score output by the discriminator network;

[0016] The segmentation network is self-trained based on the pseudo labels and the corresponding unlabeled images, and the segmentation network parameters are updated; the prediction graph of the data in the unlabeled training set is re-acquired based on the updated segmentation network; iterative training is performed using the above method, and the optimal segmentation network parameters are saved as a remote sensing image semantic segmentation model.

[0017] Preferably, before each iteration, the images in the training set are randomly cropped, randomly horizontally slid, randomly flipped, rotated, and color enhanced.

[0018] Preferably, the segmentation network includes:

[0019] The feature extraction module builds an encoder based on the ResNet-18 backbone network to extract features and extract the semantic information feature map of the input image; the network structure of the feature extraction module is divided into 5 layers, and the semantic information feature map extracted in each layer is ;

[0020] Feature enhancement module, which enhances the semantic information feature map based on spatial attention mechanism and channel attention mechanism;

[0021] The prediction module includes 5 upsampling operations, which are used to restore the enhanced feature map to the size of the original image through upsampling, and use 1×1 convolution to output the final prediction result.

[0022] Preferably, a cross entropy loss function is used in the segmentation network training process, and the segmentation network parameters are optimized by back propagation.

[0023] Preferably, the discriminator network includes 4 convolutional layers with a convolution kernel size of 4×4, each convolutional layer is followed by a Leaky ReLU activation function and a Dropout layer, and the last convolutional layer is connected to a global average pooling layer and a fully connected layer; finally, the confidence score is output through a Sigmoid activation function.

[0024] Preferably, a binary cross entropy loss function is used in the discriminator network training process, and the discriminator network parameters are optimized by back propagation.

[0025] Preferably, the selecting a pseudo label based on the confidence score output by the discriminator network comprises:

[0026] The prediction graph of the unlabeled training set data is passed through the discriminator network to obtain a confidence score, and the prediction graph with a score higher than the preset confidence threshold is selected as the pseudo label.

[0027] Preferably, the self-training the segmentation network based on the pseudo-labels and the corresponding unlabeled images comprises:

[0028] The pseudo labels and the corresponding unlabeled images are combined into training samples, and input into a segmentation network to generate a new prediction map of the unlabeled images;

[0029] The cross entropy loss between the generated prediction map and the pseudo label, i.e., the self-training loss, is calculated, and the segmentation network parameters are updated through back-propagation.

[0030] In a second aspect, the present invention provides a remote sensing image semantic segmentation device based on semi-supervised adversarial learning, which is used to implement the above-mentioned remote sensing image semantic segmentation method based on semi-supervised adversarial learning, and the device comprises:

[0031] A data acquisition module, used for acquiring remote sensing images to be detected;

[0032] Remote sensing image semantic segmentation model training module, used to train remote sensing image semantic segmentation model;

[0033] A detection output module, used for inputting the remote sensing image to be detected into the remote sensing image semantic segmentation model to obtain a remote sensing image semantic segmentation result, wherein the remote sensing semantic segmentation result refers to ground object recognition and classification for each pixel in the remote sensing image;

[0034] The training process of the remote sensing image semantic segmentation model training module is as follows:

[0035] Obtain a remote sensing image benchmark data set and divide it into a labeled training set and an unlabeled training set, wherein the label refers to the identification and classification of objects for each pixel in the remote sensing image;

[0036] Constructing a segmentation network and a discriminator network, wherein the segmentation network is used to extract the semantic information feature map of the input image and perform feature enhancement, and output the semantic segmentation result of the remote sensing image; the discriminator network is used to distinguish whether the input data comes from real data annotation or data annotation generated by the segmentation network;

[0037] Using the labeled training set to train the segmentation network, and obtaining a prediction graph of the data in the unlabeled training set based on the trained segmentation network;

[0038] Training the discriminator network based on the prediction graph of the data in the unlabeled training set and the labeled training set, and selecting a pseudo label based on a confidence score output by the discriminator network;

[0039] The segmentation network is self-trained based on the pseudo labels and the corresponding unlabeled images, and the segmentation network parameters are updated; the prediction map of the data in the unlabeled training set is re-acquired based on the updated segmentation network; iterative training is performed using the above method, and the optimal segmentation network parameters are saved as a remote sensing image semantic segmentation model.

[0040] In a third aspect, the present invention provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform any of the above-mentioned remote sensing image semantic segmentation methods based on semi-supervised adversarial learning.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. This invention adopts a semi-supervised learning strategy, which greatly reduces the dependence on a large amount of accurately labeled data. In traditional supervised learning, it is expensive and time-consuming to obtain a large amount of accurately labeled data. This method introduces unlabeled data and uses a semi-supervised learning mechanism to enable the model to learn effectively under limited labeled data conditions. This method not only reduces the cost of data preparation, but also improves the performance of the model when labeled data is scarce, especially in the field of remote sensing image processing. This advantage is particularly obvious. It can be used to improve the efficiency of extracting and identifying important ground object information in remote sensing images.

[0043] 2. The feature enhancement module of the present invention significantly enhances the robustness of the model to noise and interference by introducing spatial and channel attention mechanisms. In remote sensing images, some areas may be blocked by clouds, shadows or other objects, all of which may affect the accurate extraction of objects. The feature enhancement module improves the model's resistance to these interference factors by accurately identifying and weighting key features while suppressing irrelevant or noisy features. In this way, the model can stably output high-quality image extraction results even in the case of uneven image quality or complex environmental conditions. This robustness is crucial to improving the reliability and applicability of the model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the training process of the remote sensing image semantic segmentation prediction model in an embodiment of the present invention;

[0045] Figure 2 Schematic diagram of the structure of the segmentation network and the discriminator network in an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of the data processing process of the feature enhancement module in an embodiment of the present invention;

[0047] Figure 4 Schematic diagram of the discriminator network data processing process in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0049] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0050] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0051] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0052] It should be emphasized here that the step marks mentioned below are not intended to limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0053] The first embodiment of the present invention provides a remote sensing image semantic segmentation method based on semi-supervised adversarial learning, comprising:

[0054] Acquire a remote sensing image to be detected;

[0055] The remote sensing image to be detected is input into the pre-built remote sensing image semantic segmentation model to obtain the remote sensing image semantic segmentation result. The remote sensing semantic segmentation result refers to the identification and classification of ground objects for each pixel in the remote sensing image.

[0056] It should be noted that this embodiment performs semantic segmentation on high-resolution remote sensing images, such as images in the ISPRS Potsdam and GID datasets.

[0057] It should be noted that the remote sensing images to be detected are usually obtained through sensors carried on satellites, aircraft or drones. These sensors capture surface information based on imaging principles (such as photographic imaging, scanning imaging, radar imaging) and transmit the data to ground stations for processing and storage; visible light, panchromatic, multispectral and synthetic aperture radar remote sensing images can all be used for detection, and each type has its specific application range and advantages.

[0058] It should be noted that the achievement of semantic segmentation of remote sensing images is reflected in its ability to achieve pixel-by-pixel classification of remote sensing images. Specifically, semantic segmentation technology will produce a prediction map that is the same size as the original remote sensing image. This map performs detailed object identification and classification for each pixel in the image. In the prediction map, pixels of different colors represent different types of objects. For example, water bodies may be marked in blue, trees in green, buildings in red, and white is used to represent the background. Such a visual representation makes the identification and analysis of objects more intuitive and accurate.

[0059] In the embodiment of the present invention, a remote sensing image semantic segmentation model is pre-built, see Figure 1 The specific implementation process is as follows:

[0060] (1) Obtain a remote sensing image benchmark dataset and divide it into a labeled training set and an unlabeled training set. Labeling refers to the identification and classification of objects for each pixel in the remote sensing image.

[0061] (2) Construct the segmentation network and the discriminator network, see Figure 2 The segmentation network includes a feature extraction module, a feature enhancement module and a prediction module; the feature extraction module is used to extract the semantic information feature map of the input image; the feature enhancement module uses the spatial attention mechanism and the channel attention mechanism to achieve feature enhancement; the prediction module is used to obtain the prediction result based on the fused feature map, and the prediction result is the semantic segmentation result of the remote sensing image; the discriminator network consists of a convolutional layer, an activation function, a pooling layer and a fully connected layer, which is used to distinguish whether the input data comes from the real data annotation or the data annotation generated by the segmentation network;

[0062] (3) Input the labeled training set data into the segmentation network to generate prediction results; use the cross entropy loss function to accurately evaluate the difference between the predicted image and the true annotated image, and then calculate the supervision loss, and optimize the segmentation network parameters through back propagation;

[0063] (4) Input the unlabeled training set data into the segmentation network trained in step (3) to obtain the predicted image. The predicted image and the labeled training set are input into the discriminator network together. The adversarial loss is calculated using the binary cross entropy loss function. The discriminator network parameters are optimized through back propagation to improve the discriminator's ability to distinguish between true annotations and predicted image annotations.

[0064] (5) The prediction graph of the unlabeled training set data is passed through the discriminator network to obtain a confidence score, and the prediction graph with a score higher than the preset confidence threshold is selected as a pseudo-label for the subsequent self-training process to further improve the performance of the segmentation network;

[0065] (6) Use the pseudo-labels in step (5) and the corresponding unlabeled images to form training samples, and input them into the segmentation network to generate new prediction images. Calculate the cross entropy loss between the prediction image and the pseudo-label, i.e., the self-training loss, and back-propagate to update the segmentation network parameters to optimize the performance on unlabeled data;

[0066] (7) Repeat steps (4)-(6) until the model performance no longer improves or reaches the predetermined number of iterations, and save the optimal network parameters to obtain the remote sensing image semantic segmentation model;

[0067] (8) Input the test set into the remote sensing image semantic segmentation model to test the performance of the remote sensing image semantic segmentation model.

[0068] In this embodiment, the ISPRS Potsdam dataset is selected as the remote sensing image benchmark dataset.

[0069] It should be noted that before each iteration, the images in the training set and the test set are randomly cropped, randomly horizontally slid, randomly flipped, rotated, and color enhanced to enhance the generalization performance of the model.

[0070] It should be noted that by optimizing the feature matching loss and self-training loss between the prediction results and the true labels, and by using unlabeled data to enhance the training process, a network model with optimal parameters is obtained.

[0071] In one embodiment, a remote sensing image benchmark dataset is obtained and divided into a labeled training set and an unlabeled training set, as follows:

[0072] In remote sensing image semantic segmentation, labeled datasets refer to datasets that have been manually or automatically labeled and clearly define the categories to which each pixel in the image belongs. Labels are information used to mark the category to which each pixel or region in the image belongs, and are the key to achieving image semantic segmentation. In this solution, the labels of remote sensing datasets will clearly indicate the category to which each pixel in the image belongs. For example, in remote sensing images, different objects (such as roads, buildings, water bodies, vegetation, etc.) will be assigned different labels. These labels are usually expressed in the form of numbers or strings, and each number or string corresponds to a specific category.

[0073] Get the ISPRS Potsdam remote sensing image dataset ,Will Divide into training sets according to the ratio of 8:2 and test set ; The training set Select 10% of the data sample As a labeled training set, the remaining training set images are used as an unlabeled training set after removing the labels. .

[0074] In one embodiment, the segmentation network is based on an encoder-decoder structure, with ResNet-18 as the backbone network to construct an encoder for feature extraction, and the decoder still uses a convolutional form.

[0075] The feature extraction module is used to extract the semantic feature map of the input remote sensing image. Its structure is the feature extraction network ResNet-18. The specific data processing process is:

[0076] Take a set of input remote sensing images , the input size is , the network structure of the feature extraction module is divided into 5 layers, and the feature map that can be extracted in each layer is ;in , , Respectively represent The height, width, and number of channels of the feature map output by the feature extraction layer.

[0077] In one embodiment, the feature enhancement module performs feature enhancement through spatial and channel attention mechanisms, weighting the semantics related to the ground objects, thereby suppressing the image background noise. Figure 3 ,The data processing process of the feature enhancement module is as follows:

[0078] According to the feature map obtained in the encoder, it is represented as:

[0079] ,

[0080] in, , , Respectively represent The height, width, and number of channels of the feature map output by the feature extraction layer.

[0081] In this embodiment, =1,2,3,4. It should be noted that the feature map extracted from the fifth layer is directly input into the decoder, and the obtained feature map does not need to pass through the feature enhancement module.

[0082] 31) Generation of spatial attention map:

[0083] Feature Map First, pass a convolution block with a convolution kernel size of 3×3, and then perform maximum pooling and average pooling: , ,

[0084] in, and They represent the maximum pooling and average pooling of feature maps in the channel dimension respectively. and The shapes of ;

[0085] The feature map after the maximum pooling and average pooling operations is further modeled using a convolution with a convolution kernel size of 7×7. Finally, the Sigmoid function is used to generate a spatial attention map, which is expressed as:

[0086] ,

[0087] in, Indicates cascade operation, represents a 7×7 convolution operation, represents the Sigmoid activation function, The shape is ;

[0088] 32) Generation of channel attention map:

[0089] For feature maps Perform global average pooling, then use 1×1 convolution operation, and finally use Sigmoid function to generate channel attention map;

[0090] ,

[0091] in, represents a 1×1 convolution operation, represents the Sigmoid activation function, The shape is 1×1× ;

[0092] 33) Adoption and For feature maps Weighted to enhance the feature map, expressed as: , ,

[0093] in, Represents pixel-by-pixel multiplication.

[0094] These enhanced feature maps are sent to the corresponding part of the decoder, namely the prediction module, based on the idea of ​​skip connection to generate the final semantic segmentation results.

[0095] In one embodiment, the prediction module contains five upsampling operations, the feature map is restored to the size of the original image through upsampling, and a 1×1 convolution is used to output the final prediction result.

[0096] In one embodiment, during the segmentation network training process, the cross entropy loss function is used as follows:

[0097] ,

[0098] Where: and They are the original image and the real label of the labeled image respectively; is the prediction output of the segmentation network.

[0099] In this embodiment, the discriminator network receives the predicted image of the segmentation network and the real label image, that is, the labeled training set. Its task is to distinguish the difference between the predicted image and the real label image, which helps to optimize the performance of the segmentation network. The discriminator network consists of 4 convolutions with a kernel size of 4×4. Each convolution layer is followed by a Leaky ReLU activation function and a Dropout layer. The last convolution layer is connected to a global average pooling layer and a fully connected layer.

[0100] See also Figure 4, the specific implementation process of the discriminator network is as follows:

[0101] 41) The discriminator network receives as input the concatenation of the prediction map from the segmentation network and the labeled image. The shapes of the prediction map and the labeled image are , so the input size is .

[0102] 42) The discriminator network consists of multiple convolutional layers. Specifically, it contains 4 convolutional layers with a convolution kernel size of 4×4 and an output size of .

[0103] 43) After each convolutional layer, apply the Leaky ReLU activation function to increase the nonlinearity of the network. After activation, the size remains unchanged and the output size is still .

[0104] The activation function formula is as follows:

[0105] ,

[0106] Where α is a small positive number, usually 0.01 or 0.02. Leaky ReLU allows negative inputs to have a non-zero gradient, which helps alleviate the gradient vanishing problem.

[0107] 44) In addition, each convolutional layer is followed by a Dropout layer, which randomly discards the output of some neurons to reduce overfitting. .

[0108] 45) After the last convolutional layer, a global average pooling layer and a fully connected layer are connected. The feature map passes through the global average pooling layer to aggregate the spatial information of each channel, and the output size is ; The feature vector after global average pooling is input to the fully connected layer for final decision making, and the output size is .

[0109] 46) Finally, the Sigmoid activation function outputs a value between 0 and 1, indicating the probability that the input is the true label. The output size is , a single confidence score that evaluates how close the predicted image is to the true label image. This score is used to calculate the feature matching loss, which encourages the generator to produce predicted images that are closer to the true label.

[0110] 47) The prediction result of the segmentation network will output a confidence score after passing through the discriminator network. This score is used as a measure of the quality of the predicted image. The confidence score is greater than the predefined confidence threshold. The prediction map will be used as the pseudo label of the training segmentation network for supervised training. The self-training loss calculation formula is as follows:

[0111] ,

[0112] In the formula, is the confidence threshold; represents a pseudo label; is the prediction map generated by the segmentation network is the confidence score of the prediction graph.

[0113] 48) The training goal of the discriminator is to distinguish whether the input is a prediction result or a real label. Use the original GAN ​​loss for training, the formula is:

[0114] ,

[0115] Where D represents the discriminator, Represents a labeled data pair, represents unlabeled data, S represents the segmentation network, Represents a concatenation operation on the channel dimension.

[0116] Based on the above inventive concept, the first embodiment of the present invention provides a remote sensing image semantic segmentation device based on semi-supervised adversarial learning, which is used to implement the above remote sensing image semantic segmentation method based on semi-supervised adversarial learning. The device includes:

[0117] A data acquisition module, used for acquiring remote sensing images to be detected;

[0118] A remote sensing image semantic segmentation model training module is used to train a remote sensing image semantic segmentation model, which includes a segmentation network and a discriminator network. The segmentation network includes a feature extraction module, a feature enhancement module, and a prediction module. The feature extraction module is used to extract the semantic information feature map of the input image. The feature enhancement module uses a spatial attention mechanism and a channel attention mechanism to achieve feature enhancement. The prediction module is used to obtain a prediction result based on the fused feature map, and the prediction result is the semantic segmentation result of the remote sensing image. The discriminator network consists of a convolutional layer, an activation function, a pooling layer, and a fully connected layer, which is used to distinguish whether the input data comes from real data annotations or data annotations generated by the segmentation network.

[0119] The training process of the remote sensing image semantic segmentation model is as follows:

[0120] (1) Obtain a remote sensing image benchmark dataset and divide it into a labeled training set and an unlabeled training set. Labeling refers to the identification and classification of objects for each pixel in the remote sensing image.

[0121] (2) Construct segmentation network and discriminator network;

[0122] (3) Input the labeled training set data into the segmentation network to generate prediction results; use the cross entropy loss function to accurately evaluate the difference between the predicted image and the true annotated image, and then calculate the supervision loss, and optimize the segmentation network parameters through back propagation;

[0123] (4) Input the unlabeled training set data into the trained segmentation network to obtain the prediction map, which is consistent with the labeled training set Figure 1 And input into the discriminator network, use the binary cross entropy loss function to calculate the adversarial loss, optimize the discriminator network parameters through back propagation, and improve the discriminator's ability to distinguish between real annotations and predicted image annotations;

[0124] (5) The prediction graph of the unlabeled training set data is passed through the discriminator network to obtain a confidence score, and the prediction graph with a score higher than the preset confidence threshold is selected as a pseudo-label for the subsequent self-training process to further improve the performance of the segmentation network;

[0125] (6) Use the pseudo-labels in step (5) and the corresponding unlabeled images to form training samples, and input them into the segmentation network to generate new prediction images. Calculate the cross entropy loss between the prediction image and the pseudo-label, i.e., the self-training loss, and back-propagate to update the segmentation network parameters to optimize the performance on unlabeled data;

[0126] (7) Repeat steps (4)-(6) until the model performance no longer improves or reaches the predetermined number of iterations, and save the optimal network parameters to obtain the remote sensing image semantic segmentation model;

[0127] (8) Input the test set into the remote sensing image semantic segmentation model to test the performance of the remote sensing image semantic segmentation model.

[0128] The device also includes a detection output module, which is used to input the remote sensing image to be detected into a pre-built remote sensing image semantic segmentation model to obtain the remote sensing image semantic segmentation result, wherein the remote sensing semantic segmentation result refers to the identification and classification of ground objects for each pixel in the remote sensing image.

[0129] The specific working process of each module described above can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. The module division is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system.

[0130] Based on the same inventive concept, the present invention provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the above-mentioned remote sensing image semantic segmentation method based on semi-supervised adversarial learning are implemented.

[0131] Based on the same inventive concept, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned remote sensing image semantic segmentation method based on semi-supervised adversarial learning.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0133] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A remote sensing image semantic segmentation method based on semi-supervised adversarial learning, characterized in that: include: Acquire a remote sensing image to be detected; Inputting the remote sensing image to be detected into a pre-built remote sensing image semantic segmentation model to obtain a remote sensing image semantic segmentation result, wherein the remote sensing semantic segmentation result refers to ground object recognition and classification for each pixel in the remote sensing image; The remote sensing image semantic segmentation model is constructed as follows: Obtain a remote sensing image benchmark data set and divide it into a labeled training set and an unlabeled training set, wherein the label refers to the identification and classification of objects for each pixel in the remote sensing image; Constructing a segmentation network and a discriminator network, wherein the segmentation network is used to extract the semantic information feature map of the input image and perform feature enhancement, and output the semantic segmentation result of the remote sensing image; the discriminator network is used to distinguish whether the input data comes from real data annotation or data annotation generated by the segmentation network; Using the labeled training set to train the segmentation network, and obtaining a prediction graph of the data in the unlabeled training set based on the trained segmentation network; Training the discriminator network based on the prediction graph of the data in the unlabeled training set and the labeled training set, and selecting a pseudo label based on a confidence score output by the discriminator network; The segmentation network is self-trained based on the pseudo labels and the corresponding unlabeled images, and the segmentation network parameters are updated; the prediction map of the data in the unlabeled training set is re-acquired based on the updated segmentation network; iterative training is performed using the above method, and the optimal segmentation network parameters are saved as a remote sensing image semantic segmentation model.

2. The method for semantic segmentation of remote sensing images based on semi-supervised adversarial learning according to claim 1, characterized in that: Before each iteration, the images in the training set are randomly cropped, randomly horizontally slid, randomly flipped, rotated, and color enhanced.

3. The method for semantic segmentation of remote sensing images based on semi-supervised adversarial learning according to claim 1, characterized in that: The segmentation network includes: The feature extraction module builds an encoder based on the ResNet-18 backbone network to perform feature extraction and extract the semantic information feature map of the input image; the network structure of the feature extraction module is divided into 5 layers, and the semantic information feature map extracted in each layer is ; Feature enhancement module, which enhances the semantic information feature map based on spatial attention mechanism and channel attention mechanism; The prediction module includes 5 upsampling operations, which are used to restore the enhanced feature map to the size of the original image through upsampling, and use 1×1 convolution to output the final prediction result.

4. The method for semantic segmentation of remote sensing images based on semi-supervised adversarial learning according to claim 3, characterized in that: The cross entropy loss function is used in the segmentation network training process, and the segmentation network parameters are optimized through back propagation.

5. The method for semantic segmentation of remote sensing images based on semi-supervised adversarial learning according to claim 1, characterized in that: The discriminator network includes 4 convolutional layers with a convolution kernel size of 4×4, each convolutional layer is followed by a Leaky ReLU activation function and a Dropout layer, and the last convolutional layer is connected to a global average pooling layer and a fully connected layer; finally, the confidence score is output through the Sigmoid activation function.

6. The method for semantic segmentation of remote sensing images based on semi-supervised adversarial learning according to claim 5, characterized in that: The binary cross entropy loss function is used in the discriminator network training process, and the discriminator network parameters are optimized through back propagation.

7. The method for semantic segmentation of remote sensing images based on semi-supervised adversarial learning according to claim 1, characterized in that: The selecting a pseudo label based on the confidence score output by the discriminator network comprises: The prediction graph of the unlabeled training set data is passed through the discriminator network to obtain a confidence score, and the prediction graph with a score higher than the preset confidence threshold is selected as the pseudo label.

8. The remote sensing image semantic segmentation method based on semi-supervised adversarial learning according to claim 7, characterized in that: The self-training of the segmentation network based on the pseudo-labels and the corresponding unlabeled images comprises: The pseudo labels and the corresponding unlabeled images are combined into training samples, and input into a segmentation network to generate a new prediction map of the unlabeled images; The cross entropy loss between the generated prediction map and the pseudo label, i.e., the self-training loss, is calculated, and the segmentation network parameters are updated through back-propagation.

9. A remote sensing image semantic segmentation device based on semi-supervised adversarial learning, characterized in that: The device is used to implement the remote sensing image semantic segmentation method based on semi-supervised adversarial learning according to any one of claims 1 to 8, comprising: A data acquisition module, used for acquiring remote sensing images to be detected; Remote sensing image semantic segmentation model training module, used to train remote sensing image semantic segmentation model; A detection output module, used for inputting the remote sensing image to be detected into the remote sensing image semantic segmentation model to obtain a remote sensing image semantic segmentation result, wherein the remote sensing semantic segmentation result refers to ground object recognition and classification for each pixel in the remote sensing image; The training process of the remote sensing image semantic segmentation model training module is as follows: Obtain a remote sensing image benchmark data set and divide it into a labeled training set and an unlabeled training set, wherein the label refers to the identification and classification of objects for each pixel in the remote sensing image; Constructing a segmentation network and a discriminator network, wherein the segmentation network is used to extract the semantic information feature map of the input image and perform feature enhancement, and output the semantic segmentation result of the remote sensing image; the discriminator network is used to distinguish whether the input data comes from real data annotation or data annotation generated by the segmentation network; Using the labeled training set to train the segmentation network, and obtaining a prediction graph of the data in the unlabeled training set based on the trained segmentation network; Training the discriminator network based on the prediction graph of the data in the unlabeled training set and the labeled training set, and selecting a pseudo label based on a confidence score output by the discriminator network; The segmentation network is self-trained based on the pseudo labels and the corresponding unlabeled images, and the segmentation network parameters are updated; the prediction map of the data in the unlabeled training set is re-acquired based on the updated segmentation network; iterative training is performed using the above method, and the optimal segmentation network parameters are saved as a remote sensing image semantic segmentation model.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods for remote sensing image semantic segmentation based on semi-supervised adversarial learning according to claims 1 to 8.

Images