Cross-consistency self-training remote sensing image semantic segmentation network training method and device

Abstract

The present application discloses a cross-consistency self-training remote sensing image semantic segmentation network training method and device, which relates to the field of remote sensing image segmentation technology. The specific implementation scheme is: constructing a remote sensing image semantic segmentation network, the remote sensing semantic segmentation network is UNet; training the remote sensing image semantic segmentation network according to a remote sensing image sample data set; inputting the remote sensing image captured in real time into the trained remote sensing image semantic segmentation network, and outputting the semantic segmentation result of the remote sensing image. The embodiment of the present application realizes the segmentation of the remote sensing image by constructing a remote sensing image semantic segmentation network and training the remote sensing image semantic segmentation network according to a remote sensing image sample data set. The embodiment of the present application can avoid the large amount of manpower consumed in remote sensing image segmentation and annotation, improve the efficiency of training, and improve the accuracy of remote sensing image segmentation.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image segmentation, and in particular to a cross-consistency self-training remote sensing image semantic segmentation network training method and device. Background Art

[0002] Image segmentation refers to the technology and process of dividing an image into regions with distinct characteristics and extracting targets of interest. Remote sensing images are a common and typical type of color image and have received widespread attention. Using image segmentation technology to label pixels in remote sensing images as a type of land feature, such as buildings, water bodies, roads, farmland, vehicles, etc., has always been a research hotspot for many scholars. Traditional image segmentation methods (such as thresholding, k-Means clustering, region methods, and edge detection methods) only care about finding the boundary contours of land features, regardless of the category to which the land features belong. In recent years, thanks to the rapid development of deep learning and the significant improvement in computer storage and computing power, semantic segmentation methods based on deep convolutional neural networks have become a new tool for high-resolution remote sensing image segmentation.

[0003] However, semantic segmentation based on deep convolutional neural networks can be considered as a pixel-level classification task, which requires dense annotation of ground objects in the image. The annotation is more difficult, especially for high-resolution remote sensing images. The annotation of semantic segmentation datasets requires a lot of manpower and time, which brings great difficulties to the semantic segmentation of remote sensing images based on deep convolutional neural networks. Summary of the Invention

[0004] This application provides a cross-consistency self-training remote sensing image semantic segmentation network training method and device. The technical solution of this application is as follows:

[0005] According to a first aspect of an embodiment of the present application, a cross-consistency self-training remote sensing image semantic segmentation network training method is provided, comprising:

[0006] Constructing a remote sensing image semantic segmentation network, wherein the remote sensing semantic segmentation network is UNet;

[0007] Training the remote sensing image semantic segmentation network according to a remote sensing image sample dataset;

[0008] The remote sensing image captured in real time is input into the trained remote sensing image semantic segmentation network, and the semantic segmentation result of the remote sensing image is output.

[0009] Optionally, the training of the remote sensing image semantic segmentation network according to the remote sensing image dataset includes:

[0010] Acquire the remote sensing image sample data set, wherein the remote sensing image sample data set includes remote sensing images;

[0011] Labeling the ground object categories of pixels in the remote sensing image to generate corresponding labeled remote sensing images;

[0012] Preprocessing the remote sensing image sample dataset to obtain a training set, a validation set, and a test set;

[0013] Construct a remote sensing image semantic segmentation network consisting of an encoder, a main decoder, and multiple auxiliary decoders;

[0014] The training set is divided into labeled samples and unlabeled samples, and input into the remote sensing image semantic segmentation network for training.

[0015] Optionally, inputting the remote sensing image into the semantic segmentation network for training includes:

[0016] Determining hyperparameters and loss functions used in training the remote sensing image semantic segmentation network;

[0017] Optimizing the parameters of the remote sensing image semantic segmentation network until the prediction accuracy of the validation set prediction results reaches a preset accuracy threshold;

[0018] The test set is input into the trained remote sensing image semantic segmentation network to verify the network segmentation accuracy.

[0019] Optionally, the training of the remote sensing semantic segmentation network includes:

[0020] The remote sensing images in the training set are divided into labeled samples and unlabeled samples, wherein the labeled samples include remote sensing images and corresponding labeled remote sensing images, and the unlabeled samples only include remote sensing images;

[0021] In the first training phase, the labeled samples and unlabeled samples are input into the encoder of the remote sensing semantic segmentation network. The encoder extracts the labeled sample feature map and inputs it into the main decoder. The obtained prediction results and the label data are used to calculate the supervision loss.

[0022] In the first part of the second training phase, the unlabeled sample feature maps extracted by the encoder are randomly transformed and input into the main decoder and the auxiliary decoder, and the consistency loss is calculated based on the prediction results of the auxiliary decoder and the prediction results of the main decoder;

[0023] In the second part of the second training phase, the prediction results of the main decoder and the auxiliary decoder are fused to obtain pseudo labels, and the pseudo labels and corresponding remote sensing images are used as labeled samples for supervised training.

[0024] Optionally, preprocessing the remote sensing image sample dataset includes:

[0025] Randomly sampling remote sensing images in the remote sensing image sample dataset into small images for multiple rounds of batch training, wherein the sampling frequency of each round of training is 480;

[0026] Set the sampling size and training batch value of the remote sensing image according to the size of the video memory space. The default input image size is 512×512, and the default training batch size is 10.

[0027] Performing multiple rounds of random sampling on the remote sensing image and the labeled remote sensing image, wherein each sampling obtains a 512×512 sized labeled remote sensing image and a corresponding label map, as well as an unlabeled image;

[0028] Data enhancement is performed on samples of each round of sampling with a random number and degree of randomness, and a training sample enhancement parameter range is set, wherein the data enhancement includes at least one of the following: random rotation of n×90° (n=0, 1, 2, 3); random horizontal or vertical 180° flip; random scale scaling, the scale scaling multiple is in the range of [0.5, 2]; random brightness enhancement, the brightness enhancement multiple is in the range of [0.5, 2]; random contrast enhancement, the contrast enhancement multiple is in the range of [0.5, 2]; random saturation enhancement, the saturation enhancement multiple is in the range of [0.5, 2].

[0029] Optionally, the remote sensing image semantic segmentation network is a classic semantic segmentation network UNet, and the UNet includes an encoder and a main decoder, and the number of auxiliary decoders is 3.

[0030] Optionally, the strategies for changing the learning rate of the hyperparameters and loss function during training include:

[0031]

[0032] Among them, base_lr = 4.2e-6; warmup_factor = 1.2; epoch is the number of training iterations; lr is the learning rate, which gradually increases with time; warm_epoch is the number of warm start iterations, and warm_epoch is set to 30;

[0033] Perform a warm start. When the training exceeds warm_epoch, use the polynomial learning rate decay strategy, set the maximum number of training iterations maxlr to 1500, the decay exponent pewer to 0.9, and the maximum learning rate maxlr to 1e-3.

[0034] The loss function used to calculate the supervision loss is the cross entropy loss function, the loss function used to calculate the consistency loss is the mean square error loss function, and the overall loss function of the remote sensing image semantic segmentation network is for:

[0035]

[0036] is the prediction result; y i To label images; To monitor losses; m The prediction result of the main decoder; e a is the prediction result of the auxiliary decoder; is the consistency loss; λ(epoch) is The weight of .

[0037] In the first training stage, the supervision loss partially dominates the overall loss;

[0038] In the second training phase, the overall loss is dominated by the consistency loss, and the specific formula of λ(epoch) is:

[0039]

[0040] Where num_epochs is the total number of training iterations. The training round threshold a is set to 200. λ(epoch) gradually increases before the ath training iteration. After training for more than a times, λ(epoch) stabilizes to w. num_epochs is set to 1500, and w is set to 1.

[0041] Optionally, the random transformation of the unlabeled sample feature map includes:

[0042] Dropout = 0.5, randomly flip 180° horizontally and vertically, and add Gaussian noise N~U(-0.2, 0.2) that follows a uniform distribution.

[0043] Optionally, fusing the prediction results of the primary decoder and the auxiliary decoder includes:

[0044] The prediction results of the main decoder and the prediction results of the auxiliary decoder are voted according to positions, and the category with the most votes for each pixel is taken as the final fusion result to generate the pseudo label.

[0045] According to a second aspect of an embodiment of the present application, a cross-consistency self-training remote sensing image semantic segmentation network training device is provided, comprising:

[0046] A construction module is used to construct a remote sensing image semantic segmentation network, wherein the remote sensing semantic segmentation network is UNet;

[0047] A training module, configured to train the remote sensing image semantic segmentation network based on a remote sensing image sample dataset;

[0048] The segmentation module is used to input the remote sensing image captured in real time into the trained remote sensing image semantic segmentation network and output the semantic segmentation result of the remote sensing image.

[0049] According to a third aspect of an embodiment of the present application, a cross-consistency self-training remote sensing image semantic segmentation network training device is provided, characterized by comprising:

[0050] processor;

[0051] a memory for storing instructions executable by the processor;

[0052] The processor is configured to execute the instructions to implement the method as described in any one of the first aspects above.

[0053] According to the fourth aspect of an embodiment of the present application, a non-temporary computer-readable storage medium is provided, characterized in that when the instructions in the storage medium are executed by a processor of a cross-consistency self-training remote sensing image semantic segmentation network training device, the cross-consistency self-training remote sensing image semantic segmentation network training device is enabled to perform a method as described in any one of the above-mentioned first aspects.

[0054] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0055] In the second training phase, a small amount of labeled data and a large amount of unlabeled data are used. By enhancing the data and applying consistency loss to the outputs of the primary and auxiliary decoders, the model is effectively prevented from overfitting and its generalization ability is improved. The output results of all decoders are fused to obtain pseudo-labeled images, which are used with the corresponding remote sensing images as labeled samples for supervised training, further improving the performance of the model. Therefore, when labeled data is insufficient, the present invention can use a large amount of unlabeled data to train a model with better performance, reducing the need for labeled samples and lowering the labor cost of data annotation.

[0056] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0058] Figure 1 The present invention is a flowchart of a cross-consistency self-training remote sensing image semantic segmentation network training method according to an exemplary embodiment.

[0059] Figure 2The present invention is a flowchart of a cross-consistency self-training remote sensing image semantic segmentation network training method according to an exemplary embodiment.

[0060] Figure 3 The present invention is a flowchart of a cross-consistency self-training remote sensing image semantic segmentation network training method according to an exemplary embodiment.

[0061] Figure 4 The present invention is a block diagram of a cross-consistency self-training remote sensing image semantic segmentation network training device according to an exemplary embodiment.

[0062] Figure 5 The present invention is a block diagram of a remote sensing image semantic segmentation model training system using cross-consistency learning according to an exemplary embodiment.

[0063] Figure 6 The present invention is a block diagram of a remote sensing image semantic segmentation model training system using cross-consistency learning according to an exemplary embodiment.

[0064] Figure 7 It is a block diagram of a device according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] In order to enable ordinary people in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0066] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0067] The goal of image segmentation is to label each pixel as a category. For remote sensing images, this involves labeling pixels as a type of feature, such as buildings, water bodies, roads, farmland, vehicles, and so on. Image semantic segmentation has evolved from traditional image segmentation methods. Traditional image segmentation methods (thresholding, k-Means clustering, region analysis, and edge detection) focus solely on finding the boundary contours of features, regardless of their category. Semantic segmentation not only accurately locates the outlines of features but also accurately determines their category, i.e., assigns them semantic meaning. Thanks to the rapid development of deep learning and the significant increase in computer storage and computing power, semantic segmentation methods based on deep convolutional neural networks have become a new and powerful tool for high-resolution remote sensing image segmentation.

[0068] Semantic segmentation based on deep convolutional neural networks can be considered as a pixel-level classification task, which requires dense labeling of ground objects in the image. Labeling is more difficult, especially for high-resolution remote sensing images. The labeling of semantic segmentation datasets requires a lot of manpower and time, which brings great difficulties to the semantic segmentation of remote sensing images based on deep convolutional neural networks.

[0069] There are two solutions in the related art: 1. The self-training method consists of three steps: First, train a model on labeled data. Second, use the pre-trained model to generate pseudo-labels for the unlabeled dataset. Third, retrain the model using the true labels from the labeled dataset and the pseudo-labels from the unlabeled dataset. Fourth, repeat this process several times. This method is effective, but repeated training increases the time cost.

[0070] 2. The consistency learning-based method encourages the model to produce similar outputs for the same sample after different transformations, such as random rotation, flipping, and color changes. This entire process is performed simultaneously with supervised training, which is more effective, but there is still room for improvement.

[0071] Figure 1 This is a flow chart of a cross-consistency self-training remote sensing image semantic segmentation network training method according to an exemplary embodiment. Figure 1 As shown, the method is used in remote sensing image semantic segmentation and includes the following steps.

[0072] Step 101, constructing a remote sensing image semantic segmentation network, wherein the remote sensing semantic segmentation network is UNet;

[0073] In an embodiment of the present application, a remote sensing image semantic segmentation network is constructed. The remote sensing image semantic segmentation network is UNet. UNet consists of two parts: an encoder and a decoder. The first half of the encoder is used for feature extraction, and the second half of the decoder is used for upsampling.

[0074] Step 102: training the remote sensing image semantic segmentation network based on the remote sensing image sample dataset;

[0075] In this embodiment, after constructing the remote sensing image semantic segmentation network, the network is trained based on the collected remote sensing image dataset. The training process is divided into a first training phase and a second training phase. In the first training phase, the supervision loss partially dominates the overall loss; in the second training phase, the consistency loss dominates the overall loss.

[0076] Step 103: input the remote sensing image captured in real time into the trained remote sensing image semantic segmentation network, and output the semantic segmentation result of the remote sensing image.

[0077] In an embodiment of the present application, after the remote sensing image semantic segmentation network training is completed, the remote sensing image captured in real time can be input into the remote sensing image semantic segmentation network to obtain the semantic segmentation result of the remote sensing image.

[0078] In the second training phase, the embodiment of the present application utilizes a small amount of labeled data and a large amount of unlabeled data. By enhancing the data and seeking consistency loss for the outputs of the main decoder and the auxiliary decoder, the model is effectively prevented from overfitting and the generalization ability of the model is improved. The output results of all decoders are fused to obtain pseudo-labeled images, which are used as labeled samples for supervised training with the corresponding remote sensing images, which can further improve the performance of the model. Therefore, the present invention can use a large amount of unlabeled data to train a model with better performance when labeled data is insufficient, reduce the demand for labeled samples, and reduce the labor cost of data annotation.

[0079] Figure 2 This is a flow chart of a cross-consistency self-training remote sensing image semantic segmentation network training method according to an exemplary embodiment. Figure 2 As shown, the method is used in remote sensing image semantic segmentation and includes the following steps.

[0080] Step 201: Acquire the remote sensing image sample dataset, wherein the remote sensing image sample dataset includes remote sensing images;

[0081] In the embodiment of the present application, it is necessary to collect remote sensing images as sample data to train the remote sensing image semantic segmentation model, and obtain historical remote sensing data from the database to form the remote sensing image sample data set.

[0082] Step 202: labeling the ground object categories of the pixels in the remote sensing image to generate a corresponding labeled remote sensing image;

[0083] In an embodiment of the present application, the task of the remote sensing image semantic segmentation model is to identify and label the object categories to which pixels in the remote sensing image belong. In a possible embodiment, the object categories include: buildings, water bodies, roads, cultivated land, and vehicles.

[0084] Step 203: preprocessing the remote sensing image sample dataset to obtain a training set, a validation set, and a test set;

[0085] In an embodiment of the present application, in order to improve the efficiency of training and reduce the workload of manually labeling the object categories, it is necessary to preprocess the remote sensing images in the remote sensing image sample data set. The remote sensing images are sampled into the same size, the object categories of the pixels are labeled, and random data enhancement is performed on the sampled images. The remote sensing image sample data set is divided into a training set, a validation set, and a test set. The training set is used to train the remote sensing image semantic segmentation model; the validation set is used to verify whether the trained remote sensing image semantic segmentation model can accurately segment the remote sensing image; and the test set is used to test the segmentation accuracy of the trained remote sensing image semantic segmentation model.

[0086] Step 204: constructing a remote sensing image semantic segmentation network including an encoder, a main decoder, and a plurality of auxiliary decoders;

[0087] In an embodiment of the present application, the teacher network and the student network have the same structure and both include an encoder and a decoder. The encoder is used to perform upsampling and extract high-dimensional features of the remote sensing image to generate a feature tensor. The decoder is used to perform downsampling and reduce the dimension of the feature tensor to generate a segmentation result. The segmentation result is the object category of each pixel in the remote sensing image and the probability corresponding to the object category.

[0088] Step 205: Divide the training set into labeled samples and unlabeled samples, and input them into the remote sensing image semantic segmentation network for training.

[0089] Labeled and unlabeled samples are fed into the encoder and primary decoder of the remote sensing image semantic segmentation network, and supervised loss is calculated and training is performed. Unlabeled samples are fed into the encoder, primary decoder, and auxiliary decoder of the remote sensing image semantic segmentation network, and pseudo labels are generated based on the output of the auxiliary decoder and the prediction result of the primary decoder. The pseudo labels and the corresponding remote sensing images are used as labeled samples for supervised training.

[0090] Figure 3 This is a flow chart of a cross-consistency self-training remote sensing image semantic segmentation network training method according to an exemplary embodiment. Figure 3 As shown, the method is used in remote sensing image semantic segmentation and includes the following steps.

[0091] Step 301, determining the hyperparameters and loss function used for training the remote sensing image semantic segmentation network;

[0092] In the embodiments of the present application, the remote sensing image semantic segmentation model has parameters set to certain values before training begins. During training, these hyperparameters are continuously optimized to improve the performance and effectiveness of segmenting remote sensing images. A loss function is used to calculate the difference between the segmentation result and the labeled sample. The loss is calculated based on this loss function to measure the accuracy of the remote sensing image segmentation result.

[0093] Step 302, optimizing the parameters of the remote sensing image semantic segmentation network until the prediction accuracy of the validation set prediction results reaches a preset accuracy threshold;

[0094] In an embodiment of the present application, it is necessary to optimize the remote sensing image semantic segmentation network to a certain accuracy, verify the accuracy of the remote sensing image semantic segmentation network segmentation based on the verification set, compare the semantic segmentation results and the ratio of pixels with label matching in the labeled remote sensing image, and if the ratio is greater than or equal to the preset accuracy threshold, it means that the training has been sufficient and the training can be stopped.

[0095] Step 303: Input the test set into the trained remote sensing image semantic segmentation network to verify the network segmentation accuracy.

[0096] In an embodiment of the present application, the trained remote sensing image semantic segmentation network is tested based on a test set, and the accuracy of the trained remote sensing image semantic segmentation network is calculated based on the comparison of the segmentation results of the test set with the labels of the test set, and based on whether the ground object categories of the same pixel are the same, and the proportion of the same pixel to all pixels.

[0097] Optionally, the training of the remote sensing semantic segmentation network includes:

[0098] The remote sensing images in the training set are divided into labeled samples and unlabeled samples, wherein the labeled samples include remote sensing images and corresponding labeled remote sensing images, and the unlabeled samples only include remote sensing images;

[0099] In the first training phase, the labeled samples and unlabeled samples are input into the encoder of the remote sensing semantic segmentation network. The encoder extracts the labeled sample feature map and inputs it into the main decoder. The obtained prediction results and the label data are used to calculate the supervision loss.

[0100] In an embodiment of the present application, the supervised loss is calculated by a cross entropy loss function.

[0101] In the first part of the second training phase, the unlabeled sample feature map extracted by the encoder is randomly transformed and input into the main decoder and the auxiliary decoder, and the consistency loss is calculated based on the prediction results of the auxiliary decoder and the main decoder.

[0102] In an embodiment of the present application, the random transformation includes dropout = 0.5, random 180° flipping in the horizontal and vertical directions, and adding Gaussian noise N~U(-0.2, 0.2) that obeys a uniform distribution. The number of the auxiliary decoders is 3, and the prediction results of each auxiliary decoder are compared with the prediction results of the main decoder to calculate the consistency loss, and the consistency loss is calculated by the mean square error loss function. The maximum value of the coefficient of the consistency loss is set to 1. The consistency loss reflects the difference between the output of the auxiliary decoder and the main decoder. The remote sensing semantic segmentation network is trained with the goal of reducing the consistency loss, that is, the prediction results of the auxiliary decoder are encouraged to be as consistent as possible with the prediction results of the main decoder.

[0103] In the second part of the second training phase, the prediction results of the main decoder and the auxiliary decoder are fused to obtain pseudo labels, and the pseudo labels and corresponding remote sensing images are used as labeled samples for supervised training.

[0104] In this embodiment, the prediction results of the primary decoder and the auxiliary decoder are fused, and a voting method is used to select the feature category with the most votes for each pixel to generate the pseudo-label. These pseudo-labels can be used as labels for the corresponding remote sensing images and input into the encoder and primary decoder together with the remote sensing images for training. This helps increase the number of training samples and improves training effectiveness.

[0105] Optionally, preprocessing the remote sensing image sample dataset includes:

[0106] Randomly sampling remote sensing images in the remote sensing image sample dataset into small images for multiple rounds of batch training, wherein the sampling frequency of each round of training is 480;

[0107] Set the sampling size and training batch value of the remote sensing image according to the size of the video memory space. The default input image size is 512×512, and the default training batch size is 10.

[0108] Performing multiple rounds of random sampling on the remote sensing image and the labeled remote sensing image, wherein each sampling obtains a 512×512 sized labeled remote sensing image and a corresponding label map, as well as an unlabeled image;

[0109] Data enhancement is performed on samples of each round of sampling with a random number and degree of randomness, and a training sample enhancement parameter range is set, wherein the data enhancement includes at least one of the following: random rotation of n×90° (n=0, 1, 2, 3); random horizontal or vertical 180° flip; random scale scaling, the scale scaling multiple is in the range of [0.5, 2]; random brightness enhancement, the brightness enhancement multiple is in the range of [0.5, 2]; random contrast enhancement, the contrast enhancement multiple is in the range of [0.5, 2]; random saturation enhancement, the saturation enhancement multiple is in the range of [0.5, 2].

[0110] Optionally, the remote sensing image semantic segmentation network is a classic semantic segmentation network UNet, and the UNet includes an encoder and a main decoder, and the number of auxiliary decoders is 3.

[0111] Optionally, the strategies for changing the learning rate of the hyperparameters and loss function during training include:

[0112]

[0113] Among them, base_lr = 4.2e-6; warmup_factor = 1.2; epoch is the number of training iterations; lr is the learning rate, which gradually increases with time; warm_epoch is the number of warm start iterations, and warm_epoch is set to 30;

[0114] Perform a warm start. When the training exceeds warm_epoch, use the polynomial learning rate decay strategy, set the maximum number of training iterations maxlr to 1500, the decay exponent pewer to 0.9, and the maximum learning rate maxlr to 1e-3.

[0115] The loss function used to calculate the supervision loss is the cross entropy loss function, the loss function used to calculate the consistency loss is the mean square error loss function, and the overall loss function of the remote sensing image semantic segmentation network is for:

[0116]

[0117] is the prediction result; y i To label images; To monitor losses; m The prediction result of the main decoder; e a is the prediction result of the auxiliary decoder; is the consistency loss; λ(epoch) is The weight of .

[0118] In the first training stage, the supervision loss partially dominates the overall loss;

[0119] In the second training phase, the overall loss is dominated by the consistency loss, and the specific formula of λ(epoch) is:

[0120]

[0121] Where num_epochs is the total number of training iterations. The training round threshold a is set to 200. λ(epoch) gradually increases before the ath training iteration. After training for more than a times, λ(epoch) stabilizes to w. num_epochs is set to 1500, and w is set to 1.

[0122] Optionally, the random transformation of the unlabeled sample feature map includes:

[0123] Dropout = 0.5, randomly flip 180° horizontally and vertically, and add Gaussian noise N~U(-0.2, 0.2) that follows a uniform distribution.

[0124] Optionally, fusing the prediction results of the primary decoder and the auxiliary decoder includes:

[0125] The prediction results of the main decoder and the prediction results of the auxiliary decoder are voted according to positions, and the category with the most votes for each pixel is taken as the final fusion result to generate the pseudo label.

[0126] Figure 5 FIG. 1 is a block diagram of a remote sensing image semantic segmentation model training system using cross-consistency learning according to an exemplary embodiment. Figure 5 As shown, corresponding to the first training stage mentioned above, the remote sensing image and its corresponding labeled data are input into the encoder and the main decoder, the supervision loss is calculated according to the output of the main decoder and the labeled data of the remote sensing image, and the parameters of the encoder and the main decoder are preliminarily optimized.

[0127] Figure 6 FIG. 1 is a block diagram of a remote sensing image semantic segmentation model training system using cross-consistency learning according to an exemplary embodiment. Figure 6 As shown, it corresponds to the second training stage mentioned above. In the first part of the second training stage, the main decoder of the unlabeled remote sensing image data is used to extract the feature tensor; the feature tensor is then randomly transformed and input into the main decoder and the auxiliary decoder, and the prediction results of the main decoder are used to calculate the consistency loss with the prediction results of each auxiliary decoder. In the second part of the second training stage, the prediction results of the main decoder and the auxiliary decoder are fused to obtain pseudo labels, and the pseudo labels and the corresponding remote sensing images are input again as labeled samples. Figure 5The encoder and main decoder in

[15] are trained supervised.

[0128] Figure 4 This is a block diagram of a cross-consistency self-training remote sensing image semantic segmentation network training device according to an exemplary embodiment. Figure 4 The device includes a construction module 410, a training module 420 and a segmentation module 430.

[0129] A construction module 410 is used to construct a remote sensing image semantic segmentation network, wherein the remote sensing semantic segmentation network is UNet;

[0130] A training module 420 is used to train the remote sensing image semantic segmentation network based on a remote sensing image sample dataset;

[0131] The segmentation module 430 is used to input the remote sensing image captured in real time into the trained remote sensing image semantic segmentation network and output the semantic segmentation result of the remote sensing image.

[0132] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0133] Figure 7 3 is a block diagram of an apparatus 700 for training a cross-consistency self-training remote sensing image semantic segmentation network according to an exemplary embodiment.

[0134] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 710 including instructions and an interface 730. The instructions can be executed by a processor 720 of the apparatus 700 to perform the above method. Alternatively, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0135] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0136] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A cross-consistency self-training remote sensing image semantic segmentation network training method, characterized by: include: Constructing a remote sensing image semantic segmentation network, wherein the remote sensing image semantic segmentation network is UNet; Training the remote sensing image semantic segmentation network according to a remote sensing image sample dataset; Inputting the remote sensing image captured in real time into the trained remote sensing image semantic segmentation network, and outputting the semantic segmentation result of the remote sensing image; The training of the remote sensing image semantic segmentation network according to the remote sensing image dataset includes: Acquire the remote sensing image sample data set, wherein the remote sensing image sample data set includes remote sensing images; Labeling the ground object categories of pixels in the remote sensing image to generate corresponding labeled remote sensing images; Preprocessing the remote sensing image sample dataset to obtain a training set, a validation set, and a test set; Construct a remote sensing image semantic segmentation network consisting of an encoder, a main decoder, and multiple auxiliary decoders; Dividing the training set into labeled samples and unlabeled samples, and inputting the samples into the remote sensing image semantic segmentation network for training; The training of the remote sensing image semantic segmentation network includes: The remote sensing images in the training set are divided into labeled samples and unlabeled samples, wherein the labeled samples include remote sensing images and corresponding labeled remote sensing images, and the unlabeled samples only include remote sensing images; In the first training phase, the labeled samples and unlabeled samples are input into the encoder of the remote sensing image semantic segmentation network. The encoder extracts the labeled sample feature map and inputs it into the main decoder. The obtained prediction results and the label data are used to calculate the supervision loss. In the first part of the second training phase, the unlabeled sample feature maps extracted by the encoder are randomly transformed and input into the main decoder and the auxiliary decoder, and the consistency loss is calculated based on the prediction results of the auxiliary decoder and the prediction results of the main decoder; In the second part of the second training phase, the prediction results of the primary decoder and the auxiliary decoder are fused to obtain pseudo labels, and the pseudo labels and the corresponding remote sensing images are used as labeled samples for supervised training; The strategies for changing the learning rate of hyperparameters and loss functions during training include: （1） Among them, base_lr=4.2e-6; warm_factor=1.2; epoch is the number of training iterations; lr is the learning rate, which gradually increases with time; warm_epoch is the number of hot start iterations, and warm_epoch is set to 30; Perform a warm start. When the training exceeds warm_epoch, use the polynomial learning rate decay strategy, set the maximum number of training iterations max_epoch to 1500, the decay exponent power to 0.9, and the maximum learning rate max_lr to 1e-3. The loss function used to calculate the supervision loss is the cross entropy loss function, the loss function used to calculate the consistency loss is the mean square error loss function, and the overall loss function of the remote sensing image semantic segmentation network is for: （2） To predict the results; To label images; To monitor losses; is the prediction result of the main decoder; is the prediction result of the auxiliary decoder; is the consistency loss; for The weight of In the first training stage, the supervision loss partially dominates the overall loss; In the second training phase, the overall loss is dominated by the consistency loss, The specific formula is: （3） Among them, num_epochs is the total number of training iterations; the training round threshold a is set to 200, and the training iteration is before the ath time. Gradually increase, after training more than a times Stabilize to w; num_epochs is set to 1500 and w is set to 1.

2. The method according to claim 1, characterized in that The inputting the remote sensing image semantic segmentation network for training includes: Determining hyperparameters and loss functions used in training the remote sensing image semantic segmentation network; Optimizing the parameters of the remote sensing image semantic segmentation network until the prediction accuracy of the validation set prediction results reaches a preset accuracy threshold; The test set is input into the trained remote sensing image semantic segmentation network to verify the network segmentation accuracy.

3. The method according to claim 1, characterized in that The preprocessing of the remote sensing image sample data set includes: Randomly sampling remote sensing images in the remote sensing image sample dataset into small images for multiple rounds of batch training, wherein the sampling frequency of each round of training is 480; Set the sampling size and training batch value of the remote sensing image according to the size of the video memory space. The default input image size is 512×512, and the default training batch size is 10. Performing multiple rounds of random sampling on the remote sensing image and the labeled remote sensing image, wherein each sampling obtains a 512×512 sized labeled remote sensing image and a corresponding label map, as well as an unlabeled image; Data enhancement is performed on samples of each round of sampling with a random number and degree of randomness, and a training sample enhancement parameter range is set, wherein the data enhancement includes at least one of the following: random rotation of n×90° (n=0, 1, 2, 3); random horizontal or vertical 180° flipping; random scale scaling, the multiple of the random scale scaling is in the range of [0.5, 2]; random brightness enhancement, the multiple of the brightness enhancement is in the range of [0.5, 2]; random contrast enhancement, the multiple of the contrast enhancement is in the range of [0.5, 2]; random saturation enhancement, the multiple of the saturation enhancement is in the range of [0.5, 2].

4. The method according to claim 1, wherein The remote sensing image semantic segmentation network is a classic semantic segmentation network UNet, which includes an encoder and a main decoder, and the number of auxiliary decoders is 3.

5. The method according to claim 1, wherein The random transformation method of the unlabeled sample feature map includes: Dropout=0.5, randomly flip 180° horizontally and vertically, and add Gaussian noise N~U(-0.2, 0.2) that follows a uniform distribution.

6. The method according to claim 1, characterized in that The prediction results of the main decoder and the auxiliary decoder are integrated, comprising: The prediction results of the main decoder and the prediction results of the auxiliary decoder are voted according to position, and the category with the most votes for each pixel is taken as the final fusion result to generate the pseudo label.

7. A cross-consistency self-training remote sensing image semantic segmentation network training device, characterized in that: include: A construction module is used to construct a remote sensing image semantic segmentation network, wherein the remote sensing image semantic segmentation network is UNet; A training module, configured to train the remote sensing image semantic segmentation network based on a remote sensing image sample dataset; A segmentation module is used to input the remote sensing image captured in real time into the trained remote sensing image semantic segmentation network and output the semantic segmentation result of the remote sensing image; The training of the remote sensing image semantic segmentation network according to the remote sensing image dataset includes: Acquire the remote sensing image sample data set, wherein the remote sensing image sample data set includes remote sensing images; Labeling the ground object categories of pixels in the remote sensing image to generate corresponding labeled remote sensing images; Preprocessing the remote sensing image sample dataset to obtain a training set, a validation set, and a test set; Construct a remote sensing image semantic segmentation network consisting of an encoder, a main decoder, and multiple auxiliary decoders; Dividing the training set into labeled samples and unlabeled samples, and inputting the samples into the remote sensing image semantic segmentation network for training; The training of the remote sensing image semantic segmentation network includes: The remote sensing images in the training set are divided into labeled samples and unlabeled samples, wherein the labeled samples include remote sensing images and corresponding labeled remote sensing images, and the unlabeled samples only include remote sensing images; In the first training phase, the labeled samples and unlabeled samples are input into the encoder of the remote sensing image semantic segmentation network. The encoder extracts the labeled sample feature map and inputs it into the main decoder. The obtained prediction results and the label data are used to calculate the supervision loss. In the first part of the second training phase, the unlabeled sample feature maps extracted by the encoder are randomly transformed and input into the main decoder and the auxiliary decoder, and the consistency loss is calculated based on the prediction results of the auxiliary decoder and the prediction results of the main decoder; In the second part of the second training phase, the prediction results of the primary decoder and the auxiliary decoder are fused to obtain pseudo labels, and the pseudo labels and the corresponding remote sensing images are used as labeled samples for supervised training; The strategies for changing the learning rate of hyperparameters and loss functions during training include: （1） Among them, base_lr=4.2e-6; warm_factor=1.2; epoch is the number of training iterations; lr is the learning rate, which gradually increases with time; warm_epoch is the number of hot start iterations, and warm_epoch is set to 30; Perform a warm start. When the training exceeds warm_epoch, use the polynomial learning rate decay strategy, set the maximum number of training iterations max_epoch to 1500, the decay exponent power to 0.9, and the maximum learning rate max_lr to 1e-3. The loss function used to calculate the supervision loss is the cross entropy loss function, the loss function used to calculate the consistency loss is the mean square error loss function, and the overall loss function of the remote sensing image semantic segmentation network is for: （2） To predict the results; To label images; To monitor losses; is the prediction result of the main decoder; is the prediction result of the auxiliary decoder; is the consistency loss; for The weight of In the first training stage, the supervision loss partially dominates the overall loss; In the second training phase, the overall loss is dominated by the consistency loss, The specific formula is: （3） Among them, num_epochs is the total number of training iterations; the training round threshold a is set to 200, and the training iteration is before the ath time. Gradually increase, after training more than a times Stabilize to w; num_epochs is set to 1500 and w is set to 1.

8. A cross-consistency self-training remote sensing image semantic segmentation network training device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by the processor of the cross-consistency self-training remote sensing image semantic segmentation network training device, the cross-consistency self-training remote sensing image semantic segmentation network training device is enabled to execute the method as described in any one of claims 1 to 6.

Images