A cross-domain semantic segmentation method for remote sensing images based on domain adaptation

Abstract

The present disclosure provides a method for cross-domain semantic segmentation of remote sensing images based on domain adaptation, characterized in that the method includes: obtaining a target domain remote sensing image; and inputting the target domain remote sensing image into a pre-trained target semantic segmentation model to obtain a semantic segmentation result of the target domain remote sensing image. The target semantic segmentation model is obtained by training adversarial domain adaptation and self-training domain adaptation, and the adversarial domain adaptation uses adversarial learning to perform feature-level domain adaptation and output-level domain adaptation. Self-training domain adaptation generates high-confidence pseudo labels on the basis of adversarial domain adaptation to continue training the semantic segmentation model, thereby improving the performance of target domain segmentation. The present disclosure also provides a device and device-based medium for cross-domain semantic segmentation of remote sensing images based on domain adaptation.

Description

Technical Field

[0001] The present disclosure relates to the field of remote sensing application technology, and in particular to a cross-domain semantic segmentation method of remote sensing images based on domain adaptation. Background Art

[0002] Remote sensing refers to the technology of using long-distance sensors such as satellites and drones to obtain electromagnetic radiation information of ground targets non-contactly and implement remote sensing of the surface. As the information carrier of remote sensing research, remote sensing images have important value in military and civilian fields. With the advancement of satellite and drone technology, the quality and resolution of remote sensing images have been significantly improved. At the same time, the difficulty of obtaining them has also been greatly reduced. Various remote sensing satellites (such as Landsat, Sentinel and Gaofen-2) provide a large amount of observation data as the basis for remote sensing image research. These images provide an important information basis for people to understand the earth system at different scales. In recent years, deep learning technology has begun to attract widespread attention, and various deep learning models have emerged. Among them, semantic segmentation models in deep learning technology are widely used in remote sensing images, such as full convolutional networks (FCN), U-Net and SegNet with encoder-decoder structures, and DeepLab series networks. These models have excellent performance, but in practical applications, there are some problems that prevent these models from being better applied.

[0003] Using deep learning methods for semantic segmentation usually requires sufficient training samples, which means that a large number of pixel-level labels must be manually labeled, but the cost is very expensive. Secondly, deep learning models are particularly sensitive to changes in data distribution, requiring that training data (source domain data) and test data (target domain data) meet the assumption of independent and identical distribution. However, in practical applications, due to differences in imaging sensors, geographical locations, spatial resolutions, and light intensities, there is a large gap in the appearance and structural features between different remote sensing images. At this point, if the semantic segmentation model that has been trained on the source domain image is directly applied to another set of completely different target domain images, the performance of the model will drop significantly. The main reason for this result is that different imaging sensors and light intensities can cause the same category to show completely different color information and texture features on different data. Therefore, we need to re-label the new image data for model training, and cannot make full use of the existing labeled data. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In order to solve at least one of the above-mentioned technical problems arising from the cross-domain semantic segmentation method of remote sensing images based on domain adaptation in the prior art, the embodiments of the present invention provide a cross-domain semantic segmentation method, device, equipment and medium for remote sensing images based on domain adaptation, which use source domain images and target domain images to simultaneously train the segmentation model, effectively solving the problem that the performance of the model will be significantly reduced when the semantic segmentation model that has been trained on the source domain images is directly applied to another set of completely different target domain images, and the problem that the target domain images need to be re-labeled each time.

[0006] (II) Technical solution

[0007] In view of the above problems, the embodiments of the present disclosure provide a method, apparatus, device and medium for cross-domain semantic segmentation of remote sensing images based on domain adaptation.

[0008] According to a first aspect of the present disclosure, a method for cross-domain semantic segmentation of remote sensing images based on domain adaptation is provided, characterized in that the method comprises: obtaining a remote sensing image in a target domain; and inputting the remote sensing image in the target domain into a pre-trained target semantic segmentation model to obtain a semantic segmentation result of the remote sensing image in the target domain. Pre-training the target semantic segmentation model includes adversarial domain adaptation training and self-training domain adaptation training.

[0009] In some exemplary embodiments, the adversarial domain adaptation network includes a segmentation network, a feature-level discriminator and an output-level discriminator, and pre-trains the target semantic segmentation model, including: obtaining source domain data and target domain data, wherein the source domain data carries a source domain label; using the feature-level discriminator and the segmentation network, through adversarial learning, the distribution of the first feature of the source domain data and the second feature of the target domain data are close to each other, to achieve feature-level domain adaptation, and obtain feature-level adversarial loss; using the output-level discriminator and the segmentation network, through collaborative training, the output levels of the source domain data and the target domain data are aligned at the category level, to obtain output-level adversarial loss; based on the source domain data, the supervised segmentation loss of the source domain data is calculated using a cross entropy loss function and a Dyce loss function; and a first loss function is constructed based on the feature-level adversarial loss, the feature-level adversarial loss and the supervised segmentation loss, and the segmentation network, the feature-level discriminator and the output-level discriminator are alternately trained using the first loss function to minimize the value of the first loss function, to obtain a domain-adapted semantic segmentation model.

[0010] In some exemplary embodiments, the segmentation network includes a feature extractor, and the feature-level discriminator and the segmentation network are used to make the distribution of the first feature of the source domain data and the second feature of the target domain data approach consistency through adversarial learning, thereby achieving feature-level domain adaptation and obtaining feature-level adversarial loss, specifically including: based on the source domain data and the target domain data, using the feature extractor to extract the first feature of the source domain data and the second feature of the target domain data; based on the first feature and the second feature, using the feature-level discriminator to obtain the domain classification of the source domain data and the domain classification of the target domain data; implementing an adversarial learning process based on a second loss function so that the feature extractor and the feature-level discriminator achieve feature-level domain adaptation, and the feature-level domain adaptation indicates that the distribution of the first feature of the source domain data and the second feature of the target domain data obtained by the feature extractor approach consistency; and calculating the feature-level adversarial loss based on the second loss function.

[0011] In some exemplary embodiments, the segmentation network further includes a first classifier and a second classifier, and the output level of the source domain data and the target domain data are aligned at the category level by collaborative training using the output level discriminator and the segmentation network, including: based on the source domain data and the source domain label, using the feature extractor to extract the network features of the source domain data; based on the network features, using the first classifier and the second classifier to respectively obtain the first pixel-level prediction result and the second pixel-level prediction result of the network features; based on the first pixel-level prediction result and the second pixel-level prediction result, using the output level discriminator to generate a first adversarial loss; based on the target domain data, using the first classifier and the second classifier to predict the target domain data A third pixel-level prediction result and a fourth pixel-level prediction result; based on the third pixel-level prediction result and the fourth pixel-level prediction result, using the cosine distance, generating a difference map between the prediction results; based on the third pixel-level prediction result and the fourth pixel-level prediction result, using the output-level discriminator to obtain an adversarial loss map; based on the difference map and the adversarial loss map, using element-by-element multiplication to obtain a weighted second adversarial loss, wherein the weighted loss weight is related to the size of the cosine distance between the prediction results; constructing a third loss function based on the first adversarial loss and the second adversarial loss, using the third loss function for training, and achieving output category-level alignment of the source domain data and the target domain data at the output level; and based on the third loss function, calculating the output-level adversarial loss.

[0012] In some exemplary embodiments, the supervised segmentation loss of the source domain data is calculated based on the source domain data using a cross entropy loss function and a Deuce loss function, including: based on the second pixel-level prediction result and the source domain label, using a cross entropy loss function to calculate the cross entropy loss of the source domain data; based on the second pixel-level prediction result and the source domain label, using a Deuce loss function to calculate the Deuce loss of the source domain data; and obtaining the supervised segmentation loss based on the cross entropy loss and the Deuce loss.

[0013] In some exemplary embodiments, the pre-training of the target semantic segmentation model further includes: based on the target domain data, using the domain adaptation semantic segmentation model to obtain a prediction result of the target domain data; based on the prediction result of the target domain data, using a superpixel segmentation and entropy self-training method to obtain a pseudo-label of the target domain data; and loading the trained domain adaptation semantic segmentation model, and continuing to train the domain adaptation semantic segmentation model based on the target domain data and the pseudo-label of the target domain data.

[0014] In some exemplary embodiments, the prediction result based on the target domain data uses a superpixel segmentation and entropy self-training method to obtain a pseudo label for the target domain data, including: based on the target domain data, using the Filsenzwab superpixel segmentation algorithm to generate a superpixel segmentation result of the target domain data; using the superpixel segmentation result and the prediction result of the target domain data to obtain the category label contained in the superpixel block in the target domain data; selecting superpixel blocks with consistent category labels, and retaining the prediction results of the superpixel blocks with consistent category labels as high-confidence results; further performing confidence evaluation on the high-confidence results based on the entropy to obtain the entropy value of each pixel, removing pixels with entropy values ​​greater than a threshold, and obtaining the pseudo label of the target domain data.

[0015] A second aspect of the present disclosure provides a cross-domain semantic segmentation device for remote sensing images based on domain adaptation, characterized in that the device includes the following modules: an acquisition module, used to acquire a target domain remote sensing image; and a prediction module, used to input the target domain remote sensing image into a pre-trained target semantic segmentation model to obtain a semantic segmentation result of the target domain remote sensing image, wherein pre-training the target semantic segmentation model includes adversarial domain adaptation training and self-training domain adaptation training.

[0016] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0017] The fourth aspect of the present disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0018] (III) Beneficial effects

[0019] It can be seen from the above technical solutions that the cross-domain semantic segmentation method, apparatus and device provided by the embodiments of the present disclosure have at least one of the following beneficial effects:

[0020] (1) Use source domain data and target domain data to simultaneously train the adversarial domain adaptation network to achieve feature-level domain adaptation and output-level domain adaptation, effectively solving the problem that the performance of the model will be significantly reduced due to different data distributions and the problem that a large number of target domain image labels need to be re-labeled.

[0021] (2) A new self-training domain adaptation method is proposed. This method uses superpixel segmentation and entropy filtering to obtain high-confidence pseudo-labels in the target domain. Based on the domain adaptation semantic segmentation model, the model is further trained with high-confidence pseudo-labels to improve the performance of the model in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 A schematic diagram of a process of pre-training an adversarial domain adaptation network according to an embodiment of the present disclosure is shown;

[0024] Figure 2 A schematic diagram of a process of using the feature-level discriminator and the segmentation network to make the distribution of the first feature of the source domain data and the second feature of the target domain data close to each other through adversarial learning, achieving feature-level domain adaptation, and obtaining feature-level adversarial loss according to an embodiment of the present disclosure is shown;

[0025] Figure 3 A schematic diagram of a process of using the output level discriminator and the segmentation network to achieve category-level alignment of the output levels of the source domain data and the target domain data through collaborative training according to an embodiment of the present disclosure is shown;

[0026] Figure 4 A schematic diagram of a process of calculating the supervised segmentation loss of the source domain data based on the source domain data using a cross entropy loss function and a Dyce loss function according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 5 The structure diagram of the adversarial domain adaptation network according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 6 The flowchart of the method for self-training domain adaptation of a domain adaptation semantic segmentation model according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 7 A schematic diagram of the structure of a method for self-training domain adaptation of a domain adaptation semantic segmentation model according to an embodiment of the present disclosure is shown;

[0030] Figure 8 The flowchart of obtaining the pseudo label of the target domain data by using the self-training method of superpixel segmentation and entropy based on the prediction result of the target domain data according to the embodiment of the present disclosure is schematically shown;

[0031] Fig. 9 A schematic diagram of a process flow of a remote sensing image cross-domain semantic segmentation method based on domain adaptation according to an embodiment of the present disclosure is schematically shown;

[0032] Fig.10 A structural block diagram of a remote sensing image cross-domain semantic segmentation device 800 based on domain adaptation according to an embodiment of the present disclosure is schematically shown; and

[0033] Fig.11 A block diagram of an electronic device for a remote sensing image cross-domain semantic segmentation method based on domain adaptation according to an embodiment of the present disclosure is schematically shown.

[0034] Illustration Description:

[0035] 600-electronic device; 601-processor; 602-read only memory (ROM); 603-random access memory (RAM); 604-bus; 605-input / output (I / O) interface; 606-input part; 607-output part; 608-storage part; 609-communication part; 610-drive; 611-removable medium. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0037] Pre-training the target semantic segmentation model includes adversarial domain adaptation training and self-training domain adaptation training. Adversarial domain adaptation uses adversarial learning to perform feature-level domain adaptation and output-level domain adaptation. Self-training domain adaptation generates high-confidence pseudo labels on the basis of adversarial domain adaptation to continue training the semantic segmentation model and improve the performance of target domain segmentation. In order to obtain a target semantic segmentation model that can accurately predict the semantic segmentation results of the target domain remote sensing image, the target semantic segmentation model needs to be pre-trained.

[0038] Figure 1 A flowchart of a method for pre-training an adversarial domain adaptation network according to an embodiment of the present disclosure is schematically shown.

[0039] like Figure 1 As shown, the method for pre-training an adversarial domain adaptation network according to an embodiment of the present disclosure includes steps S110-S150. The adversarial domain adaptation network includes a segmentation network, a feature-level discriminator, and an output-level discriminator. For example, the network structure consists of a segmentation network G and a feature-level discriminator D f and the output-level discriminator D0. The segmentation network consists of a feature extractor E and two classifiers C1 and C2.

[0040] In step S110, source domain data and target domain data are obtained, wherein the source domain data carries a source domain label.

[0041] In step S120, the feature-level discriminator and the segmentation network are used to make the distribution of the first feature of the source domain data and the second feature of the target domain data closer to each other through adversarial learning, so as to achieve feature-level domain adaptation and obtain feature-level adversarial loss.

[0042] In step S130, the output level discriminator and the segmentation network are used to achieve category-level alignment of the output levels of the source domain data and the target domain data through collaborative training, thereby obtaining an output level adversarial loss.

[0043] In step S140, based on the source domain data, a supervised segmentation loss of the source domain data is calculated using a cross entropy loss function and a Dyce loss function.

[0044] In step S150, a first loss function is constructed based on the feature-level adversarial loss, the feature-level adversarial loss and the supervised segmentation loss, and the segmentation network, the feature-level discriminator and the output-level discriminator are alternately trained using the first loss function to minimize the value of the first loss function, thereby obtaining a domain-adapted semantic segmentation model.

[0045] In some exemplary embodiments, the segmentation network includes a feature extractor, and step S120 further includes steps S121-S124, see Figure 2 .

[0046] In step S121, based on the source domain data and the target domain data, the feature extractor is used to extract a first feature of the source domain data and a second feature of the target domain data.

[0047] In step S122, based on the first feature and the second feature, the feature-level discriminator is used to obtain the domain classification of the source domain data and the domain classification of the target domain data.

[0048] In step S123, an adversarial learning process is implemented based on the second loss function, so that the feature extractor and the feature-level discriminator achieve feature-level domain adaptation, and the feature-level domain adaptation indicates that the distribution of the first feature of the source domain data and the second feature of the target domain data obtained by the feature extractor tends to be consistent.

[0049] In step S124, the feature-level adversarial loss is calculated based on the second loss function.

[0050] For example, we obtain high-dimensional features f from feature extractor E. s and f t Then, by introducing the feature-level discriminator D f , f s and f t As input, it produces domain classification output. In the adversarial learning process, f s and f t The distribution of is close to the same, so that domain-invariant features can be learned and feature-level domain alignment can be achieved. The second loss function is as follows:

[0051] (1)

[0052] In the formula, represents the feature-level adversarial loss, D f represents the feature-level discriminator, f s and f t High-dimensional features representing the source and target domains.

[0053] In some exemplary embodiments, the segmentation network further includes a first classifier and a second classifier, and step S130 further includes steps S131-S138, see Figure 3 .

[0054] In step S131, based on the source domain data and the source domain label, the feature extractor is used to extract network features of the source domain data.

[0055] In step S132, based on the network feature, the first classifier and the second classifier are used to obtain a first pixel-level prediction result and a second pixel-level prediction result of the network feature respectively.

[0056] In step S133, based on the first pixel-level prediction result and the second pixel-level prediction result, a first adversarial loss is generated using the output-level discriminator.

[0057] In step S134, based on the target domain data, the first classifier and the second classifier are used to predict and obtain a third pixel-level prediction result and a fourth pixel-level prediction result of the target domain data.

[0058] In step S135, based on the third pixel-level prediction result and the fourth pixel-level prediction result, a difference map between the prediction results is generated using the cosine distance.

[0059] In step S136, based on the third pixel-level prediction result and the fourth pixel-level prediction result, an adversarial loss map is obtained using the output-level discriminator.

[0060] In step S137, based on the difference map and the adversarial loss map, a weighted second adversarial loss is obtained by element-by-element multiplication, wherein the weighted loss weight is related to the size of the cosine distance between the prediction results.

[0061] In step S138, a third loss function is constructed based on the first adversarial loss and the second adversarial loss, and the third loss function is used for training to achieve output category-level alignment of the source domain data and the target domain data at the output level.

[0062] In step S139, based on the third loss function, the output level adversarial loss is calculated.

[0063] For example, in the source domain data stream, for the source domain image , the last two layers of network features obtained by feature extractor E are input into classifiers C1 and C2 respectively to produce the first pixel-level prediction results and the second pixel-level prediction result Then, the output-level discriminator is introduced .Will and As input to generate adversarial loss, learning domain invariance.

[0064] In the target domain data stream, for the target domain image , and also input it into the segmentation network G to obtain the prediction results of the two classifiers, and the third pixel-level prediction result And the fourth pixel prediction result However, unlike the source domain, the target domain has no labels to guide the segmentation loss, and relying solely on the discriminator for global alignment will lead to negative transfer of some already adapted categories. and Generate a difference map in , represents the cosine distance between two prediction results. Similarly, and Input to discriminator D0 to obtain adversarial loss graph , and then and Do element-by-element multiplication between them, so that you can get the weighted adversarial loss ,in Indicates traversing all pixels in the image. When it is larger, it means that the semantic information of the two prediction results is inconsistent, and the category has not been better adapted between the two domains. Therefore, a larger loss weight is given to encourage the generation network G to deceive the discriminator D0. When it is small, it indicates that the problem of inconsistent semantic information between the two prediction results is not serious. The category may be able to adapt between the two domains, so a smaller loss weight is given to ignore the adversarial penalty of D0. Through this weighted approach, more attention can be paid to those categories that are difficult to adapt at the output level to achieve category-level alignment. Finally, the output-level adversarial loss calculation formula is as follows:

[0065] (2)

[0066] In the formula, represents the output-level adversarial loss, G represents the segmentation network, D0 represents the output-level discriminator, and They are and In the prediction result of the target domain, M represents the cosine distance. is the weight that controls the class-level adversarial loss. In order to stabilize the training process, a decimal is added to the weight .

[0067] In some exemplary embodiments, step S140 further includes steps S141-S143, see Figure 4 .

[0068] In step S141, based on the second pixel-level prediction result and the source domain label, a cross entropy loss of the source domain data is calculated using a cross entropy loss function.

[0069] In step S142, based on the second pixel-level prediction result and the source domain label, the Deuce loss of the source domain data is calculated using the Deuce loss function.

[0070] In step S143, a supervised segmentation loss is obtained based on the cross entropy loss and the Deuce loss.

[0071] For example, based on the second pixel-level prediction result Source domain label Calculate the segmentation loss to guide the supervised semantic information of the segmentation network G. For the segmentation loss, a combination of soft cross-entropy loss (SoftCross-EntropyLoss) and Dice loss is used. Soft cross-entropy loss uses label-smoothed cross entropy to increase the generalization ability of the model, while Dice loss can alleviate the problem of category imbalance to a certain extent.

[0072] (3)

[0073] (4)

[0074] In the formula, represents the soft cross entropy loss, represents Dice loss, C represents category, H and W represent the height and width of a source domain image slice, represents the true label of the source domain, Represents the source domain prediction map.

[0075] Adversarial domain adaptation is achieved by jointly minimizing all the above losses, i.e., the supervised segmentation loss on the source domain, the feature-level adversarial loss for both domains, and the output-level adversarial loss.

[0076] During the training process, the segmentation network G and the feature-level discriminator and the output stage discriminator Alternate training. Therefore, the first loss function constructed based on feature-level adversarial loss, feature-level adversarial loss and supervised segmentation loss is as follows:

[0077] (5)

[0078] In the formula, represents the balanced feature-level adversarial loss, represents the weight of the output level adversarial loss.

[0079] The structure diagram of the adversarial domain adaptation network according to the embodiment of the present disclosure is shown in FIG. Figure 5 .

[0080] In order to further improve the accuracy of the target semantic segmentation model, superpixel segmentation and entropy filtering are used to obtain high-confidence pseudo labels for the target domain images. Figure 1Based on the trained domain-adapted semantic segmentation model, the model is further trained through high-confidence pseudo-labels to improve the performance of the model in the target domain.

[0081] Figure 6 The flowchart of the domain adaptation method of self-training domain adaptation of the domain adaptation semantic segmentation model according to an embodiment of the present disclosure is schematically shown.

[0082] like Figure 6 As shown, the domain adaptation semantic segmentation model self-training domain adaptation method according to the embodiment of the present disclosure includes steps S210-S230.

[0083] In step S210, based on the target domain data, the domain adaptation semantic segmentation model is used to obtain a prediction result of the target domain data.

[0084] In step S220, based on the prediction result of the target domain data, a pseudo label of the target domain data is obtained by using a superpixel segmentation and entropy self-training method.

[0085] In step S230, the domain adaptation semantic segmentation model is loaded with the trained model weights, and the domain adaptation semantic segmentation model is further trained based on the target domain data and the pseudo labels of the target domain data. Figure 7 .

[0086] In some exemplary embodiments, step S220 further includes steps S221-S224, see Figure 8 .

[0087] In step S221, based on the target domain data, a superpixel segmentation result of the target domain data is generated using the Filsenzwab superpixel segmentation algorithm.

[0088] In step S222, the category label contained in the superpixel block in the target domain data is obtained by using the superpixel segmentation result and the prediction result of the target domain data.

[0089] In step S223, super pixel blocks with consistent category labels are selected, and the prediction results of the super pixel blocks with consistent category labels are retained as high confidence results.

[0090] In step S224, the high confidence result is further evaluated for confidence based on the entropy to obtain an entropy value for each pixel, and pixels with entropy values ​​greater than a threshold are removed to obtain a pseudo label for the target domain data.

[0091] For example, the Felzenszwalb superpixel segmentation algorithm is first used to generate the superpixel segmentation results of the target domain image. Then this result is compared with the prediction results of the target domain that have been obtained to obtain the category labels contained in each superpixel block in each image. Based on the object-oriented segmentation criterion, the categories within a pixel block should be consistent. Therefore, the prediction results corresponding to the pixel blocks containing consistent categories are retained as high-confidence results, and the prediction results corresponding to the pixel blocks with inconsistent categories are assigned a value of 255. These pixels will be ignored in subsequent training. At the same time, entropy is continued to be used to evaluate the confidence of each pixel, and pixels with low confidence in the sample are removed to generate more reliable pseudo labels. Specifically, the entropy values ​​of the prediction results of each category are sorted in descending order. μ is a hyperparameter that defines the proportion of retained pixels (μ is a percentage), and the entropy value of the category at 1-μ is selected as the threshold of this category. Pixels whose entropy values ​​are less than the threshold are retained, and pixels whose entropy values ​​are greater than the threshold are assigned a value of 255. The specific implementation process of the algorithm is shown in Table 1.

[0092] Table 1 Self-training pseudo-label generation algorithm

[0093]

[0094] Fig. 9 A flowchart of a method for cross-domain semantic segmentation of remote sensing images based on domain adaptation according to an embodiment of the present disclosure is schematically shown.

[0095] like Fig. 9 As shown, according to an embodiment of the present disclosure, a remote sensing image cross-domain semantic segmentation method based on domain adaptation includes steps S310 to S320.

[0096] In step S310, a remote sensing image of a target domain is acquired.

[0097] In step S320, the target domain remote sensing image is input into a pre-trained target semantic segmentation model to obtain a semantic segmentation result of the target domain remote sensing image, wherein pre-training the target semantic segmentation model includes adversarial domain adaptation training and self-training domain adaptation training.

[0098] Fig.10 The structural block diagram of a remote sensing image cross-domain semantic segmentation device 800 based on domain adaptation according to an embodiment of the present disclosure is schematically shown.

[0099] like Fig.10 As shown, a remote sensing image cross-domain semantic segmentation device 800 based on domain adaptation according to an embodiment of the present disclosure includes an acquisition module 810 and a prediction module 820.

[0100] The acquisition module 810 is used to acquire the remote sensing image of the target domain.

[0101] The prediction module 820 is used to input the target domain remote sensing image into a pre-trained target semantic segmentation model to obtain a semantic segmentation result of the target domain remote sensing image, wherein the pre-training of the target semantic segmentation model includes adversarial domain adaptation training and self-training domain adaptation training.

[0102] In some specific embodiments, any multiple modules in the acquisition module 810 and the prediction module 820 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0103] In some specific embodiments, at least one of the acquisition module 810 and the prediction module 820 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 810 and the prediction module 820 may be at least partially implemented as a computer program module, which may perform corresponding functions when the computer program module is executed.

[0104] Fig.11 A block diagram of an electronic device for a remote sensing image cross-domain semantic segmentation method based on domain adaptation according to an embodiment of the present disclosure is schematically shown.

[0105] like Fig.11 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0106] In RAM603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM602 and RAM603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM602 and / or RAM603. It should be noted that the program can also be stored in one or more memories other than ROM602 and RAM603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0107] In some specific embodiments, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0108] An embodiment of the present disclosure shows a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0109] In some specific embodiments, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM602 and / or RAM603 described above and / or one or more memories other than ROM602 and RAM603.

[0110] An embodiment of the present disclosure shows a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0111] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0112] In some specific embodiments, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0113] In some specific embodiments, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0114] In some specific embodiments, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0115] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0117] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A cross-domain semantic segmentation method for remote sensing images based on domain adaptation, characterized in that: The method comprises: Acquire a target domain remote sensing image; and input the target domain remote sensing image into a pre-trained target semantic segmentation model to obtain a semantic segmentation result of the target domain remote sensing image, Wherein, pre-training the target semantic segmentation model includes adversarial domain adaptation network training and self-training domain adaptation training; The adversarial domain adaptation network includes a segmentation network, a feature-level discriminator, and an output-level discriminator. The method for training an adversarial domain adaptation network comprises: Acquire source domain data and target domain data, wherein the source domain data has a source domain label; Using the feature-level discriminator and the segmentation network, the distribution of the first feature of the source domain data and the second feature of the target domain data are made close to each other through adversarial learning, so as to achieve feature-level domain adaptation and obtain feature-level adversarial loss; Using the output level discriminator and the segmentation network, the output levels of the source domain data and the target domain data are aligned at the category level through collaborative training to obtain an output level adversarial loss; Based on the source domain data, using a cross entropy loss function and a Dyce loss function to calculate the supervised segmentation loss of the source domain data; and constructing a first loss function based on the feature-level adversarial loss, the feature-level adversarial loss and the supervised segmentation loss, and alternately training the segmentation network, the feature-level discriminator and the output-level discriminator using the first loss function to minimize the value of the first loss function, thereby obtaining a domain-adapted semantic segmentation model; The method for self-training domain adaptation training includes: Based on the target domain data, using the domain adaptation semantic segmentation model, obtaining a prediction result of the target domain data; Based on the prediction result of the target domain data, a pseudo label of the target domain data is obtained by using a superpixel segmentation and entropy self-training method; and The trained domain adaptation semantic segmentation model is loaded, and the domain adaptation semantic segmentation model is continued to be trained based on the target domain data and the pseudo-labels of the target domain data.

2. The method according to claim 1, characterized in that The segmentation network includes a feature extractor, The method of using the feature-level discriminator and the segmentation network to make the distribution of the first feature of the source domain data and the second feature of the target domain data close to each other through adversarial learning, achieving feature-level domain adaptation, and obtaining feature-level adversarial loss specifically includes: Based on the source domain data and the target domain data, using the feature extractor to extract a first feature of the source domain data and a second feature of the target domain data; Based on the first feature and the second feature, using the feature-level discriminator to obtain a domain classification of the source domain data and a domain classification of the target domain data; Implementing an adversarial learning process based on a second loss function so that the feature extractor and the feature-level discriminator achieve feature-level domain adaptation, wherein the feature-level domain adaptation indicates that the distribution of the first feature of the source domain data and the second feature of the target domain data obtained by the feature extractor are close to being consistent; and The feature-level adversarial loss is calculated based on the second loss function.

3. The method according to claim 2, characterized in that The segmentation network further comprises a first classifier and a second classifier, Using the output level discriminator and the segmentation network, the output levels of the source domain data and the target domain data are aligned at the category level through collaborative training, including: Based on the source domain data and the source domain label, using the feature extractor to extract network features of the source domain data; Based on the network feature, using the first classifier and the second classifier to obtain a first pixel-level prediction result and a second pixel-level prediction result of the network feature respectively; generating a first adversarial loss using the output-level discriminator based on the first pixel-level prediction result and the second pixel-level prediction result; Based on the target domain data, using the first classifier and the second classifier to predict and obtain a third pixel-level prediction result and a fourth pixel-level prediction result of the target domain data; Based on the third pixel-level prediction result and the fourth pixel-level prediction result, generating a difference map between the prediction results using cosine distance; Based on the third pixel-level prediction result and the fourth pixel-level prediction result, using the output-level discriminator to obtain an adversarial loss map; Based on the difference map and the adversarial loss map, obtaining a weighted second adversarial loss by element-by-element multiplication, wherein the weight of the weighted loss is related to the size of the cosine distance between the prediction results; constructing a third loss function based on the first adversarial loss and the second adversarial loss, performing training using the third loss function, and achieving output category-level alignment of the source domain data and the target domain data at the output level; and Based on the third loss function, the output level adversarial loss is calculated.

4. The method according to claim 3, characterized in that The step of calculating the supervised segmentation loss of the source domain data based on the source domain data by using a cross entropy loss function and a Dyce loss function includes: Based on the second pixel-level prediction result and the source domain label, a cross entropy loss of the source domain data is calculated using a cross entropy loss function; Based on the second pixel-level prediction result and the source domain label, using a Dyce loss function to calculate a Dyce loss of the source domain data; The supervised segmentation loss is obtained based on the cross entropy loss and the Deuce loss.

5. The method according to claim 1, characterized in that The method of obtaining a pseudo label of the target domain data based on the prediction result of the target domain data by using a superpixel segmentation and entropy self-training method includes: Based on the target domain data, using the Filsenzwab superpixel segmentation algorithm to generate a superpixel segmentation result of the target domain data; Using the superpixel segmentation result and the prediction result of the target domain data, obtaining the category label contained in the superpixel block in the target domain data; Selecting superpixel blocks with consistent category labels, and retaining the prediction results of the superpixel blocks with consistent category labels as high confidence results; The high confidence result is further evaluated based on the entropy to obtain an entropy value for each pixel, and pixels with entropy values ​​greater than a threshold are removed to obtain a pseudo label for the target domain data.

6. A remote sensing image cross-domain semantic segmentation device based on domain adaptation, characterized in that: The device comprises the following modules: An acquisition module, used to acquire remote sensing images of a target domain; and A prediction module is used to input the target domain remote sensing image into a pre-trained target semantic segmentation model to obtain a semantic segmentation result of the target domain remote sensing image. Wherein, pre-training the target semantic segmentation model includes adversarial domain adaptation network training and self-training domain adaptation training, The adversarial domain adaptation network includes a segmentation network, a feature-level discriminator, and an output-level discriminator. The method for training an adversarial domain adaptation network comprises: Acquire source domain data and target domain data, wherein the source domain data has a source domain label; Using the feature-level discriminator and the segmentation network, the distribution of the first feature of the source domain data and the second feature of the target domain data are made close to each other through adversarial learning, so as to achieve feature-level domain adaptation and obtain feature-level adversarial loss; Using the output level discriminator and the segmentation network, the output levels of the source domain data and the target domain data are aligned at the category level through collaborative training to obtain an output level adversarial loss; Based on the source domain data, using a cross entropy loss function and a Dyce loss function to calculate the supervised segmentation loss of the source domain data; and constructing a first loss function based on the feature-level adversarial loss, the feature-level adversarial loss and the supervised segmentation loss, and alternately training the segmentation network, the feature-level discriminator and the output-level discriminator using the first loss function to minimize the value of the first loss function, thereby obtaining a domain-adapted semantic segmentation model; The method for self-training domain adaptation training includes: Based on the target domain data, using the domain adaptation semantic segmentation model, obtaining a prediction result of the target domain data; Based on the prediction result of the target domain data, a pseudo label of the target domain data is obtained by using a superpixel segmentation and entropy self-training method; and The trained domain adaptation semantic segmentation model is loaded, and the domain adaptation semantic segmentation model is continued to be trained based on the target domain data and the pseudo-labels of the target domain data.

7. An electronic device, wherein: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 5.

Images