Remote sensing map image domain migration method, system and computer equipment based on GAN

Abstract

The present application relates to a GAN-based remote sensing map image domain migration method, system, and computer equipment, belonging to the field of remote sensing image processing technology. The method includes acquiring remote sensing images and establishing an image dataset; the image dataset includes sample domain images and out-of-sample domain images, where the sample domain images are fogged images and the out-of-sample domain images are clear images; constructing a TF2GAN model, inputting images in the image dataset into the TF2GAN model for model training; the TF2GAN model includes a generator and a discriminator; wherein the model training includes sample domain image recognition processing training and out-of-sample domain image recognition training; the TF2GAN model has a preset total loss function, and the generator and discriminator are trained according to the preset total loss function and image dataset of the TF2GAN model to obtain a trained TF2GAN model; and performing image conversion on the input remote sensing image according to the trained TF2GAN model. The present application has the effect of reducing the increase in labor costs caused by different domains and improving generalization.

Description

Technical Field

[0001] The present application relates to the field of remote sensing image processing technology, and in particular to a GAN-based remote sensing map image domain migration method, system, and computer equipment. Background Art

[0002] With the continuous improvement of the spatial resolution of remote sensing images, high-resolution remote sensing images have become one of the most important modern means of obtaining geographic information, and have great application value for land observation, land use, farmland planning and mapping.

[0003] Semantic segmentation of remote sensing images is a core task in image processing and computer vision. It identifies remote sensing images on a pixel-by-pixel basis, determining the object category to which each pixel belongs. Early approaches to remote sensing image segmentation primarily relied on a combination of visual interpretation and computer vision, such as domain-based segmentation, clustering-based segmentation, edge-based segmentation, and graph-based segmentation. These traditional segmentation methods often only extract low-level image features, resulting in low segmentation efficiency and failing to meet the requirements for high-resolution remote sensing image segmentation.

[0004] Therefore, in recent years, researchers both domestically and internationally have largely adopted deep learning methods. Maggiori et al. applied FCN to a Massachusetts building dataset. While this method can better capture ground features in images compared to traditional neural networks, its semantic segmentation accuracy relies heavily on large datasets and corresponding ground-truth labels. Manually collecting large datasets and labels is time-consuming, labor-intensive, and prohibitively expensive, requiring precise labeling of each image. Network generalization issues have long been a significant and common problem in industry. Data may come from multiple sources, or our training set may consist of high-definition images, while the actual test set may be blurry. This often leads to a mismatch between the distribution of the training domain and the actual test domain, significantly impacting remote sensing map segmentation. On the one hand, labeling is required across different image domains, and on the other hand, as the number of image domains increases, the overall robustness and segmentation accuracy of the network decreases, leading to increased labor costs and performance deficiencies. Summary of the Invention

[0005] In order to solve the problems of generalization and increased labor costs caused by different satellite remote sensing map domains, the present application provides a GAN-based remote sensing map image domain migration method, system and computer equipment.

[0006] In the first aspect, this application provides a method for remote sensing map image domain migration based on GAN, which adopts the following technical solutions:

[0007] A method for remote sensing map image domain migration based on GAN, the method comprising:

[0008] Acquire remote sensing images and establish an image data set; the image data set includes a sample image domain image and an out-of-sample image domain image, the sample image domain image is a fogged image, and the out-of-sample image domain image is a clear image;

[0009] Construct a TF2GAN model and input the images in the image dataset into the TF2GAN model for model training; the TF2GAN model includes a generator and a discriminator; among them,

[0010] The model training includes sample domain image recognition processing training and out-of-sample domain image recognition training;

[0011] The TF2GAN model has a preset total loss function, and the generator and the discriminator are trained according to the preset total loss function of the TF2GAN model and the image dataset to obtain a trained TF2GAN model;

[0012] Perform image conversion on the input remote sensing image according to the trained TF2GAN model.

[0013] By sampling the above technical solutions, building and training the TF2GAN model to migrate the image domains of different satellite remote sensing maps, preprocessing the images, and migrating them to the source domain, so that the target domain is close to the source domain, the generalization of the model can be greatly improved at the preprocessing level, reducing labor costs.

[0014] Preferably, the total loss function of the TF2GAN model includes a total adversarial loss function and a cycle consistency loss function, and the formula of the total loss function is:

[0015] L(G,F,E,DX,DY)=L(G,E,F,D X ,D Y1 ,D Y2 )+λL cyc (G,F,E);

[0016] Among them, L(G,E,F,D X ,D Y1 ,D Y2 ) is the total adversarial loss function, L cyc (G, F, E) is the cycle consistency loss function, and λ is the weight coefficient.

[0017] By sampling the above technical solutions, TF2GAN adds the generation of Y once in the XYX process based on CycleGAN, enhancing its migration robustness through a two-chain structure.

[0018] Preferably, the formula of the cycle consistency loss function is:

[0019] L cyc (G,F,E)=E x-pdata(x) [||√F(G(x))-√x)||2]

[0020] +E x-pdata(x) [||√E(F(G(x)))-√G(x)||2]

[0021] +E y-pdata(y) [||√G(F(y))-√y||2]

[0022] +E y-pdata(y) [||√E(G(F(y)))-√F(y)||2];

[0023] Among them, y represents sampling in Y space, x represents the sample taken by the input generator from X space, and G(x) represents the image generated by the generator; D Y2 (y) represents the probability that the discriminator determines whether y is a sample taken from the Y space; D Y2 (G(x)) is the probability that the discriminator determines whether the image generated by the generator is a sample taken from the Y space.

[0024] By sampling the above technical solutions, the TF2GAN model improves on the CycleGAN model by changing the L1 loss to the L2 loss and adding the L2 loss of FakeA1 and FakeA2, as well as FakeB1 and FakeB2, which can greatly improve its generation robustness.

[0025] Preferably, the generator is used to process pictures in the image dataset to generate new electronic pictures, and the generator includes:

[0026] Generator A2B1 is used to convert a sample domain image into a clear image Fake B1;

[0027] Generator B2A is used to convert the generated clear image Fake B1 into the fogged image Cyclic A again;

[0028] Generator A2B2 is used to convert the generated fog image Cyclic A into a new clear image Fake B2;

[0029] Generator B2A1 is used to convert a sample external domain image into a fog image Fake A1;

[0030] Generator A2B is used to convert the generated fog image Fake A1 into a clear image Cyclic B again;

[0031] Generator B2A2, used to convert the generated clear image Cyclic B into a new fogged image Fake A2;

[0032] The discriminator is used to compare the electronic image generated by the generator with the images in the image dataset, and the discriminator includes: discriminator A1 is used to compare the sample image domain image input to generator A2B1 with the fog image Fake A1 generated by generator B2A1;

[0033] Discriminator A2 is used to compare the sample domain image input to generator A2B1 with the fog image Fake A2 generated by generator B2A2;

[0034] Discriminator B1 is used to compare the out-of-sample domain image input to generator B2A1 with the clear image Fake B1 generated by generator A2B1;

[0035] The discriminator B2 is used to compare the out-of-sample domain image input to the generator B2A1 with the clear image Fake B2 generated by the generator A2B2.

[0036] Preferably, the generator A2B1, the generator B2A, the generator B2A1 and the generator A2B use CM-U-Net++ which is a combination of U-Net++ and CABM channel attention;

[0037] The generators A2B2 and B2A2 are generators composed of residual blocks of ResNet;

[0038] The discriminator adopts the skeleton network of CNN and the Encoder-Decoder structure of Transformer.

[0039] By sampling the above technical solutions, the generator can further enhance the diversity of the TF2GAN model by adopting two different architectures, thereby retaining more original image information, better preserving its corresponding image during image domain migration, and enhancing the model's understanding ability. The Transformer is a model based on a multi-head attention mechanism. This model can solve the problem of long-distance dependencies, has better memory, reduces lengthy and complex calculations, and achieves more accurate recognition results. It introduces a CNN convolutional network to extract underlying features and better capture local features.

[0040] Preferably, the sample image domain image recognition processing training model includes the following steps:

[0041] A sample domain image is randomly input, and after being transformed by the generator A2B1, a disordered Fake B1 is formed;

[0042] The generated Fake B1 is input into the generator B2A to generate CyclicA after transformation;

[0043] Input the generated Cyclic A into the generator A2B2, and transform it again to generate a new Fake B2;

[0044] Freeze the weights of the generator and enable the recognition of the discriminator.

[0045] By sampling the above technical solution, generator A2B1 converts the input sample domain image, i.e., the foggy image (an image with unclear distribution), into a clear image, and generator B2A converts the generated clear image into a foggy image again. The purpose is to calculate the difference between the original foggy image and the generated foggy image, and it is expected that the image distributions of the two will become more and more similar through training.

[0046] Preferably, the out-of-sample image domain image recognition processing training includes the following steps:

[0047] A random out-of-sample domain image B is input, and after being transformed by the generator B2A1, a disordered FakeA1 is formed;

[0048] The generated Fake A1 is input into the generator A2B to generate Cyclic B after transformation;

[0049] Input the generated Cyclic B into the generator B2A2, and transform it again to generate a new Fake A2;

[0050] Freeze the weights of the generator and enable the recognition of the discriminator.

[0051] By sampling the above technical solution, generator B2A1 converts the input out-of-sample domain image, i.e., the clear image, into a foggy image, and generator A2B converts the generated foggy image into a clear image again. The purpose is to calculate the difference between the original clear image and the generated clear image, and it is expected that the image distributions of the two will become more and more similar through training.

[0052] Preferably, the sample domain image processing training and the out-of-sample domain image processing training are performed simultaneously.

[0053] By sampling the above technical solutions, the model converts the foggy image into a clear image while converting another clear image into a foggy image, thereby enhancing the robustness of the entire model.

[0054] In a second aspect, the present application provides a system for applying the above-mentioned GAN-based remote sensing map image domain migration method, which adopts the following technical solutions:

[0055] A system applied to the GAN-based remote sensing map image domain migration method described in the first aspect, comprising an image processing module, a model building module, and an image conversion module;

[0056] The image processing module is used to acquire remote sensing images and establish an image data set;

[0057] The model building module is used to build a TF2GAN model and input the images in the image dataset into the TF2GAN model for model training;

[0058] The image conversion module is used to perform image conversion on the input remote sensing image according to the trained TF2GAN model.

[0059] In a third aspect, the present application provides a computer device that adopts the following technical solution:

[0060] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image domain migration method described in the first aspect when executing the computer program.

[0061] In a fourth aspect, the present application provides a computer storage medium, which adopts the following technical solution:

[0062] A computer storage medium storing a computer program, wherein the computer program is read and executed by a processor to implement the image domain migration method as described in the first aspect.

[0063] In summary, this application includes at least one of the following beneficial technical effects:

[0064] 1. Based on the original CycleGAN single loop, an additional loop is added to form the ABAB generative adversarial form. A multi-core MMD and multi-layer adaptive high-level network are added to the joint CNN and Transformer structure to match the source and target domains in transfer learning. It is expected to adapt more high-level features and accelerate training and fitting.

[0065] 2. The loss function is modified, and the ABA loss is upgraded to the ABAB loss. The original four-generation and two-discrimination are improved to six-generation. The four-discrimination form uses the square root L2 loss between AA and BB to calculate the regression loss of both.

[0066] 3. The generator uses CM-U-Net++, which can retain more original image information and better preserve the corresponding image during image domain migration, while also enhancing the model's understanding ability. The discriminator uses a combination of transformer and CNN, which can solve the problem of long-distance dependency, improve recognition performance, and better capture local features. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a schematic diagram of the overall process of the domain migration method in an embodiment of the present application;

[0068] Figure 2 3. These are image effect diagrams of two image domain remote sensing images of the image domain migration method in the embodiment of the present application;

[0069] Figure 3 This is a schematic diagram of the TF2GAN model training principle of the image domain migration method in an embodiment of the present application;

[0070] Figure 4 This is a schematic diagram of the sample domain image recognition processing training process of the domain migration method in the embodiment of the present application;

[0071] Figure 5 This is a schematic diagram of the out-of-sample domain image recognition processing training process of the domain migration method in an embodiment of the present application;

[0072] Figure 6 This is an image effect diagram of the image processing of the image domain migration method in the embodiment of the present application;

[0073] Figure 7 This is a structural block diagram of the domain migration system in an embodiment of the present application;

[0074] Figure 8 It is a diagram of the internal structure of a computer device in an embodiment of the present application.

[0075] Explanation of reference numerals: 100, image processing module; 200, model building module; 300, image conversion module. DETAILED DESCRIPTION

[0076] The following is combined with the accompanying drawings Figure 1-Figure 7 This application is described in further detail.

[0077] Remote sensing technology, as a space exploration technique, can acquire large-scale data. These macroscopic images are crucial for analyzing Earth's resources and environment. Driven by energy conservation and emission reduction, the establishment of photovoltaic power plants can effectively improve solar energy utilization. The available area for photovoltaic module installation and the duration of sunlight exposure are key factors influencing the potential for photovoltaic development. Determining a region's potential is crucial for the subsequent establishment of photovoltaic power plants. Remote sensing map images allow for more intuitive and accurate selection of the area within which photovoltaic power plants will be established and calculation of the installed capacity of photovoltaic modules within that area.

[0078] The embodiment of the present application designs a TF2GAN model for a remote sensing map image domain migration method, and uses a generative adversarial network TF2GAN model with domain migration to make the domains of the two similar, so as to solve the problems of generalization and increased labor costs caused by the different domains of satellite remote sensing maps. The TF2GAN model is based on the CNN skeleton network and the Encoder-Decoder structure of the Transformer, and the Transformer is a model based on a multi-head attention mechanism. This model can solve the existing long-distance dependency problem, has better memory, reduces lengthy and complex calculations to achieve more accurate recognition effects, introduces a CNN convolutional network to extract underlying features to better capture local features, and uses a combination of CNN and Transformer as a discriminator (Discriminato); uses CM-U-Net++, which is a combination of U-Net++ and CABMS channel attention, as a generator (Generator). Domain migration is triggered from the conditional GAN of the generative adversarial network, and based on image style migration, changes the distribution domain of the image to achieve the goal of transforming an unclear distribution picture into a clear distribution picture.

[0079] The present application embodiment discloses a remote sensing map image domain migration method based on GAN, referring to Figure 1 and Figure 2 ,The graph domain migration method specifically includes the following steps:

[0080] S1. Acquire remote sensing images and establish image datasets.

[0081] Specifically, the image dataset includes sample domain images and out-of-sample domain images. For image segmentation based on deep learning, the semantic segmentation model trained with a dataset in one scene cannot adapt well to the data of another scene, especially for remote sensing images. Due to its wide and large domain, different satellites may produce remote sensing information in different domains. In addition, the occlusion of natural environments such as clouds and fog will form different domain environments. The embodiment of the present application mainly analyzes the remote sensing images of Google's 19-level map. The image structure of the remote sensing image mainly includes the source domain and the target domain, wherein the source domain is the sample domain, that is, the distribution of the target image is not clear, such as pictures occluded by clouds and fog; the target domain is the out-of-sample domain, that is, the distribution of the target image is clear.

[0082] Download the remote sensing image of the target area. In this embodiment of the application, the remote sensing image of the target area corresponding to the Google level 19 is downloaded. The target area can be divided according to the administrative area as the division standard, or the target area size radius as the division standard, or the area selected by humans as the division standard. The required images are intercepted according to the same sample size to obtain the same number of sample domain images and out-of-sample domain images. In this embodiment of the application, the sample size is preferably 640*640 (unit: pixel).

[0083] In order to further improve the training accuracy of the TF2GAN model, the captured images are manually screened twice to remove images with similar scenes, so as to ensure that the TF2GAN model can obtain higher training accuracy with a relatively small amount of training data. For example, if a large number of images show farmland, then during manual screening, some similar images need to be removed to prevent the trained TF2GAN model from being biased towards the conversion of such images. After screening, the number of out-of-sample domain images and sample domain images remains the same.

[0084] S2. Build a TF2GAN model and input the images in the image dataset into the TF2GAN model for model training; the TF2GAN model includes a generator and a discriminator.

[0085] Specifically, refer to Figure 3 The generator is used to process the images in the image dataset to generate new electronic images. In the embodiment of the present application, the generator includes generator A2B1, generator B2A, generator A2B2, generator B2A1, generator A2B, and generator B2A2.

[0086] Generator A2B1 is used to transform the fog image InputA (sample image domain image A) into a clear image Fake B1;

[0087] Generator B2A is used to convert the generated clear image Fake B1 into the fogged image Cyclic A again;

[0088] Generator A2B2 is used to convert the generated fog image Cyclic A into a new clear image FakeB2;

[0089] Generator B2A1 is used to convert the clear image InputB (out-of-sample domain image B) into a fogged image FakeA1;

[0090] Generator A2B is used to convert the generated fog image Fake A1 into a clear image Cyclic B again;

[0091] Generator B2A2 is used to generate the clear image Cyclic B and then transform it into a new fog image Fake A2.

[0092] Generators A2B1, B2A, B2A1, and A2B use CM-U-Net++, a combination of U-Net++ and CABM channel attention. U-Net++ has excellent dependency on features from previous and next layers, which improves generator performance. Adding CABM attention channels also filters channels, enhancing channels with useful information and enabling faster data fitting. The generators use two different architectures. Multi-architecture generative networks can further enhance model diversity. The first-layer generator uses U-Net++ as the generation structure, while the second-layer generator adopts CycleGAN's own generation method. U-Net++ is an improved version of the U-Net network model. Compared to U-Net, it integrates information from multiple layers based on skip links. Compared to the original CycleGAN generation model, it can retain more original image information and better preserve the corresponding image during image domain transfer. To enhance the model's understanding capabilities, a spatiotemporal attention CBAM layer is introduced in U-Net++. Generator A2B2 and Generator B2A2 use generators composed of residual blocks of ResNet.

[0093] The discriminator is used to compare the electronic images generated by the generator with the images in the image dataset; the discriminator adopts the CNN skeleton network and the Transformer Encoder-Decoder structure. Among them, the Transformer is a model based on the multi-head attention mechanism. This model can solve the existing long-distance dependency problem, has better memory, and reduces lengthy and complex calculations to achieve more accurate recognition effects. However, it also loses the ability to capture local features. Therefore, in the original model, the CNN convolutional network is introduced to extract the underlying features to better capture local features.

[0094] In the embodiment of the present application, the discriminator includes discriminator A1, discriminator A2, discriminator B1 and discriminator B2, wherein:

[0095] Discriminator A1 is used to compare the fog image InputA input to generator A2B1 with the fog image Fake A1 generated by generator B2A1;

[0096] Discriminator A2 is used to compare the fog image InputA input to generator A2B1 with the fog image Fake A2 generated by generator B2A2;

[0097] Discriminator B1 is used to compare the clean image InputB input to generator B2A1 with the clean image Fake B1 generated by generator A2B1;

[0098] The discriminator B2 is used to compare the clean image InputB input to the generator B2A1 with the clean image Fake B2 generated by the generator A2B2.

[0099] At the same time, based on CycleGAN, which is developed based on CGAN, a generation structure is added to a single cycle, and loss calculations are performed twice. The original CycleGAN will perform process generator optimization and discriminator optimization four times. In the TF2GAN model, the Cyclice generation graph is generated again and loss calculations are performed with fake generation, which helps to improve the robustness of the overall model.

[0100] TF2GAN model training specifically includes the following steps:

[0101] S21, sample domain image recognition processing training; reference Figure 4 , perform the following steps:

[0102] S211, randomly input a sample domain image (denoted as InputA), which is transformed by generator A2B1 to form a disordered Fake B1;

[0103] S212, input the generated Fake B1 into the generator B2A to generate Cyclic A after transformation;

[0104] S213, input the generated Cyclic A into the generator A2B2, and transform it again to generate a new Fake B2;

[0105] S214: Freeze the weights of the generator and enable the recognition of the discriminator.

[0106] Specifically, the FT2GAN model first inputs a sample image domain image A, which is transformed by generator A2B1 to form an unordered Fake B1, then transformed by generator B2A to generate Cyclic A, and then transformed by generator A2B2 to generate a new FakeB2. During a sample image domain image recognition training process, three image generation conversions need to be completed. During the image generation conversion process, the discriminator weights remain frozen. After the image generation conversion is completed, the generator weights are frozen, that is, the gradient is kept from backpropagating during training, and then the recognition of the Transformer-based discriminator is started. During a sample image domain image recognition training process, there are two discriminators, and the recognition of discriminator B1 and discriminator B2 is performed simultaneously.

[0107] Generator A2B1 converts the input sample domain image InputA, i.e., a foggy image (an image with unclear distribution), into a clear image, and generator B2A converts the generated clear image into a foggy image again. Its purpose is to calculate the gap between the original foggy image and the generated foggy image, and it is expected that the image distributions of the two will become more and more similar through training. InputA actually inputs a set of vectors of fixed length and a sample domain image, whose label is a clear image. After one training, it is expected to fit and generate a clear image through two inputs (i.e., a set of vectors of fixed length and a sample domain image) and a fixed output (i.e., the generated image). The embodiment of the present application also performs two loss calculations in succession to increase the stability of the neural network and improve the overall robustness of the TF2GAN model; wherein, the first loss calculation is used to optimize the generator A2B21 and the discriminator A1, and the second loss calculation is used to optimize the entire model. The first loss calculation is performed by generating the Cyclic generated image again, that is, converting the foggy image into a clear image, and then converting the generated clear image into a foggy image again. In theory, the sample domain image input to generator A2B1 and the foggy image generated by generator B2A are two identical images, which is also the expected conversion effect after the FT2GAN model is trained. However, in reality, during the training process, the two images are different. By comparing the two images and calculating the difference between them, the model is stabilized, so that the difference between the two images becomes smaller and smaller until the two images are the same in the later stage of training. The second loss calculation is performed on the clear image generated by generator A2B1 and the clear image generated by generator A2B2. In theory, the clear image generated by generator A2B1 should be the same as the clear image generated by generator A2B2, but in the actual training process, the two images are different. With continuous training, the difference between the two images gradually decreases until the two images are the same.

[0108] S22, out-of-sample image domain image recognition processing training; reference Figure 5 , perform the following steps:

[0109] S221, randomly input an out-of-sample domain image (denoted as InputB), which is transformed by generator B2A1 to form an unordered Fake A1;

[0110] S222, input the generated Fake A1 into generator A2B to generate Cyclic B after transformation;

[0111] S223, input the generated Cyclic B into the generator B2A2, and transform it again to generate a new Fake A2;

[0112] S224. Freeze the weight of the generator and enable the recognition of the discriminator.

[0113] Specifically, the FT2GAN model takes an out-of-sample image (InputB) as input, transforms it through generator B2A1 to form a disordered Fake A1, which is then transformed by generator A2B to generate Cyclic B, which is then transformed by generator B2A2 to generate a new Fake A2. At this point, the out-of-sample image recognition training is complete, and the generator weights are frozen, meaning that the gradients do not backpropagate during training. The discriminator weights are then unfrozen, and the Transformer-based discriminators (Discriminators A1 and A2) are enabled for recognition.

[0114] Generator B2A1 converts the input out-of-sample domain image B (a clear image) into a foggy image. Generator A2B converts the generated foggy image back into a clear image. The goal is to calculate the difference between the original and generated clear images, with the goal of making the image distributions of the two increasingly similar through training. Input B actually takes a set of fixed-length vectors and an out-of-sample domain image with a label of a foggy image. After one training run, the goal is to fit the foggy image using two inputs (a set of fixed-length vectors and an out-of-sample domain image) and a fixed output (the generated image).

[0115] This transformation process also undergoes two loss calculations: the first is used to optimize generator B2A1 and discriminator B1, and the second is used to optimize the entire model. The first loss calculation is performed by regenerating the cyclic generated graph, converting the clear image into a foggy image, and then converting the resulting foggy image back into a clear image. In theory, the out-of-sample image input to generator B2A1 and the clear image generated by generator A2B are identical, which is the expected conversion effect after training. However, during training, the two images are actually different. By comparing the two images and calculating the difference between them, the model is stabilized, reducing the difference between the two images until they are identical in the later stages of training. The second loss calculation is performed on the foggy image generated by generator B2A1 and the foggy image generated by generator B2A2. In theory, the foggy image generated by generator B2A1 should be identical to the foggy image generated by generator B2A2, but in practice, they are different. With continued training, the difference between the two images gradually decreases until they are identical.

[0116] The sample domain image processing training and the out-of-sample domain image processing training are carried out simultaneously, that is, the model converts the foggy image into a clear image while converting another clear image into a foggy image, so as to enhance the robustness of the entire model.

[0117] In the embodiment of the present application, it is expected that the scores obtained by each discriminator for InputA and InputB are as large as possible, and it is expected that the scores obtained by each group of discriminators for FakeA and FakeB are as small as possible. The scores here are category scores, that is, probability values obtained by calculating cross entropy. In this application, low scores are foggy images, that is, scores closer to 0 represent foggy images, and high scores are clear images, that is, scores closer to 1 represent clear images. When the discriminator score is lower for the input sample domain image and higher for the generated FakeB image, it means that the generated electronic image is very dissimilar to the input original image, and vice versa.

[0118] After the discriminator's recognition, due to the existence of the loss function, the discriminator is back-propagated, the discriminator's weight parameters are optimized, the generator is released, the discriminator is frozen, the generator is back-propagated, and the generator weights are updated. The loss function of CycleGAN is the sum of the XYX and YXY adversarial loss functions and the cycle consistency loss function. The adversarial loss function ensures the same distribution, and the cycle consistency loss function ensures the consistency of images between different domains. In the neural network, the unknown number is the weight W, and the final output result is the loss function Loss. At this time, if the entire neural network is regarded as a function equation and the weight W is reversely optimized, then its derivative equation is: Obviously, the weight W corresponds to the independent variable x in the equation, and the loss function Loss corresponds to the dependent variable y. When calculating the minimum value of a function, from the perspective of data, it is to calculate the extreme point of the function. Calculating the minimum value of the function can be used to find the minimum value of the derivative, or the reciprocal is 0, which may be the minimum value of the function. Therefore, when optimizing the model, x is continuously updated to find the minimum of y. Based on the idea of gradient descent, W + =W-η

[0119] Among them, W + represents the updated weight, Represents the derivative of loss with respect to the unupdated weights, and η represents a hyperparameter learning rate, which adjusts the update speed of the weights. At the same time, adjustment can prevent the loss function from entering a local optimum.

[0120] Based on CycleGAN, TF2GAN adds the generation of Y in the XYX process, enhancing its migration robustness through a two-step chain structure. The overall loss function now adds two additional losses, XY loss and YY loss, to the existing loss in CycleGAN. The XY loss is an adversarial loss, similar to the loss in CycleGAN, but with the addition of the new XY process. The YY loss is a cycle consistency loss. Based on the loss calculation between XX, the YY loss calculation is added to form a secondary chain structure of XYXY.

[0121] During the sample domain image processing training process, TF2GAN is obtained by improving the task based on CycleGAN. Therefore, the total loss function formula of TF2GAN is preset to:

[0122] L(G,F,E,DX,DY)=L(G,E,F,D X ,D Y1 ,D Y2 )+λL cyc (G,F,E);

[0123] Among them, L(G,E,F,D X ,D Y1 ,D Y2 ) is the total adversarial loss function, L cyc (G, F, E) is the cycle consistency loss function, λ is the weight system, and its value range is 0-1. Its value is adjusted according to the TF2GAN model training results.

[0124] The total adversarial loss function formula is set as:

[0125] L(G,E,F,D X ,D Y1 ,D Y2 )=L GAN (G,D Y1 ,X,Y)+L GAN (F,D X ,Y,X)+L GAN (E,D Y2 ,X,Y).

[0126] Among them, the loss function formula of the adversarial loss of each generator is set as follows:

[0127] Loss function of generator A2B1: min G max Dy L GAN (G, D Y1 , X, Y);

[0128] Where G is the generator A2B1 model; D Y is the discriminator B1 model for Y; min G max Dy L GAN (G, D Y1 , X, Y) means minimizing the generator A2B1 loss and maximizing the discriminator B1 loss.

[0129] L GAN (G,D Y1 ,X,Y)=E y-pdata(y) [log(D Y1(y))]+E x-pdata(x) [log(1-D Y1 (G(x)))];

[0130] Among them, X is the sample of X space, X space is the sample domain; Y is the sample of Y space, Y is the sample domain; E y-pdata(y) Refers to the sample taken from the Y space, E x-pdata(x) Refers to the sample taken from the X space, x is the sample taken from the X space by the input generator A2B1, G(x) represents the picture generated by the generator A2B1, D Y1 (y) represents the probability that the discriminator B1 judges whether y1 is a sample taken from the Y space, because the Y space represents the out-of-sample image domain, that is, the sample taken from the Y space is a clear image, so D Y1 The closer (y) is to 1, the better, so it is max Dy . D Y1 (G(x)) is the probability that the discriminator B1 determines whether the image generated by the generator A2B1 is a sample taken from the Y space, so D Y1 The closer (G(x)) is to 0, the better, so it is min G .

[0131] Generator A2B1 hopes that the generated picture is as close to the sample in Y space as possible, that is, generator A2B1 hopes that D Y1 The discrimination probability is as large as possible, then [log(1-D Y1 (G(x)))] is smaller, then L GAN (G, D Y , X, Y) will become smaller, and the generator A2B1 hopes that L GAN (G, D Y , X, Y) the smaller the better, so it is min G , so the calculation optimization is to minimize L GAN (G, D Y , X, Y). The stronger the ability of the discriminator B1, the better the Y1 (y) should be larger, D Y1 (G(x)) should be smaller, so the generator A2B1 and the discriminator B1 are different, and the discriminator B1 hopes to be L GAN (G, D Y , X, Y) the bigger the better, so the optimization is to maximize L GAN (G, D Y , X, Y).

[0132] Similarly, generator B2A: min F max Dx L GAN (F,D X ,Y,X);

[0133] Among them, F is the generator B2A model, DX is the discriminator B2's model for FakeB2, min F max Dx L GAN (F,D X ,Y,X) means minimizing the generator B2A loss and maximizing the discriminator B2 loss.

[0134] L GAN (F,D X ,Y,X)=E x-pdata(x) [log(D X (x))]+E y-pdata(y) [log(1-D X (G(z)))];

[0135] Among them, x represents the sample input to generator B2A from X space, G(z) represents the picture generated by generator B2A, and D X (x) represents the probability that the discriminator B2 judges whether x is a sample taken from the X space, D X (G(z)) is the probability that the discriminator B2 judges whether the image generated by the generator B2A is a sample taken from the X space. The generator B2A hopes that the image it generates is as close to the sample of the X space as possible, that is, the generator B2A hopes that D X (G(z)) the probability of discrimination is as large as possible, then [log(1-D X (G(z)))] is smaller, at this time, L GAN (F,D X ,Y,X) will become smaller, the generator B2A hopes that L GAN (F,D X ,Y,X) the smaller the better, so it is min F , the calculation optimization is to minimize L GAN (F,D X ,Y,X). The stronger the ability of the discriminator B2 is, the better the X The larger (x) is, the greater D X The smaller (G(z)) is, the better the discriminator D is. X I hope so GAN (F,D X ,Y,X) the larger the better, the calculation optimization is to maximize L GAN (F,D X ,Y,X).

[0136] Generator A2B2:min E max Dx L GAN (E,D Y2 ,X,Y);

[0137] Among them, E is the generator A2B2 model, D Y is the discriminator B2 model for Y, min E max Dx L GAN (E,D Y2 ,X,Y) means minimizing the loss of generator A2B2 and maximizing the loss of discriminator B2.

[0138] L GAN (E,D Y2 ,X,Y)=E y-pdata(y) [log(D Y2 (y))]+E x-pdata(x) [log(1-D Y2 (G(x)))];

[0139] Where y represents sampling in the Y space, x represents the sample taken from the X space by the input generator A2B2, and G(x) represents the image generated by the generator A2B2; D Y2 (y) represents the probability that the discriminator B2 judges whether y is a sample taken from the Y space; D Y2 (G(x)) is the probability that the discriminator B2 judges whether the image generated by the generator A2B2 is a sample taken from the Y space. The generator A2B2 hopes that the image generated by the generator is as close to the sample of the Y space as possible, that is, the generator A2B2 hopes that D Y2 (G(x)) the probability of discrimination is as large as possible, then [log(1-D Y2 The smaller (G(x)))] is, the lower the L GAN (E,D Y ,X,Y) will become smaller, and the generator A2B2 hopes that L GAN (E,D Y ,X,Y) the smaller the better, so it is min E , and finally because of min E , when the cycle consistency loss is not considered for optimization, it is to minimize L GAN (E,D Y ,X,Y). The stronger the ability of the discriminator B2, the better the D Y2 The larger (y) is, the greater D Y2 The smaller (G(x)) is, the more the discriminator B2 hopes that L GAN (E,D Y ,X,Y) the larger the better, so the optimization is to maximize L GAN (E,D Y ,X,Y).

[0140] The cycle consistency loss is to ensure that the graphs can correspond one to one during the migration process. In the CycleGAN generation mode, the loss function formula of CycleGAN is set as follows:

[0141] L cyc (G,F)=E x-pdata(x) [||F(G(x))-x||1]+E y-pdata(y) [||G(F(y))-y||1];

[0142] Among them, E x-pdata(x) [||F(G(x))-x||1] This part calculates the L1 loss of the migration change between the two Xs XYX; E y-pdata(y) [||G(F(y))-y||1] is the L1 loss of the migration change between two Y values YXY. L1 is the sum of the absolute differences between the target value Yi and the estimated value f(xi) is minimized.

[0143] In the CycleGAN model, cycle consistency only calculates the L1 loss between inputA and cyclicA, and inputB and cyclicB. In the embodiment of this application, the TF2GAN model improves on the CycleGAN model, so the L1 loss is changed to L2 loss on this basis, and the L2 loss between FakeA1 and FakeA2, and FakeB1 and FakeB2 is added. Therefore, the cycle consistency loss function formula of TF2GAN is set as the following formula:

[0144] L cyc (G,F,E)=E x-pdata(x) [||√F(G(x))-√x)||2]

[0145] +E x-pdata(x) [||√E(F(G(x)))-√G(x)||2]

[0146] +E y-pdata(y) [||√G(F(y))-√y||2]

[0147] +E y-pdata(y) [||√E(G(F(y)))-√F(y)||2];

[0148] Among them, E x-pdata(x) [||√F(G(x))-√x)||2] This part calculates the L2 loss of the migration change between InputA and CycleA; E x-pdata(x) [||√E(F(G(x)))-√G(x)||2] This part calculates the L2 loss of the migration change between InputB and CycleB; Ey-pdata(y) [||√G(F(y))-√y||2] This part calculates the L2 loss of the migration change between fakeB1 and fakeB2; E y-pdata(y) [||√E(G(F(y)))-√F(y)||2] This part calculates the L2 loss of the migration change between fakeA1 and fakeA2. The L2 loss is to minimize the sum of the squared differences S' between the target value Yiyu and the estimated value f(xi). Using these four L2 losses to calculate the correspondence between the generated graphs can greatly improve their generation robustness.

[0149] S3. Perform image conversion on the input remote sensing image based on the trained TF2GAN model.

[0150] Specifically, after a large amount of image processing training, the trained TD2GAN network is exported as an onnx neural network node graph. The exported onnx neural network model node is a topological structure. TensorRT is used to deploy a C++ program to build an onnx-based prediction model.

[0151] Reference Figure 6 After the prediction model is established, the remote sensing image domain is first determined before the remote sensing image is input into the TF2GAN model. If the remote sensing image domain is the source domain, the image to be tested is input into the prediction model and converted into an image in the target domain. The image is then input into the segmentation network to perform module region segmentation. This reduces the labor intensity and annotation time of manually establishing the migration domain in the dataset. If the remote sensing image domain is the target domain, the image is directly input into the segmentation network to perform module region segmentation.

[0152] In one embodiment, a remote sensing map image domain migration system based on GAN is provided, referring to Figure 7 The image domain migration system includes: an image processing module 100, a model building module 200, and an image conversion module 300, wherein:

[0153] Image processing module 100, used to acquire remote sensing images and establish image data sets;

[0154] The model construction module 200 is used to construct a TF2GAN model and input the images in the image dataset into the TF2GAN model for model training;

[0155] The image conversion module 300 is used to perform image conversion on the input remote sensing image according to the trained TF2GAN model.

[0156] The image processing module 100 downloads remote sensing images within the corresponding target area and screens and classifies the remote sensing images according to whether the distribution of the target images is clear. The target images with clear distribution are used as out-of-sample domain images, and the target images with unclear distribution are used as sample domain images. The sample domain images and the out-of-sample domain images are used as image data to establish an image data set.

[0157] The model construction module 200 includes a generator, a discriminator and a training unit. The training unit is pre-set with a loss function. The generator and the discriminator are supervised and trained through the loss function to obtain a trained TF2GAN model, which is used to improve the recognition effect of the model.

[0158] After the TF2GAN model training is completed, the image conversion module 300 obtains the remote sensing image and inputs it into the trained TF2GAN model for image domain migration conversion. The image generated after the image domain migration conversion is then segmented by the remote sensing map segmentation model.

[0159] In one embodiment, a computer device is provided, referring to Figure 8 , the computer device can be a terminal, which includes a processor, memory, network interface, display screen and input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities; the memory includes a non-volatile storage medium and an internal memory, the non-volatile storage medium stores an operating system and a computer program, and the internal memory provides a running environment for the operating system and computer program in the non-volatile storage medium; the network interface is used to communicate with an external terminal through a network connection; the display screen is used to display a graphical interface during and after processing; the input device can be a keyboard, mouse, touchpad that comes with or is external to the computer device, or it can be a touch layer on the display screen. The memory stores a computer program, and when the computer program is executed by the processor, it implements a remote sensing map image domain migration method based on GAN.

[0160] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0161] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, the processes in the above-described method embodiments can be implemented. Non-volatile memory includes read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.

[0162] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A GAN-based remote sensing map image domain migration method, characterized by: The method comprises: Acquire remote sensing images and establish an image data set; the image data set includes a sample image domain image and an out-of-sample image domain image, the sample image domain image is a fogged image, and the out-of-sample image domain image is a clear image; Construct a TF2GAN model and input the images in the image dataset into the TF2GAN model for model training; the TF2GAN model includes a generator and a discriminator; among them, The model training includes sample domain image recognition processing training and out-of-sample domain image recognition training; The TF2GAN model has a preset total loss function, and the generator and the discriminator are trained according to the preset total loss function of the TF2GAN model and the image dataset to obtain a trained TF2GAN model; Perform image conversion on the input remote sensing image according to the trained TF2GAN model; The total loss function of the TF2GAN model includes the total adversarial loss function and the cycle consistency loss function. The formula of the cycle consistency loss function is: L cyc (G,F,E)=E x-pdata(x) [||√F(G(x))-√x)||2] +E x-pdata(x) [||√E(F(G(x)))-√G(x)||2] +E y-pdata(y) [||√G(F(y))-√y||2] +E y-pdata(y) [||√E(G(F(y)))-√F(y)||2]; Among them, y represents sampling in Y space, x represents the sample taken by the input generator from X space, and G(x) represents the image generated by the generator; D Y2 (y) represents the probability that the discriminator determines whether y is a sample taken from the Y space; D Y2 (G(x)) is the probability that the discriminator determines whether the image generated by the generator is a sample taken from the Y space.

2. The GAN-based remote sensing map image domain migration method according to claim 1, characterized in that: The formula for the total loss function is: L(G,F,E,DX,DY)=L(G,E,F,D X ,D Y1 ,D Y2 )+λL cyc (G,F,E)? Among them, L(G,E,F,D X ,D Y1 ,D Y2 ) is the total adversarial loss function, L cyc (G, F, E) is the cycle consistency loss function, and λ is the weight coefficient.

3. The GAN-based remote sensing map image domain migration method according to claim 1, characterized in that: The generator is used to process pictures in the image dataset to generate new electronic pictures, and the generator includes: Generator A2B1 is used to convert a sample domain image into a clear image Fake B1; Generator B2A is used to convert the generated clear image Fake B1 into the fogged image Cyclic A again; Generator A2B2 is used to convert the generated fog image Cyclic A into a new clear image Fake B2; Generator B2A1 is used to convert a sample external domain image into a fog image Fake A1; Generator A2B is used to convert the generated fog image Fake A1 into a clear image Cyclic B again; Generator B2A2 is used to convert the generated clear image Cyclic B into a new fog image Fake A2 again; The discriminator is used to compare the electronic image generated by the generator with the images in the image dataset, and the discriminator includes: discriminator A1 is used to compare the sample image domain image input to generator A2B1 with the fog image FakeA1 generated by generator B2A1; Discriminator A2 is used to compare the sample domain image input to generator A2B1 with the fog image FakeA2 generated by generator B2A2; Discriminator B1 is used to compare the out-of-sample domain image input to generator B2A1 with the clear image Fake B1 generated by generator A2B1; The discriminator B2 is used to compare the out-of-sample domain image input to the generator B2A1 with the clear image Fake B2 generated by the generator A2B2.

4. The GAN-based remote sensing map image domain migration method according to claim 3, characterized in that: The generators A2B1, B2A, B2A1, and A2B use CM-U-Net++, which is a combination of U-Net++ and CABM channel attention. The generators A2B2 and B2A2 are generators composed of residual blocks of ResNet; The discriminator adopts the skeleton network of CNN and the Encoder-Decoder structure of Transformer.

5. The GAN-based remote sensing map image domain migration method according to claim 3, characterized in that: The sample image domain image recognition processing training model includes the following steps: A sample domain image is randomly input, and after being transformed by the generator A2B1, a disordered Fake B1 is formed; The generated Fake B1 is input into the generator B2A to generate CyclicA after transformation; Input the generated Cyclic A into the generator A2B2, and transform it again to generate a new Fake B2; Freeze the weights of the generator and enable the recognition of the discriminator.

6. The GAN-based remote sensing map image domain migration method according to claim 4, characterized in that: The out-of-sample image domain image recognition processing training includes the following steps: A random out-of-sample domain image B is input, and after being transformed by the generator B2A1, a disordered Fake A1 is formed; The generated Fake A1 is input into the generator A2B to generate Cyclic B after transformation; Input the generated Cyclic B into the generator B2A2, and transform it again to generate a new Fake A2; Freeze the weights of the generator and enable the recognition of the discriminator.

7. The GAN-based remote sensing map image domain migration method according to claim 1, characterized in that: The sample domain image processing training and the out-of-sample domain image processing training are performed simultaneously.

8. A system using the GAN-based remote sensing map image domain migration method according to any one of claims 1 to 7, characterized in that: It includes an image processing module (100), a model building module (200) and an image conversion module (300); The image processing module (100) is used to acquire remote sensing images and establish an image data set; The model construction module (200) is used to construct a TF2GAN model, and input the images in the image dataset into the TF2GAN model for model training; The image conversion module (300) is used to perform image conversion on the input remote sensing image according to the trained TF2GAN model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the image domain migration method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is read and executed by a processor, the image domain migration method according to any one of claims 1 to 7 is implemented.

Images