A method for constructing a remote sensing image deep learning model, recording media and system

Abstract

The present invention belongs to the field of image data processing technology, and particularly relates to a method for constructing a remote sensing image deep learning model. The present invention proposes a method for constructing a dual generative adversarial network model including residual-resampling for unsupervised image conversion and cross-domain semantic segmentation of drone remote sensing images based on a generative adversarial network model based on cycle consistency. The model constructed by the method uses a resampling module to fully utilize the scale information of drone images, and through residual connection, the function of the generator is changed from generating new images to generating residual information, thereby enhancing the degree of retention of semantic information in the generation process, and a discriminator is added at the end of an adaptive framework to optimize the output space and semantic segmentation results. The present invention also provides a readable recording medium storing the method program and a system containing the medium, and the program can be called through a processing circuit to execute the above method.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing and discloses a method for constructing a remote sensing image deep learning model, a recording medium and a system. Background Art

[0002] Semantic segmentation technology plays an important role in tasks such as water area extraction and identification of environmental elements around water bodies. However, due to the limitations of deep learning models themselves, when a model trained on a well-labeled training set is deployed in a real environment, the performance of the model often declines due to differences in imaging equipment, time, and region between the real environment and the training set images. The purpose of domain adaptation methods is to improve the performance of the model in this case. There are two main problems with existing domain adaptation (referred to as "domain adaptation", the same below) methods: First, they ignore the "real 1-real 2" characteristics of drone image translation (traditional computer vision image translation is virtual-real translation of GTA5 dataset to Cityscape); second, they ignore the impact of different resolutions on domain adaptation, which leads to a deviation between the results of deep learning and the actual expected results of drone image acquisition.

[0003] The core of the adaptive method based on adversarial learning is to let the model learn the distribution difference of such data by training the discriminator to identify whether the input information comes from a specific data domain. The discriminator can be involved in different stages of the entire deep learning framework to achieve data alignment in the two fields or let the network's feature extractor and decoder learn the data distribution difference in the two fields, thereby improving the performance on the target domain data, but such technology has two main limitations. First, aligning the marginal distribution does not enforce any semantic consistency. For example, the target feature of a water body may be mapped to the source feature of a building. Second, alignment at a higher level of deep representation may not simulate the low-level appearance differences that are critical to the final visual task.

[0004] As for domain adaptation of remote sensing image semantic segmentation, the performance of cross-domain semantic segmentation of high-resolution remote sensing images has been greatly improved by using an unsupervised domain adaptation method based on a generative adversarial network. The generative adversarial network structures used for remote sensing image translation mostly directly reference models in the field of computer vision, such as CycleGAN and DualGAN. However, there are large differences between remote sensing images and everyday images in computer vision tasks, so directly using the corresponding model may bring the following problems:

[0005] First, the existing unsupervised image translation algorithms cannot solve the spatial scale differences of remote sensing images across domains, and scale is one of the important attributes of earth observation images. Since optical sensors for aerospace or aviation earth observation usually have a fixed focal length, changes in the sensor's viewing angle and height will directly affect the resolution of the image, so fixed camera angles are usually used for imaging at a fixed height. The number of pixels occupied by the same ground object target in remote sensing images with different spatial resolutions is different. In the field of computer vision, taking the on-board camera used for autonomous driving as an example, during the process of motion imaging, the scale of different objects changes continuously according to the distance from the camera position. In different street view datasets, the target objects exist in a multi-scale state, so there is no scale effect problem in remote sensing images. Therefore, when performing cross-domain image conversion of remote sensing images, the spatial scale difference problem requires additional network modules to be aligned, otherwise it will lead to a decrease in the segmentation accuracy of specific scale-sensitive ground object categories. Considering that the scale effect caused by the spatial scale difference between domain data is itself one of the sources of data distribution differences, explicit spatial scale conversion is an important way to reduce distribution differences.

[0006] Secondly, the field of image conversion in computer vision includes not only real-world images, but also non-realistic image data such as paintings and games. Therefore, the generator and discriminator are not designed to retain or enhance local semantic information, but only to make the generated data look like it comes from the corresponding field. This network structure may cause the loss of key semantic information. However, the cross-domain adaptation of remote sensing images is all real-world information and has real geographical significance. The data generated during the image translation process is likely to bring changes to the structural information of a certain object, thereby affecting the subsequent semantic segmentation task. Summary of the invention

[0007] In view of the above problems, the present invention provides a method for constructing a remote sensing image deep learning model, and the specific scheme includes the following steps:

[0008] S1. Construct a bidirectional generative adversarial neural network, wherein the bidirectional generative adversarial neural network comprises:

[0009] Generator of source-to-target resampling residuals , used to perform image conversion from the source domain S to the target domain T, and adjust the image resolution to be consistent with the target domain through resampling technology;

[0010] Generator of target-to-source resampling residuals , used to perform image conversion from the target domain T to the source domain S, and adjust the image resolution to be consistent with the source domain through resampling technology;

[0011] Discriminator , used to determine whether the image is from the source domain S or from Generator, Discriminator , used to determine whether the image is from the target domain T through adversarial training, or generate;

[0012] S2. Using bidirectional generative adversarial neural networks to obtain target domain-specific source domain data And the corresponding labeled data , set up supervised training of a semantic segmentation model For unlabeled target domain data Prediction of; in the semantic segmentation model The output space of the discriminator is set to a discriminator that can discriminate the field to which the segmentation result of the output space belongs through adversarial training. , complete the construction of the deep learning model of remote sensing images.

[0013] Preferably, the adversarial training is performed by iteratively calculating the discriminator loss , until Reduce to the set threshold and complete the training.

[0014] Preferably, the method further comprises the following steps:

[0015] Calculate the reconstruction loss between the original image in the same domain and the image transferred to another domain by the generator and then transferred back from another domain ; Calculate by alternating iterations and , and compare it with the preset threshold to train the generator and discriminator.

[0016] Preferably, the method of adjusting the image resolution to be consistent with the target domain is a process of adjusting the source domain to a spatial scale similar to the target domain data through a resampling function, and a bilinear interpolation function is selected as the resampling function.

[0017] Preferably, the generator is a fully convolutional network including an encoder and a decoder, and there is a skip connection between the encoder and the decoder. The encoder acts as a downsampling layer, sets the convolution kernel size to 4, the padding to 1, and the step size to 2, and the convolution kernel depths of each convolution layer are 64, 128, 256, 512, 512, 512, 512, respectively, wherein except for the first and last layers, each convolution layer is instance normalized and includes a Leaky ReLU activation function with a negative slope of 0.2; the decoder acts as an upsampling layer, and adopts the same convolution kernel depth, size, step size and padding parameters as the encoder, and the layers in the encoder and decoder where the convolution kernel depth exceeds 256 also adopt a removal layer with a probability of 0.5.

[0018] Preferably, the discriminator adopts a fully convolutional network, and the convolution kernel depth of each layer is 64, 128, 256, 512, 512, 1. Except for the last layer, other layers include a batch normalization layer and a Leaky ReLU activation function with a negative slope of 0.2.

[0019] Another embodiment of the present invention is to provide a readable recording medium for storing one or more programs containing multiple instructions. When the program is run, it will cause the processing circuit to execute the above-mentioned method for constructing the remote sensing image deep learning model.

[0020] Another embodiment of the present invention is to provide a system for constructing a deep learning model for remote sensing images, including a processing circuit and a memory electrically coupled thereto, wherein the memory is configured to store at least one program, wherein the program includes multiple instructions, and the processing circuit runs the program to execute the above-mentioned method for constructing a deep learning model for remote sensing images.

[0021] Compared with the prior art, the present invention produces the following beneficial effects:

[0022] The remote sensing image deep learning model constructed by the method proposed in this invention aims to learn the semantic segmentation model of remote sensing images from labeled datasets to unlabeled datasets to minimize the domain gap at the pixel level. Considering the scale differences of objects of different scales in drone images, a resampling module is used to improve the accuracy of categories of different scales.

[0023] Considering the characteristics of real-to-real translation of drone images, residual connections are used at the same time. Combined with the resampling module, the residual connection structure plays a key role in achieving cross-domain data distribution matching performance. The residual connection retains the semantic information of the original data through the identity connection and avoids the modification of the structural information. The drone image contains the real scene of ground observation, so the conversion between domains is also from "reality" to "reality", in which all pixels have geographical significance. At the same time, the model should be modified as little as possible to affect the real scene structure information of the subsequent segmentation task. Therefore, the residual connection structure is specially designed in the model to retain the original structural information as much as possible, and focus on the conversion of other basic features, such as color, shadow, etc. Therefore, the residual connection improves the effective subsequent segmentation performance and is more suitable for mutual conversion between drone image domains. In addition, the residual connection can also stabilize the training process of the model. The function of the generator of the traditional dual generative adversarial network is to generate a target stylized image from the input image, while the function of the generator of this method is to generate a residual term from the input image, which frees the generator in the traditional dual generative adversarial network from the burden of generating new images with the target style and retains the information of the input image to a large extent, thereby stabilizing the training process.

[0024] The present invention also combines the adaptability of the output space to further improve the performance of the model. The entire adversarial learning training framework achieves a higher accuracy performance on the benchmark, and has the advantages of simple structure and good stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of data flow of a remote sensing image generative residual-resampling adversarial network model in an embodiment of the present invention;

[0026] Figure 2 Schematic diagram of adversarial learning between the supervised semantic segmentation sub-model and the output space in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. The described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making innovative efforts are within the scope of protection of the present invention.

[0028] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0029] like Figure 1 As shown in FIG, a method for constructing a deep learning model for remote sensing images is constructed. The construction of the model includes two stages. Stage I uses a generative spatial scale conversion adversarial network (ResiDualGAN) to perform unsupervised image conversion from the source domain to the target domain; Stage II uses the target-stylized source domain data and corresponding label data generated in Stage I to perform semantic segmentation on the model. Conduct supervised training to establish a mapping relationship between the target domain data distribution and the labeled data. In order to optimize the segmentation results of the target domain and increase the application of adversarial learning in this framework, an output space adversarial learning discriminant network module is added to the terminal output space of the segmentation model to improve the performance of the cross-domain semantic segmentation model. The following is a detailed introduction to the two-stage training process.

[0030] 1. Generative Residual-Resampling Adversarial Network

[0031] like Figure 1 As shown in Figure 2, the main goal of Phase I is to align pixel-level data distribution in the input space through a generative adversarial network to generate target-domain source domain data, which is expressed as Converts to something like data distribution. To this end, the present invention designs a generative residual-resampling adversarial network for image data distribution feature matching in the input space, named ResiDualGAN. Similar to conventional generative adversarial networks, ResiDualGAN consists of dual generators and discriminators. The difference is that the resampled residual generator (ResiGenerator) in ResiDualGAN can resample while generating images. The discriminator is used to determine whether the input image is an image generated by ResiGenerator for adversarial learning. Specifically, ResiDualGAN contains two resampled residual generators: and , and two discriminators: and .in Used to perform image conversion and resampling from the source domain S to the target domain T, Perform image conversion and resampling from the target domain T to the source domain S. The discriminator Used to determine whether the image is from the source domain S or from , discriminator Used to determine whether the image is from the target domain T or from The principle of ResiDualGAN for image style conversion is as follows: Randomly input source domain data Target domain data , source domain input image Through the scale conversion generator Then the source domain data is converted to the target domain and the source domain data is target-stylized. . and Sent to the target domain discriminator , Try to distinguish whether the input of the discriminator network comes from the source domain generation or the target domain itself, and calculate the discriminative loss This allows for adversarial learning. Then through the reverse generator Convert back to the source domain to get the reconstructed image Based on the cycle consistency principle, ideally, the source domain input image of the forward and reverse generators The image that should be reconstructed with So the difference between the two can be used to calculate the consistency loss (also called reconstruction loss). . Target domain input data The process through the network is the above input The inverse process of is not repeated here. By alternately calculating the reconstruction loss and the discriminator loss, the two generators and the discriminator are trained to perform bidirectional generative adversarial learning.

[0032] Modeling process in Phase I:

[0033] (1) Construction of the resampling residual generator sub-model

[0034] The conversion process from source domain to target domain like Figure 2 As shown in Figure 2. This process is completed by the resampled residual generator (ResiGenerator), which converts the source domain data into target domain data and completes the resampling at the same time. ResiGenerator contains a generator based on a fully convolutional neural network. and a resampling module . Generator Generate a residual term from the randomly sampled source domain data, the resampling module is a resampling function that adjusts the source domain to a spatial scale similar to the target domain data. Based on the ablation experiment, we chose the bilinear interpolation function as the function used by this module. The resampled residual generator Transformed into target domain The process is as follows:

[0035]

[0036] The formula omits the random noise involved in the above process. According to the principle of spatial resolution scale consistency, after resampling, The spatial resolution and be consistent. The structure and Similar, the only difference is the resampling module yes The reverse process is to convert the size The image is resampled to a size of Therefore, The size and resolution should be consistent with the labeled data in the source domain. same.

[0037] (2) Sub-model network structure setting

[0038] The structure of the resampled residual generator consists of a generator and a resampling module. The generator is a fully convolutional network including an encoder and a decoder, with skip connections between the encoder and the decoder. For the encoder (downsampling layer), the convolution kernel size is set to 4, the padding is 1, and the stride is 2. The convolution kernel depths of each convolution layer are 64, 128, 256, 512, 512, 512, 512, respectively. Except for the first and last layers, each convolution layer has been instance normalized and a Leaky ReLU activation function with a negative slope of 0.2. The decoder (upsampling layer) uses the same convolution kernel depth, size, stride, and padding as the encoder. The layers in the encoder and decoder with a convolution kernel depth exceeding 256 also use a Dropout layer with a probability of 0.5. The resampling module is implemented using bilinear interpolation. The discriminator in ResiDualGAN is also implemented using a fully convolutional network. The convolution kernel depths of each layer are 64, 128, 256, 512, 512, 1. Except for the last layer, all other layers contain batch normalization and Leaky ReLU activation function with negative slope of 0.2. Due to the limitation of Wasserstein GAN (WGAN), the output of the last layer does not pass through any activation function.

[0039] (3) Sub-model objective optimization function

[0040] The resampled residual generator generates images for training the discriminator, and the discriminator attempts to determine the source of the generated image, thereby obtaining the loss function of adversarial learning . The present invention uses the loss of WGAN for adversarial loss. WGAN uses the Earh-Mover (EM) distance as a metric for the difference in data distribution, which solves the problem that the distance is 0 when the two data distributions have no intersection at all, thereby avoiding the problem of gradient vanishing. Unlike the traditional generative adversarial network that uses Sigmoid output as the final output of the discriminator, ResiDualGAN removes the Sigmoid layer as the final output. In addition, the gradient penalty is used to stabilize the training process and avoid the weight clipping operation in WGAN to achieve the Lipschitz constraint. For simplicity, the gradient penalty is not reflected in the formula. In summary, the adversarial loss from the source domain S to the target domain T As shown below:

[0041]

[0042] Will Reconstructed to the style of the source domain to enforce cycle consistency of dual learning. Ideally, Should be with Exactly the same. But there are differences in practice, and these differences are used to calculate the reconstruction loss function. Reconstruction loss function It can be measured by the L1 regularization term:

[0043]

[0044] Taking into account the adversarial loss and reconstruction loss, the total loss of ResiDualGAN can be written as:

[0045]

[0046] in and are the hyperparameters corresponding to the reconstruction loss and the discrimination loss. In the training of ResiDualGAN, the discriminator tries to maximize , and ResiGenerator tries to minimize, which can be written as the following min-max criterion:

[0047]

[0048] After the training of phase I is completed, a resampled residual generator can be obtained To perform the conversion task in the principle of spatial resolution scale consistency, generate Used for Phase II training.

[0049] 2. Supervised Semantic Segmentation Sub-Model and Output Space Adversarial Learning

[0050] The goal of Phase II is to use the source domain data generated by ResiDualGAN to generate the target domain And the corresponding labeled data , supervised training of a semantic segmentation model For unlabeled target domain data In addition to the semantic segmentation model, Phase II also Set a discriminator in the output space , which is used to determine which field the segmentation result of the output space belongs to, so as to perform adversarial learning and enhance the ability of the semantic segmentation model to extract invariant features of the target domain and the source domain. and the output space discriminator The combination of can be regarded as a typical generative adversarial network model: As the generator in the traditional generative adversarial network, and Generate prediction results and set a semantic discriminator in the output space of the model To judge The output is from still .

[0051] (1) Sub-model network structure setting

[0052] The entire generative residual-resampling adversarial learning domain adaptation framework can be applied to any semantic segmentation model, so the semantic segmentation network can be adapted according to the actual task. Make a selection. In the method, The Deeplabv3+ semantic segmentation model is used as the segmentation model benchmark. In order to accelerate the convergence speed, the backbone feature extraction network (backbone) uses the relatively shallow ResNet-34 and uses the ImageNet pre-trained weights. Output space discriminator It is also a full convolutional network, and the convolution kernel size is , stride 2, kernel depth 64, 128, 256, 512, 1. Except for the last layer, each layer contains a Leaky ReLU activation function with a negative slope of 0.2. The last layer is upsampled to the size of the input data. In this way, the semantic segmentation model can learn the domain-invariant distribution characteristics of the target domain data and the source domain data, thereby improving the performance of the model on the target domain data.

[0053] (2) Sub-model objective function optimization

[0054] Discriminator The training uses a binary cross entropy loss :

[0055]

[0056] because The scale is , and labeled data The scale is Therefore, the nearest neighbor interpolation method is used to unify The scale is The reason for using the nearest neighbor difference method for scale scaling is that it assigns the nearest pixel to the original pixel during the calculation process, so all interpolation comes from the original image and does not deviate from the range of the category label value in the labeled data. Semantic segmentation model The training uses a cross entropy loss:

[0057]

[0058] in for After the nearest neighbor interpolation, the labeled data is unified to the target domain scale as The labels of the target domain are used to supervise the model training. Data passing Get the output result , and then input to the discriminator For adversarial learning, the adversarial loss is:

[0059]

[0060] In summary, the overall loss function of the Phase II semantic segmentation model training is:

[0061]

[0062] in and They are the hyperparameters of the cross entropy loss function of the semantic segmentation model and the discriminant loss function of the output space adversarial learning. After training in Phase II, a cross-domain semantic segmentation model for drone images is finally obtained.

[0063] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computers, available storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0064] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0065] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0067] Compiling the above method steps into a program and then storing it on a hard disk or other non-transitory recording media, and electrically connecting it to a processor constitutes the technical solution of "a readable recording medium" of the present invention; and electrically connecting the readable recording medium to a data acquisition device, and completing the construction of a deep learning model for remote sensing images through data processing, constitutes the technical solution of "a system for constructing a deep learning model for remote sensing images" of the present invention.

[0068] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a remote sensing image deep learning model, characterized in that The following steps are involved: S1. Construct a bidirectional generative adversarial neural network, wherein the bidirectional generative adversarial neural network comprises: Generator ResiG of source-to-target resampling residuals S→T , used to perform image conversion from the source domain S to the target domain T, and adjust the image resolution to be consistent with the target domain through resampling technology; Generator ResiG of target domain to source domain resampling residual T→S , used to perform image conversion from the target domain T to the source domain S, and adjust the image resolution to be consistent with the source domain through resampling technology; Discriminator D S , used to determine whether the image is from the source domain S or from ResiG through adversarial training T→S Generator, Discriminator D T , used to determine whether the image is from the target domain T or from ResiG through adversarial training S→T generate; S2. Using a bidirectional generative adversarial neural network to obtain the target domain source domain data X S→T And the corresponding label data Y S , set up supervised training of a semantic segmentation model f T For unlabeled target domain data X T prediction; in the semantic segmentation model f T The output space of the discriminator D is set to distinguish the field to which the segmentation result of the output space belongs through adversarial training. out , complete the construction of deep learning model for remote sensing images; The adversarial training iteratively calculates the discriminator loss L adv , until L adv Reduce to the set threshold and complete the training; The adversarial training in step S1 also includes the following steps: Calculate the reconstruction loss L between the original image in the same domain and the image transferred to another domain by the generator and then transferred back from another domain cyc ; Calculate L by alternating iterations cyc With L adv , and compare it with the preset threshold to train the generator and discriminator.

2. The method for constructing a remote sensing image deep learning model according to claim 1, characterized in that: The method for adjusting the image resolution to be consistent with the target domain is to adjust the source domain to a spatial scale similar to the target domain data through a resampling function, and a bilinear interpolation function is selected as the resampling function.

3. The method for constructing a remote sensing image deep learning model according to claim 2, characterized in that: The generator is a fully convolutional network consisting of an encoder and a decoder. There is a skip connection between the encoder and the decoder. The encoder is used as a downsampling layer, and the convolution kernel size, padding and stride are set to 4, 1, and 2, respectively. The convolution kernel depths of each convolution layer are 64, 128, 256, 512, 512, 512, 512, respectively. Except for the first and last layers, each convolution layer is instance normalized and contains a Leaky ReLU activation function with a negative slope of 0.

2. The decoder is used as an upsampling layer and uses the same convolution kernel depth, size, stride and padding parameters as the encoder. In the layers where the convolution kernel depth exceeds 256 in the encoder and decoder, a removal layer with a probability of 0.5 is also used.

4. The method for constructing a remote sensing image deep learning model according to claim 3, characterized in that: The discriminator uses a fully convolutional network with a convolution kernel depth of 64, 128, 256, 512, 512, and 1 in each layer. Except for the last layer, all other layers contain a batch normalization layer and a Leaky ReLU activation function with a negative slope of 0.

2.

5. A non-transitory readable recording medium for storing one or more programs including a plurality of instructions, characterized in that: When the instruction is executed, the processing circuit will be caused to execute a method for constructing a remote sensing image deep learning model as described in any one of claims 1-4.

6. A system for constructing a remote sensing image deep learning model, comprising a processing circuit and a memory electrically coupled thereto, characterized in that: The memory configuration stores at least one program, the program includes multiple instructions, and the processing circuit runs the program to execute a method for constructing a remote sensing image deep learning model according to any one of claims 1-4.

Images