Remote sensing image ground object sample conversion method based on diffusion model

Abstract

The invention discloses a remote sensing image ground object sample conversion method based on a diffusion model, and the method comprises the following steps: obtaining remote sensing data, and carrying out the preprocessing; constructing a neural network; the neural network takes a stable diffusion model as a basic network, and comprises a picture perception compression module, a potential diffusion model module and a seasonal invariant feature learning module. Training the neural network in stages to obtain a final neural network; and outputting a season conversion result. According to the method, a stable diffusion model is used as a basic generation model, and remote sensing images in different seasons are generated by using the strong generation capability of the diffusion model. According to the method, the seasonal invariant feature extraction network is constructed, the features of related ground features in different seasons are fully considered in the seasonal conversion process, and enough remote sensing ground feature information is ensured. According to the method, on the basis of a stable diffusion model, a domain-adaptive classifier is additionally added in a transfer learning mode to restrain the quality of a generated image, so that a remote sensing image sample with better quality is generated.

Description

Technical Field

[0001] The present invention belongs to the application of deep learning technology in the field of generation and conversion of high-resolution remote sensing images, and relates to a method for converting remote sensing image ground object samples based on a diffusion model, which converts the seasons of remote sensing image ground object samples through the diffusion model. Background Art

[0002] Currently, as the core driving force of the new round of scientific and technological and industrial revolutions, artificial intelligence technology, especially deep learning, has been deeply coupled with remote sensing applications, and breakthroughs have been made in remote sensing intelligent interpretation methods based on deep learning (Literature 2). As a data-driven technology, deep learning relies on large-scale high-quality labeled data sets. Many public sample data sets have emerged in the remote sensing community (Literature 3-7), and domestic and foreign research scholars have conducted very beneficial explorations on the openness and sharing of samples (Literature 1, 8, 9). However, the conversion of remote sensing intelligent interpretation ground object samples is still a challenging field. In the field of computer vision, the sample simulation generation and conversion method based on the generative adversarial network model provides a reference for the conversion of remote sensing intelligent interpretation samples. However, the existing methods do not fully consider the remote sensing characteristics and are difficult to meet the requirements of remote sensing intelligent interpretation tasks.

[0003] In terms of the synthesis of remote sensing image samples in different seasons, due to the inconsistencies in imaging seasons, resolutions, imaging methods, spectral characteristics, etc. between the remote sensing images to be interpreted and the existing sample data, the development of high-precision remote sensing image intelligent interpretation technology is severely restricted. With the emergence of generative models, many domestic and foreign scholars have begun to explore methods for image sample synthesis. These methods mainly use deep learning models in the field of computer vision for migration and expansion, such as classic networks such as generative adversarial network models (Literature 10, 11), autoencoders (Literature 12, 13), etc., and the diffusion model (Diffusion Model) that has developed rapidly in recent years (Literature 14). This model obtains the generated new image through the diffusion process (superimposing noise) and the reverse diffusion process (removing noise). In the DALL-E2 model proposed by Ramesh et al. (Literature 15), the CLIP (Literature 16) model is combined with the diffusion model, and contrast learning captures the robust representation of semantic and style images, generating images with diverse styles and very high definition and realism.

[0004] Although existing models have relatively mature research in the field of natural image generation, most of these models are designed for features such as space, geometry, and texture in natural images, and it is difficult to effectively model the ground object target characteristics in multi-source remote sensing image samples. Most of the existing methods for image synthesis between different seasons are aimed at the season conversion between natural images, and cannot fully consider the complex ground object target characteristics of remote sensing images, making it difficult to meet the requirements of the diversity of remote sensing image sample libraries.

[0005] References:

[0006] [1] Gong Jianya, Xu Yue, Hu Xiangyun, Jiang Liangcun, Zhang Mi, et al. Current Situation and Research of Intelligent Interpretation Sample Library for Remote Sensing Images [J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(8): 1013-1022. DOI: 10.11947 / j.AGCS.2021.20210085.

[0007] [2] Zhou Peicheng, Cheng Gong, Yao Xiwen, Han Junwei. Machine Learning Paradigms in the Interpretation of High-Resolution Remote Sensing Images. Journal of Remote Sensing. 2021, 25(1): 182-197.

[0008] [3] Xia GS, Hu J, Hu F, Shi B, Bai X, Zhong Y, Zhang L, Lu X. AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification. IEEE Transactions on Geoscience and Remote Sensing. 2017, 55(7): 3965-81.

[0009] [4] Xia GS, Bai X, Ding J, Zhu Z, Belongie S, Luo J, Datcu M, Pelillo M, Zhang L. DOTA: A Large-Scale Dataset for Object Detection in Aerial Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 3974-3983).

[0010] [5] Cheng G, Han J, Lu X. Remote Sensing Image Scene Classification: Benchmark and State of the Art. Proceedings of the IEEE. 2017, 105(10): 1865-83.

[0011] [6]Helber P,Bischke B,Dengel A,Borth D.Eurosat:A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.2019,12(7):2217-26.

[0012] [7]Li K,Wan G,Cheng G,Meng L,Han J.Object detection in optical remote sensing images:Asurvey and a new benchmark.ISPRS Journal of Photogrammetry and Remote Sensing.2020a,159:296-307.

[0013] [8]Alemohammad H.The case for open-access ML-ready geospatial training data.In 2021IEEE International Geoscience and Remote Sensing Symposium IGARSS2021(pp.1146-1148).IEEE.[9]McKinstry A,Boydell O,Le Q,Preet I,Hanafin J,Fernandez M,Warde A,Kannan V,Griffiths P.AI-Ready Training Datasets for Earth Observation:Enabling FAIR data principles for EO training data.InEGU General Assembly Conference Abstracts 2021(pp.EGU21-12384).

[0014]

[10] Creswell A, White T, Dumoulin V, Arulkumaran K, Sengupta B, Bharath AA. Generative adversarial networks: An overview. IEEE Signal Processing Magazine. 2018, 35(1): 53 - 65.

[0015]

[11] Goodfellow, I.J., Pouget - Abadie, J., Mirza, M., Xu, B., Warde - Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In NIPS’2014

[0016]

[12] Kingma DP, Welling M. An introduction to variational autoencoders. arXiv preprint arXiv:1906.02691. 2019.

[0017]

[13] Kingma DP, Welling M. Auto - encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013.

[0018]

[14] Sehwag, V., Hazirbas, C., Gordo, A., Ozgenel, F., & Ferrer, C.C. (2022). Generating High Fidelity Data from Low - density Regions using Diffusion Models. 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11482 - 11491.

[0019]

[15] Ramesh,A.,Dhariwal,P.,Nichol,A.,Chu,C.,&Chen,M.(2022).Hierarchical Text-Conditional Image Generation with CLIP Latents.ArXiv,abs / 2204.06125.

[0020]

[16] Radford,A.,Kim,J.W.,Hallacy,C.,Ramesh,A.,Goh,G.,Agarwal,S.,Sastry,G.,Askell,A.,Mishkin,P.,Clark,J.,Krueger,G.,&Sutskever,I.(2021).Learning Transferable Visual Models From Natural LanguageSupervision.International Conference on Machine Learning. Summary of the Invention

[0021] In view of the conversion requirements between remote sensing image samples in different seasons and the problems existing in the existing methods for converting ground objects in remote sensing images, the present invention uses a diffusion model to construct a remote sensing image seasonal conversion network, and adds a season-invariant feature network for the different characteristics of ground objects in different seasons, making full use of multi-source remote sensing image samples at different time series to generate remote sensing samples of the required season. The method of the present invention fully considers the characteristics of relevant ground objects in different seasons, can retain as much remote sensing ground object information as possible, and meets the needs of intelligent interpretation tasks.

[0022] The technical solution provided by the present invention is as follows:

[0023] In a first aspect, a method for converting ground object samples of remote sensing images based on a diffusion model is provided, including the following steps:

[0024] Obtain remote sensing data, perform preprocessing, and obtain preprocessed data;

[0025] Construct a neural network; the neural network uses a stable diffusion model as the basic network, and includes a picture perception compression module, a latent diffusion model module, and a season-invariant feature learning module; wherein, the picture perception compression module includes an encoder and a decoder Used to compress pictures while retaining important information; the latent diffusion model module includes a diffusion module and a temporal denoising autoencoder with a cross-attention mechanism layer for data denoising; the season-invariant feature learning module includes a feature extraction network, a domain encoder, a class classifier, and a domain adaptation classifier for extracting season-invariant features of the image as conditional guidance for image generation;

[0026] Use the preprocessed data to train the neural network in stages to obtain the final neural network;

[0027] Input the remote sensing image ground object samples and the seasonal description text information to be converted into the final neural network, and output the season conversion result.

[0028] In a possible implementation manner, the remote sensing data includes remote sensing images with season information and remote sensing images with semantic labels; the preprocessing includes cropping and the division of training set, validation set, and test set.

[0029] In a possible implementation manner, the picture perception compression module is a pre-trained variational encoder for images Using the encoder Encode the image x into the latent representation space, and the formula is:

[0030]

[0031] where represents a three-channel RGB image, H represents the image height, W represents the image width, and 3 represents the image channel dimension;

[0032] After completing the image conversion process, use the decoder to reconstruct the picture from the latent representation space, and the specific formula is:

[0033]

[0034] where represents the reconstructed picture, represents the decoder.

[0035] Furthermore, the latent diffusion model module is arranged between the encoder and decoder of the picture perception compression module, and includes a diffusion module and a temporal denoising autoencoder with a cross-attention mechanism layer; the diffusion module is used to superimpose Gaussian noise on the image; the temporal denoising autoencoder with a cross-attention mechanism layer includes a cross-attention downsampling module, a downsampling module, a cross-attention module, an upsampling module, and a cross-attention upsampling module that are connected in sequence to form a Unet; the cross-attention downsampling module and the cross-attention upsampling module are skip-connected; the upsampling module and the downsampling module are skip-connected;

[0036] The objective function L of the latent diffusion model moduleLDM is:

[0037]

[0038] Among them, the temporal denoising autoencoder with a cross-attention mechanism layer ∈ θ (z t , t) is a time-conditioned Unet network, ∈ θ is a temporal denoising autoencoder, t is the time step, and z t is the noisy image at time step t; denotes expectation, ∈ denotes the model Unet for predicting noise, denotes the Gaussian distribution, denotes the given Gaussian distribution of the noise.

[0039] Furthermore, the season-invariant feature learning module includes a feature extraction network, a domain encoder τ θ , a class classifier, and a domain adaptation classifier;

[0040] Among them, the feature extraction network is a Unet network;

[0041] The domain encoder τ θ is used to map the conditional control information y of different modalities into an intermediate representation τ θ (y), which is incorporated into the intermediate layer of the Unet network in the temporal denoising autoencoder with a cross-attention mechanism layer through the cross-attention mechanism layer, so as to realize guiding the image conversion through the semantic labels of remote sensing images and the text descriptions of target images;

[0042] The class classifier is composed of multiple connected convolutional layers, which is used to extract the semantic features of the image, so that the feature extraction network can learn season-invariant semantic features;

[0043] The domain adaptation classifier is a domain classification network composed of multiple linear layers, which is used to discriminate the season information to which the image belongs, so that the feature extraction network can judge the season to which the image belongs, and thus season-invariant feature information can be extracted.

[0044] In a possible implementation manner, the method for training the neural network in stages using the preprocessed data includes:

[0045] Train in the source domain and the target domain respectively: Encode the input image through the feature extraction network to extract image features; Classify the extracted image features through the class classifier and the domain classifier, where the class classifier determines the ground object category to which each pixel of the input image belongs, and the domain classifier determines whether the input image belongs to the source domain or the target domain; Calculate the losses of the two classifiers, and update the weights of the season-invariant feature learning module through backpropagation;

[0046] Training the latent diffusion model: Using the encoder Encode the image into the latent space, add noise to it, and complete the forward process of diffusion; Train the reverse process of the diffusion model. Encode the relevant text describing the target season to which the image needs to be converted using the pre-trained CLIP text encoder. Combine the season-invariant features learned by the season-invariant feature learning module to extract the season-invariant features of the image. Use the cross-attention mechanism to map and integrate the conditional control information into the middle layer of the Unet network of the temporal denoising autoencoder with a cross-attention mechanism layer; The temporal denoising autoencoder with a cross-attention mechanism layer predicts the noise in the latent space, and obtains the representation of the target season image in the latent space after denoising; Pass through the decoder Reconstruct the image from the latent representation space to obtain the image after season conversion;

[0047] Calculate the loss function between the image after season conversion and the real image of the target season, and update the weights of the neural network through backpropagation.

[0048] In a second aspect, there is provided a device for converting remote sensing image ground object samples based on a diffusion model, including:

[0049] An acquisition module for acquiring remote sensing data, performing preprocessing, and obtaining preprocessed data;

[0050] A construction module for constructing a neural network; The neural network takes a stable diffusion model as the basic network, and includes a picture perception compression module, a latent diffusion model module, and a season-invariant feature learning module; Among them, the picture perception compression module includes an encoder and a decoder for compressing the picture to retain important information; The latent diffusion model module includes a diffusion module and a temporal denoising autoencoder with a cross-attention mechanism layer for data denoising; The season-invariant feature learning module includes a feature extraction network, a domain encoder, a class classifier, and a domain adaptation classifier for extracting the season-invariant features of the image as conditional guidance for image generation;

[0051] A training module for training the neural network in stages using the preprocessed data to obtain the final neural network;

[0052] An output module for inputting the remote sensing image ground object samples and the text information of the season description to be converted into the final neural network and outputting the season conversion result.

[0053] In a third aspect, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the method for converting remote sensing image ground object samples based on a diffusion model as described in the first aspect.

[0054] In a fourth aspect, a non-transitory computer-readable storage medium stores a computer program, which when executed by a processor, implements the method for converting remote sensing image ground object samples based on a diffusion model as described in the first aspect.

[0055] In a fifth aspect, a computer program product includes a computer program, which when executed by a processor, implements the method for converting remote sensing image ground object samples based on a diffusion model as described in the first aspect.

[0056] The present invention has the following beneficial effects:

[0057] (1) The method of the present invention uses a stable diffusion model as a basic generation model and utilizes the powerful generation ability of the diffusion model to generate remote sensing images of different seasons.

[0058] (2) The method of the present invention constructs a season-invariant feature extraction network, fully considers the features of relevant ground objects in different seasons during the season conversion process, and ensures sufficient remote sensing ground object information.

[0059] (3) The method of the present invention additionally adds a domain adaptation classifier based on the stable diffusion model using transfer learning to constrain the quality of the generated images, thereby generating remote sensing image samples of better quality;

[0060] (4) The method of the present invention expands the season sample library through season conversion to meet the needs of intelligent interpretation tasks. Description of the Drawings

[0061] Figure 1 is a schematic flowchart of a method for converting remote sensing image ground object samples based on a diffusion model according to an embodiment of the present invention;

[0062] Figure 2 is a schematic diagram of a neural network model according to an embodiment of the present invention;

[0063] Figure 3 is a schematic diagram of a temporal denoising autoencoder with a cross-attention mechanism layer;

[0064] Figure 4 is a schematic diagram of a feature extraction network;

[0065] Figure 5 is an effect diagram of season conversion, where Figure 5 in (a) is the image after conversion in winter, Figure 5 in (b) is the image after conversion in summer, Figure 5 in (c) is the input condition image before conversion;

[0066] Figure 6 is a schematic diagram of a device for converting remote sensing image ground object samples based on a diffusion model;

[0067] Figure 7 Schematic diagram of the electronic device of the present invention. Detailed implementation manners

[0068] To facilitate the understanding and implementation of the present invention by those of ordinary skill in the art, the present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described herein are only for the purpose of illustrating and explaining the present invention, and are not intended to limit the present invention.

[0069] In view of the conversion requirements between remote sensing image samples in different seasons and the problems existing in the existing methods for converting ground objects in remote sensing images, the present invention provides a method for converting ground object samples in remote sensing images based on a diffusion model. The method uses a diffusion model (Diffusion Model) to construct a remote sensing image season conversion network, and adds a season-invariant feature learning module for different features of ground objects in different seasons, making full use of multi-source remote sensing image samples at different time series to generate remote sensing samples of the required season. This method fully considers the features of relevant ground objects in different seasons, can retain remote sensing ground object information as much as possible, and meets the needs of intelligent interpretation tasks.

[0070] As Figure 1 shown, an embodiment of the present invention provides a method for converting ground object samples in remote sensing images based on a diffusion model, and the steps are as follows:

[0071] Step 1: Select remote sensing data, perform preprocessing, and obtain preprocessed data.

[0072] In a possible implementation manner, the remote sensing data includes remote sensing images with season information and remote sensing images with semantic labels; the preprocessing includes cropping and the division of training sets, validation sets, and test sets.

[0073] Exemplarily, the main datasets used in this embodiment are the SEN1-2 dataset with season information, and the Postdam and Vaihingen datasets with semantic labels. And the datasets are preprocessed and divided into 80% as the training set, 10% as the validation set, and 10% as the test set for experiments in advance.

[0074] (1) For SEN1-2, this dataset is produced using GEE (Google Earth Engine). Sampling points are selected all over the world using GEE, and sampling is carried out in the four seasons of spring (March 1, 2017 - May 30, 2017), summer (June 1, 2017 - August 31, 2017), autumn (September 1, 2017 - November 30, 2017), and winter (December 1, 2016 - February 28, 2017). SAR images of the C-band of Sentinel-1 (SEN1) and RGB image data of bands 4, 3, and 2 of Sentinel-2 (SEN2) are obtained. The image size is 256*256, and there are a total of 282,384 RGB and SAR image pairs. In this embodiment, only the partial seasonal images of RGB are selected as the dataset for constructing the reference sample library.

[0075] (2) The Vaihingen dataset contains aerial images from Vaihingen, Germany. It is a relatively small village with many independent buildings and small multi-story buildings. The images of this dataset cover an area of approximately 1 square kilometer, with an image resolution of 9 cm and an average size of 2494×2064 pixels, including 5 foreground object classes (impervious surfaces, buildings, low vegetation, trees, cars) and 1 background class. In this embodiment, 33 images are cut into 3300 images with a size of 512×512 pixels for experiments.

[0076] (3) The Postdam dataset contains aerial images from the city of Potsdam, Germany. It is a typical historical city with large building complexes, narrow streets, and a dense settlement structure. The images of this two-dimensional semantic annotation dataset cover an area of approximately 2.3 square kilometers, with an image resolution of 5 cm, and consist of 38 images of 6000×6000 pixels. The dataset provides NIR-R-G-B channels, and the corresponding categories are the same as those of the Vaihingen dataset. In this embodiment, only the RGB three-channel images in the Postdam dataset are used, and they are cut into images with a size of 1000×1000 pixels to obtain 3660 images as the images to be converted.

[0077] Step 2: Construct a neural network; the neural network uses the Stable Diffusion Model as the basic network, which includes three network modules: a picture perception compression module, a latent diffusion model module, and a season-invariant feature learning module.

[0078] In a possible implementation, the structure of the neural network is as Figure 2As shown. Among them, the picture perception compression module is a pre-trained variational encoder, including an encoder and a decoder. This module processes the original image, can ignore the high-frequency information in the picture, and only retains some important and basic features. In this way, the computational complexity in the training and sampling stages can be greatly reduced.

[0079] It should be noted that the picture perception compression module can learn a latent representation space that is perceptually equivalent to the image space. Its advantage is that only one general autoencoder model needs to be trained and can be used for the training of different diffusion models and in different tasks.

[0080] Specifically, for a given image using the encoder encode the image into the latent representation space, and the specific formula is:

[0081]

[0082] where represents a three-channel RGB image, H represents the image height, W represents the image width, and 3 represents the image channel dimension;

[0083] After completing the image conversion process, use the decoder to reconstruct the picture from the latent representation space, and the specific formula is:

[0084]

[0085] where represents the reconstructed picture, represents the decoder.

[0086] In a possible implementation manner, the latent diffusion model module includes a diffusion module and a temporal denoising autoencoder with a cross-attention mechanism layer.

[0087] Furthermore, the diffusion module is used to add noise to the picture, including using a Markov chain to gradually superimpose Gaussian noise on the original data to obtain a noise image that satisfies a Gaussian distribution. The specific formula is:

[0088]

[0089] where x t is the image at time t, q(x t |x t-1 ) is the probability distribution of the noise image at time x t , which satisfies a Gaussian distribution, is the Gaussian distribution, I is the diagonal covariance matrix, and β t is the variance used for each time step t.

[0090] It is understandable that a temporal denoising autoencoder ∈ θ (x t , t); t = 1…T, where x t is the noise of the input image x at time t, and T represents the number of time steps. ∈ θ is a temporal denoising autoencoder that learns the data distribution by gradually denoising normal distribution variables and predicts the corresponding denoised result according to the input x t .

[0091] The objective function of the latent diffusion model module is as follows:

[0092]

[0093] Among them, the temporal denoising autoencoder with a cross-attention mechanism layer ∈ θ (z t , t) is a time-conditioned Unet network, ∈ θ is a temporal denoising autoencoder, t is the time step, and z t is the noisy image at time step t; represents the expectation, ∈ represents the model Unet for predicting the noise, represents the Gaussian distribution, represents the given Gaussian distribution of the noise.

[0094] See Figure 3 , for the structure of the temporal denoising autoencoder with a cross-attention mechanism layer. The control information is incorporated into each downsampling and upsampling layer of the Unet through a cross-attention layer mapping, including the cross-attention downsampling module, downsampling module, cross-attention module, upsampling module, and cross-attention upsampling module that are connected in sequence to form the Unet; the cross-attention downsampling module and the cross-attention upsampling module are skip-connected; the upsampling module and the downsampling module are skip-connected.

[0095] Exemplarily, the cross-attention downsampling module is a cross-attention mechanism layer; the downsampling module is a convolutional layer; the cross-attention module is a cross-attention mechanism layer; the upsampling module is a convolutional layer; the cross-attention upsampling module is a cross-attention mechanism layer.

[0096] The implementation of the cross-attention mechanism layer is as follows:

[0097]

[0098] Among them, Attention represents the attention vector, Q, K, V are the information to be queried, the vector to be queried, and the value obtained by the query; the superscript T represents the transposed matrix; d represents the dimension of the hidden layer; Is an intermediate representation of UNet, represents the feature encoder, i represents the feature vector, represents the real number field, N represents the feature dimension, represents the dimension of the hidden layer, i represents the feature vector, ∈ represents the model Unet for predicting noise; is a learnable projection matrix, d represents the dimension of the hidden layer.

[0099] It should be noted that the structure of the ∈, that is, the model Unet for predicting noise, is a cross-attention downsampling module, a downsampling module, a cross-attention module, an upsampling module, and a cross-attention upsampling module; the cross-attention downsampling module and the cross-attention upsampling module are skip-connected; the upsampling module and the downsampling module are skip-connected. That is, ∈ θ (z t ,t) has the same structure as ∈, ∈ represents the true value, ∈ θ (z t ,t) represents the value predicted by the model, and the model parameters are optimized and updated after calculating the loss between the two.

[0100] In the latent diffusion model module, given the latent representation space z obtained by encoding the image 0 After that, z is obtained through the diffusion module t , z t Passes through the temporal denoising autoencoder ∈ with a cross-attention mechanism layer θ to obtain z t-1 , z t-1 Then passes through the same temporal denoising autoencoder ∈ θ , and repeats t times, finally obtaining z 0 . z 0 represents the latent space feature at time 0.

[0101] In this embodiment, the latent diffusion model incorporates a pre-trained picture perception compression model, which includes an encoder and a decoder During training, the encoder can be used to encode the input image x to obtain z, and then z is obtained through the diffusion process t , so that the model can learn in the latent representation space, enabling the model to focus on more important semantic information in the data, and training in the low-dimensional space also makes the calculation more efficient. Since the forward process of diffusion is fixed, during training, z can be efficiently obtained from t , and the learned samples only need to be decoded once by the decoder to complete the generation of the image in the image space.

[0102] In a possible implementation manner, the season-invariant feature learning module includes a feature extraction network, a domain encoder, a class classifier, and a domain adaptation classifier.

[0103] Furthermore, the feature extraction network is a Unet network, as Figure 4 shown, including an upsampling layer, a downsampling layer, and a skip connection layer; the upsampling layer and the downsampling layer are skip-connected. Both the upsampling layer and the downsampling layer are composed of convolutional layers.

[0104] Furthermore, the domain encoder is 5 convolutional layers, each convolutional layer having a 3×3 convolutional kernel, for encoding seasonal image features.

[0105] Furthermore, the class classifier is composed of 4 connected convolutional layers, each convolutional layer having a 3×3 convolutional kernel, for extracting semantic features of the image, enabling the feature extraction network to learn season-invariant semantic features.

[0106] Furthermore, the domain adaptation classifier is a domain classification network composed of multiple linear layers (fully connected layers), and the parameter of each linear layer is the number of feature channels. The domain adaptation classifier is used to discriminate the seasonal information to which the image belongs, enabling the feature extraction network to determine the season to which the image belongs, thereby enabling the extraction of season-invariant feature information.

[0107] Data flow: As Figure 2 shown, on the one hand, the original image extracts the features of the image through the feature extraction network to obtain a feature representation, and then the domain encoder integrates the feature representation into the temporal denoising autoencoder with a cross-attention mechanism layer ∈ θ ; on the other hand, the text is integrated into the temporal denoising autoencoder with a cross-attention mechanism layer ∈ θ through the domain encoder τ θ . The domain encoder encodes the seasonal features of the image to determine the season of the input image; the class classifier determines the semantic category of the image and learns season-invariant features.

[0108] It can be understood that in view of the complexity of multiple ground objects in remote sensing images, a conditional mechanism is introduced, that is, by introducing a conditional temporal denoising autoencoder ∈ θ (z t ,t,y) to expand the diffusion model, where y is the introduced conditional control information, so that the process of remote sensing image synthesis can be guided and controlled through the conditional control information y.

[0109] Specifically, the temporal noise predictor ∈ θ (z t ,t,y) is implemented through the cross-attention mechanism on the Unet network of the temporal denoising autoencoder with a cross-attention mechanism layer, and through the domain encoder τ θMap the conditional control information y of different modalities to an intermediate representation (M is the feature dimension of different modalities, d represents the dimension of the hidden layer, and τ represents the representation of the domain encoder). Finally, the model can integrate the conditional control information into the intermediate layer of the temporal denoising autoencoder UNet with a cross-attention mechanism layer through a cross-attention layer mapping, and finally guide the conversion of the image season or modality through the semantic labels of the remote sensing image and the text description of the target image. Among them, the conditional control information y includes season-invariant features and text descriptions.

[0110] Based on the stable diffusion model, the present invention additionally adds a domain adaptation classifier by using transfer learning to constrain the quality of the generated images, so as to generate remote sensing image samples with better quality.

[0111] Step 3: Use the preprocessed data to train the neural network in stages to obtain the final neural network.

[0112] The training process is divided into two stages: (1) Train the season-invariant feature learning module; (2) Train the latent diffusion model module to perform image season conversion, and add season-invariant features and text descriptions as conditions during denoising in the training to guide the image to perform season conversion.

[0113] In a possible implementation manner, step 3 includes the following sub-steps:

[0114] S3.1, train in the source domain and the target domain respectively:

[0115] (1) Encode the input image through the feature extraction network to extract image features;

[0116] (2) Classify the extracted image features through the class classifier and the domain adaptation classifier; among them, the class classifier determines the ground object class to which each pixel of the input image belongs, and the domain classifier determines whether the input image belongs to the source domain or the target domain;

[0117] (3) Calculate the losses of the two classifiers and update the weights of the season-invariant feature module through backpropagation.

[0118] S3.2, train the latent diffusion model module:

[0119] (1) Use the encoder Encode the image into the latent space and add noise to it to complete the forward process of diffusion;

[0120] (2) In the reverse process of training the diffusion model, the relevant text describing the target season to which the image needs to be converted is encoded using a pre-trained CLIP text encoder; the season-invariant features of the image are extracted using a season-invariant feature learning module, and the conditional control information is mapped and incorporated into the middle layer of the UNet of the temporal denoising autoencoder with a cross-attention mechanism layer through the cross-attention mechanism layer;

[0121] (3) The temporal denoising autoencoder with a cross-attention mechanism layer predicts the noise in the latent space, and after denoising, the representation of the target season image in the latent space is obtained;

[0122] (4) Through the decoder The picture is reconstructed from the latent representation space to obtain the image after season conversion;

[0123] (5) Calculate the loss function between the image after season conversion and the real image of the target season, and update the weights of the neural network through backpropagation.

[0124] It should be noted that in this embodiment, considering the number of network parameters and the amount of training data, a fine-tuning method is adopted to fine-tune the pre-trained stable diffusion model with remote sensing data.

[0125] Step 4: Input the remote sensing image ground object samples and the text information of the season to be converted into the final neural network, and output the season conversion result.

[0126] In a possible implementation manner, after the neural network is trained, the network weights are saved to a ckpt file. With the help of the ckpt file, the season conversion can be directly performed through the input remote sensing image and the text information of the season to be converted, and the final result is output. Figure 5 It is the effect diagram of season conversion, where Figure 5 (a) in it is the image after conversion to winter, Figure 5 (b) in it is the image after conversion to summer, Figure 5 (c) in it is the input condition image before conversion;

[0127] Next, a device for converting remote sensing image ground object samples based on a diffusion model provided by the present invention will be described. The device for converting remote sensing image ground object samples based on a diffusion model described below can be mutually referred to with the method for converting remote sensing image ground object samples based on a diffusion model described above.

[0128] Figure 6 It is a schematic structural diagram of a device for converting remote sensing image ground object samples based on a diffusion model provided by an embodiment of the present invention. As Figure 6 shown, it includes: an acquisition module 61, a construction module 62, a training module 63, and an output module 64, where:

[0129] An acquisition module 61 for acquiring remote sensing data and performing preprocessing;

[0130] A construction module 62 for constructing a neural network; the neural network is based on a stable diffusion model as the basic network and includes an image perception compression module, a latent diffusion model module, and a season-invariant feature learning module; among them, the image perception compression module includes an encoder and a decoder for compressing the picture to retain important information; the latent diffusion model module includes a diffusion module and a temporal denoising autoencoder with a cross-attention mechanism layer for data denoising; the season-invariant feature learning module includes a feature extraction network, a domain encoder, a class classifier, and a domain adaptation classifier for extracting the season-invariant features of the image as conditional guidance for image generation;

[0131] A training module 63 for training the neural network in stages to obtain the final neural network;

[0132] An output module 64 for inputting the remote sensing image ground object samples and the seasonal description text information to be converted into the final neural network and outputting the seasonal conversion result.

[0133] Figure 7 An example of the physical structure diagram of an electronic device is shown in Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communication interface 720, and the memory 730 complete mutual communication through the communication bus 740. The processor 710 can call the logical instructions in the memory 730 to execute a method for converting remote sensing image ground object samples based on a diffusion model.

[0134] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0135] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a method for converting remote sensing image ground object samples based on a diffusion model provided by each of the above methods.

[0136] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute a method for converting remote sensing image ground object samples based on a diffusion model provided by each of the above methods.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0138] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image ground object sample conversion method based on a diffusion model, characterized in that: The following steps are involved: Acquire remote sensing data, perform preprocessing, and obtain preprocessed data; Construct a neural network; the neural network uses a stable diffusion model as a basic network, including an image perception compression module, a potential diffusion model module and a seasonal invariant feature learning module; wherein the image perception compression module includes an encoder ε and a decoder It is used to compress images and retain important information. The latent diffusion model module includes a diffusion module and a temporal denoising autoencoder with a cross-attention mechanism layer, which is used for data denoising. The seasonal invariant feature learning module includes a feature extraction network, a domain encoder, a category classifier, and a domain adaptation classifier, which is used to extract seasonal invariant features of images as conditions to guide image generation. The neural network is trained in stages using the preprocessed data to obtain the final neural network; The remote sensing image object samples and the seasonal description text information to be converted are input into the final neural network, and the seasonal conversion results are output.

2. The remote sensing image ground object sample conversion method based on diffusion model according to claim 1 is characterized in that: The remote sensing data includes remote sensing images with seasonal information and remote sensing images with semantic labels; the preprocessing includes cropping and division into a training set, a validation set and a test set.

3. The remote sensing image ground object sample conversion method based on diffusion model according to claim 1 is characterized in that: The image-aware compression module is a pre-trained variational encoder that The encoder ε is used to encode the image x into the latent representation space, the formula is: z=ε(x) in, Represents a three-channel RGB image, H represents the image height, W represents the image width, and 3 represents the image channel dimension; After completing the image conversion process, the decoder is used to reconstruct the image from the latent representation space. The specific formula is: in, Represents the reconstructed image, Represents a decoder.

4. The remote sensing image ground object sample conversion method based on diffusion model according to claim 1 is characterized in that: The latent diffusion model module is arranged between the encoder and the decoder of the image perception compression module, and includes a diffusion module and a temporal denoising autoencoder with a cross-attention mechanism layer; the diffusion module is used to superimpose Gaussian noise on the image; the temporal denoising autoencoder with a cross-attention mechanism layer includes a cross-attention downsampling module, a downsampling module, a cross-attention module, an upsampling module and a cross-attention upsampling module which are sequentially connected to form a Unet; the cross-attention downsampling module and the cross-attention upsampling module are jump-connected; the upsampling module and the downsampling module are jump-connected; The objective function L of the latent diffusion model module LDM for: Among them, the temporal denoising autoencoder ∈ θ (z t ,t) is the time condition Unet network,∈ θ is a time series denoising autoencoder, t is the time step, z t is the noise image at time step t; represents the expectation, ∈ represents the model Unet for predicting noise, represents a Gaussian distribution, Represents a given Gaussian distribution noise.

5. The remote sensing image ground object sample conversion method based on diffusion model according to claim 4 is characterized in that: The seasonal invariant feature learning module includes a feature extraction network, a domain encoder τ θ , category classifiers and domain adaptation classifiers; Wherein, the feature extraction network is a Unet network; The domain encoder τ θ Used to map the conditional control information y of different modalities into the intermediate representation τ θ (y) The cross-attention layer is integrated into the middle layer of the Unet network in the temporal denoising autoencoder with cross-attention layer to guide the image conversion through the semantic labels of the remote sensing image and the text description of the target image; The category classifier is composed of a plurality of convolutional layers connected to extract the semantic features of the image, so that the feature extraction network can learn the semantic features that are invariant to the season; The domain adaptation classifier is a domain classification network composed of multiple linear layers, which is used to determine the season information of the image, so that the feature extraction network can determine the season of the image, thereby extracting the season-invariant feature information.

6. The remote sensing image ground object sample conversion method based on diffusion model according to claim 1 is characterized in that: The method for training a neural network in stages using preprocessed data comprises: Training is performed in the source domain and the target domain respectively: the input image is encoded through the feature extraction network to extract image features; the extracted image features are classified through the category classifier and the domain classifier, where the category classifier determines the ground object category to which each pixel of the input image belongs, and the domain classifier determines whether the input image belongs to the source domain or the target domain; the losses of the two classifiers are calculated, and the weights of the seasonal invariant feature learning module are updated through back propagation; Training the latent diffusion model: Use the encoder ε to encode the image into the latent space and add noise to it to complete the forward process of diffusion; train the reverse process of the diffusion model, encode the relevant text describing the target season to which the image needs to be converted using the pre-trained CLIP text encoder, and extract the seasonal invariant features of the image by combining the seasonal invariant feature learning module, and use the cross-attention mechanism to map the conditional control information and integrate it into the middle layer of the Unet network of the temporal denoising autoencoder with a cross-attention mechanism layer; the temporal denoising autoencoder with a cross-attention mechanism layer predicts the noise in the latent space, and obtains the representation of the target season image in the latent space after denoising; after the decoder Reconstruct the image from the latent representation space to obtain the seasonally converted image; The loss function of the seasonally converted image and the real image of the target season is calculated, and the weights of the neural network are updated through back propagation.

7. A remote sensing image ground object sample conversion device based on a diffusion model, characterized in that: include: An acquisition module is used to acquire remote sensing data, perform preprocessing, and obtain preprocessed data; Building blocks for constructing neural networks; The neural network uses a stable diffusion model as a basic network, including a picture perception compression module, a potential diffusion model module and a seasonal invariant feature learning module; wherein the picture perception compression module includes an encoder ε and a decoder It is used to compress images and retain important information. The latent diffusion model module includes a diffusion module and a temporal denoising autoencoder with a cross-attention mechanism layer, which is used for data denoising. The seasonal invariant feature learning module includes a feature extraction network, a domain encoder, a category classifier, and a domain adaptation classifier, which is used to extract seasonal invariant features of images as conditions to guide image generation. The training module is used to train the neural network in stages using the preprocessed data to obtain the final neural network; The output module is used to input the remote sensing image object samples and the seasonal description text information that needs to be converted into the final neural network and output the seasonal conversion results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the remote sensing image ground object sample conversion method based on the diffusion model as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the remote sensing image ground object sample conversion method based on the diffusion model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the remote sensing image ground object sample conversion method based on the diffusion model as described in any one of claims 1 to 6 is implemented.

Images