Remote sensing image controllable editing method and system based on image denoising diffusion model

Abstract

The invention discloses a remote sensing image controllable editing method and system based on an image denoising diffusion model. The system comprises remote sensing image acquisition equipment and a data processor, the data processor comprises a remote sensing image generation module, a cue word integration module, a generated image feature extraction module, an editing instruction text feature extraction module and an image-text similarity calculation module. The method comprises the following steps: acquiring a single remote sensing scene image of a target area; generating an optical remote sensing image by adopting an image generation method based on a de-noising diffusion model; carrying out diversified expansion on the group of text editing instructions; image feature coding and text feature coding are carried out, similarity calculation is carried out, and fine adjustment for a text editing instruction in the remote sensing image generation process is achieved by optimizing a loss function. According to the method, the model can be trained and generated according to the single remote sensing image, the use of a large number of paired images is avoided, and a stable and controllable remote sensing image editing effect with text guidance can be realized only by using the single image.

Description

Technical Field

[0001] The present invention belongs to the field of intersectional technology of artificial intelligence generation technology and remote sensing, and relates to a controllable editing method and system of remote sensing images for optical remote sensing sensors, which are used for producing corresponding images according to text guidance. Background Art

[0002] Remote sensing image generation technology has important application value in the fields of geographic information systems, environmental monitoring, urban planning, etc. By generating high-quality remote sensing images, the insufficiency of field data collection can be made up. Key technical points include image synthesis, image enhancement, and image super-resolution reconstruction. Image synthesis can simulate the surface landscape under different conditions and provide diverse ground object information; image enhancement aims to improve image quality and enhance the visibility and recognition of specific targets; image super-resolution reconstruction improves the spatial resolution of the image through algorithms to obtain finer ground object details. The development of remote sensing image generation technology has provided rich spatial data resources for various applications and promoted the advancement and application of geographic information technology.

[0003] At present, the mainstream remote sensing image generation technology includes two categories: traditional methods and deep learning methods. Traditional methods usually rely on mathematical models and image processing techniques, such as interpolation methods, multispectral image synthesis and restoration algorithms. These methods perform well in specific scenarios and data conditions, but have certain limitations when dealing with complex geographical environments and multi-source data fusion. Deep learning methods have been widely used in recent years, especially models such as generative adversarial networks and variational autoencoders. These models can learn and simulate complex land features and spatial distributions, generate realistic remote sensing images with high semantic information, and significantly improve the quality and diversity of generated images.

[0004] Despite significant progress in remote sensing image generation technology, it still faces some challenges. First, the lack of training data in specific scenarios limits the model's ability to adapt to specific environments, especially in extreme weather or complex terrain. The lack of diverse training data may lead to insufficient generalization of generated images, affecting their performance in practical applications. Second, existing technologies have limitations in achieving user-controllable image editing. Finally, text-guided image editing models often cause semantic confusion due to the inherent ambiguity of text, resulting in illogical generation results. Although deep learning models can generate high-quality remote sensing images, current methods often lack sufficient flexibility and sophistication when users need to precisely control and adjust specific objects or scenes. Summary of the invention

[0005] In view of the technical problems existing in the controllable editing capability of image content in the existing remote sensing image generation technology, the purpose of the present invention is to provide a remote sensing image controllable editing method and system based on an image denoising diffusion model. The present invention adopts a denoising diffusion model based on a single image and a text-guided image controllable editing method, trains the denoising diffusion model through a single remote sensing image, and uses a text image pre-training model fine-tuned on the remote sensing image for text guidance, so as to achieve stable and controllable text-guided remote sensing image editing.

[0006] The present invention discovers the similarity association between text encoding and image encoding for remote sensing images through research, combines the flexibility and powerful generation ability of the denoising diffusion model with the multi-scale architecture, realizes the single sample training paradigm, and uses the prompt word integration technology to deal with the semantic ambiguity of complex text, thereby providing a method for stable and controllable remote sensing image editing guided by text. Stable and controllable remote sensing image editing can be achieved by only training on a single image, which gives new practical significance to remote sensing image generation in practical applications.

[0007] The technical solution of the present invention is:

[0008] A controllable editing method and system for remote sensing images based on an image denoising diffusion model, the steps of which include:

[0009] Detect the target area and obtain a single remote sensing scene image of the target area;

[0010] Using the acquired single remote sensing scene image of the target area, an optical remote sensing image is generated by using an image generation method based on a denoising diffusion model;

[0011] A text generation method based on a large language model is used to diversify and expand a set of text editing instructions to generate multiple sets of optimized text editing instructions;

[0012] An image encoder based on remote sensing image pre-training is used to encode image features of the generated results of optical remote sensing images;

[0013] A text encoder based on remote sensing image pre-training is used to encode text features of multiple sets of text editing instructions, and the encoded results are averaged.

[0014] The similarity between the image feature encoding of the optical remote sensing image generation result and the text feature encoding of the text editing instruction is calculated. According to the similarity calculation result, the text editing instruction is fine-tuned in the remote sensing image generation process by optimizing the loss function to obtain the edited remote sensing image generation result.

[0015] Furthermore, the target area is detected using a remote sensing image acquisition device, a single remote sensing scene image of the target area is obtained and sent to a data processor; the data processor includes a remote sensing image generation module, a prompt word integration module, a generated image feature extraction module, an editing instruction text feature extraction module, and a picture-text similarity calculation module.

[0016] Furthermore, the remote sensing image generation module in the data processor defines a Markov chain with a diffusion step, introduces random noise into the input data x in a progressive manner, and then learns the inverse process of the diffusion process to reconstruct the image from the noise to achieve image generation. The processing process of the remote sensing image generation module includes the following steps:

[0017] First, construct an image pyramid: {x N-1 ,x N-2 ,…,x 0}, where x N-1 represents the input image itself, x 0 represents the input image after N-1 downsampling. For a scale s in (0,1,…,N-1), the forward process is represented as: where t∈(0,T) represents the number of steps in the forward process of adding noise to the image, is the variance at each step, represents the input image after N-1 downsampling and t-step forward process noise added, represents the standard normal distribution, I represents the variance, and T represents the total number of steps in the forward process;

[0018] Then, the original data x s Noise image at any step It is expressed as: ∈ represents the added noise, and are the signal rate and noise rate constructed by the reparameterization technique;

[0019] Secondly, the backward diffusion process of the model is carried out to gradually predict the noise ∈ added in the forward stage: where p θ represents the predicted probability distribution, represents the mean value of the denoising operation at time t,

[0020] Finally, in the optimization phase of model training, the following loss function is used to achieve model optimization: Where ∈θ is the noise prediction result of the diffusion model.

[0021] Furthermore, the prompt word integration module uses a large language model, inputs an image editing target in the form of a text, outputs multiple groups of text prompts with the same semantics but different forms, and avoids generating irrelevant content, forming diversified and high-quality prompt information, and outputs multiple groups of text editing instructions.

[0022] Furthermore, the generated image feature extraction module and the edit instruction text feature extraction module use the text-image pre-trained CLIP model based on remote sensing image self-supervision to perform image feature encoding and text feature encoding on the generated results of the optical remote sensing image and the multiple groups of text edit instructions respectively. This process can be expressed as and f T (text), where f I and f T They represent the image feature encoder and text feature encoder pre-trained using multimodal contrastive learning method on remote sensing images, respectively. and text represent the currently generated image and the text editing instructions expanded by the large language model, respectively. This also obtains the feature encoding of the edited image in two different modalities: image and text.

[0023] Furthermore, the image-text similarity calculation module uses cosine distance to measure and f T The difference between the two feature encodings (text) is used as the optimization target in the fine-tuning process to update the generative model parameters again and achieve text-guided controllable editing. The optimization target is defined as: By optimizing the loss function L, fine-tuning of the text editing instructions in the remote sensing image generation process is achieved, and the edited remote sensing image generation result is obtained.

[0024] Furthermore, the controllable editing method for remote sensing images is a controllable editing method for remote sensing images based on a single image denoising diffusion model.

[0025] Furthermore, the remote sensing image acquisition device is a network camera.

[0026] Furthermore, the data processor is a computer or an embedded motherboard.

[0027] Furthermore, the remote sensing image generation module is a denoising diffusion model.

[0028] Furthermore, the prompt word integration module is a large language model.

[0029] Furthermore, the generated image feature extraction module and the editing instruction text feature extraction module are self-supervised remote sensing image text-image pre-training models.

[0030] A controllable editing method and system for remote sensing images based on an image denoising diffusion model, comprising a remote sensing image acquisition device and a data processor; the data processor comprises a remote sensing image generation module, a prompt word integration module, a generated image feature extraction module, an editing instruction text feature extraction module and a picture-text similarity calculation module; the remote sensing image acquisition device is connected to the remote sensing image generation module in the data processor, the remote sensing image generation module in the data processor is connected to the prompt word integration module in the data processor, the prompt word integration module in the data processor is connected to the generated image feature extraction module and the editing instruction text feature extraction module in the data processor, and the generated image feature extraction module and the editing instruction text feature extraction module in the data processor are connected to the picture-text similarity calculation module in the data processor;

[0031] The remote sensing image acquisition device is used to detect the target area, obtain a single remote sensing scene image of the target area and send it to the data processor;

[0032] The remote sensing image generation module generates an optical remote sensing image by using an image generation method based on a denoising diffusion model;

[0033] The prompt word integration module adopts a text generation method based on a large language model to diversify and expand a set of text editing instructions to generate multiple sets of optimized text editing instructions;

[0034] The generated image feature extraction module uses a CLIP image encoder pre-trained based on remote sensing images to perform image feature encoding on the generated result of the optical remote sensing image;

[0035] The editing instruction text feature extraction module uses a CLIP text encoder pre-trained based on remote sensing images to encode text features of multiple groups of text editing instructions, and performs an averaging operation on the encoded results;

[0036] The image-text similarity calculation module adopts the cosine similarity calculation method to calculate the similarity between the image feature coding of the optical remote sensing image generation result and the text feature coding of the text editing instruction. According to the similarity calculation result, the text editing instruction is fine-tuned in the remote sensing image generation process by optimizing the loss function to obtain the edited remote sensing image generation result.

[0037] The specific implementation of each module refers to the above description of the method of the present invention.

[0038] The beneficial effects of the above technical solution of the present invention are as follows:

[0039] In the above scheme, the remote sensing image acquisition equipment can pre-process the data to produce optical remote sensing images. The remote sensing image generation module adopts the image generation method of the single image denoising diffusion model, which can train the generation model based on a single remote sensing image, avoiding the problem of difficulty in training the generation model due to insufficient training data. The prompt word integration module uses a large language model to enrich the original text editing target as a guide for the image editing model. The two aspects work together to avoid the use of a large number of paired images. Only a single image can be used to achieve a stable and controllable remote sensing image editing effect with text guidance. The generated edited image can be used to support disaster assessment tasks and determine whether a building is destroyed by a tsunami. It has high robustness and reliability in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a block diagram of a controllable editing method for remote sensing images based on a single image denoising diffusion model. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below through examples and drawings.

[0042] See also Figure 1 The embodiment of the present invention provides a controllable editing method for remote sensing images based on an image denoising diffusion model. The controllable editing method for remote sensing images based on a single image denoising diffusion model includes: a remote sensing image acquisition device 1 and a data processor 2; the data processor 2 includes: a remote sensing image generation module 3, a prompt word integration module 4, a generated image feature extraction module 5, an editing instruction text feature extraction module 6 and a picture-text similarity calculation module 6, the picture-text similarity calculation module 7 is connected to the generated image feature extraction module 5 and the editing instruction text feature extraction module 6, the generated image feature extraction module 5 and the editing instruction text feature extraction module 6 are connected to the prompt word integration module 4, and the prompt word integration module 4 is connected to the remote sensing image generation module 3; the remote sensing image acquisition device 1 is connected to the remote sensing image generation module 3 in the data processor 2.

[0043] In the above embodiment, the remote sensing image acquisition device 1 may be a camera or other optical remote sensing image acquisition devices.

[0044] In the above embodiment, the data processor 2 may be a computer or an embedded motherboard.

[0045] In the above embodiment, the remote sensing image generation module 3 may use a denoising diffusion model to generate a single image for the optical remote sensing image.

[0046] In the above embodiment, the prompt word integration module 4 can use a large language model to diversify and expand the text editing instructions.

[0047] In the above embodiment, the generated image feature extraction module 5 and the editing instruction text feature extraction module 6 can adopt a self-supervised remote sensing image text-image pre-training model to respectively perform image feature encoding on the generation results of the optical remote sensing image and perform text feature encoding on multiple groups of text editing instructions, and perform averaging operation on the encoded results.

[0048] In the above embodiment, the image-text similarity calculation module 7 may use a cosine similarity calculation method to perform similarity calculation on the image feature code of the optical remote sensing image generation result and the text feature code of the text editing instruction.

[0049] The specific working process of the embodiment of the present invention is as follows:

[0050] First, the remote sensing image acquisition device detects the target area, obtains a single remote sensing scene image of the target area, and inputs it into the data processor 2.

[0051] Secondly, the remote sensing image generation module 3 in the data processor 2 constructs a multi-scale pyramid by downsampling the input single image during the training process, and trains the denoising model from the smallest scale. After completing the denoising of a specific scale, it transitions to the denoising of the subsequent scale by upsampling. This module defines a Markov chain with a diffusion step, which introduces random noise into the input data x in a progressive manner, and then learns the inverse process of the diffusion process to reconstruct the image from the noise to achieve generation. First, construct an image pyramid: {x N-1 ,x N-2 ,…,x 0}, for a scale s in (0,1,…,N-1), the forward process is represented as: where t∈(0,T) represents the number of steps in the forward process of adding noise to the image, is the variance of each step. Then, the original data x s Noise image at any step It is expressed as: ∈ represents the added noise, and The signal rate and noise rate are constructed by reparameterization techniques (which can be implemented using existing technologies). Secondly, the reverse diffusion process of the model is carried out to gradually predict the noise ∈ added in the forward stage: Finally, in the optimization phase of model training, the following loss function is used to achieve model optimization: Where ∈θ is the noise prediction result of the diffusion model.

[0052] Then, the prompt word integration module 4 uses a large language model, inputs an image editing target in the form of a text, and outputs multiple groups of text prompts with the same semantics but different forms, while avoiding the generation of irrelevant content, forming diversified and high-quality prompt information, and outputting multiple groups of text editing instructions.

[0053] Secondly, the image feature extraction module 5 and the editing instruction text feature extraction module 6 use the text-image pre-trained CLIP model based on remote sensing image self-supervision to perform image feature encoding and text feature encoding on the generated results of the optical remote sensing image and multiple groups of text editing instructions respectively. This process can be expressed as and f T (text), where f I and f T They respectively represent the image feature encoder and text feature encoder pre-trained using a multimodal contrastive learning method (which can be implemented using existing technologies) on remote sensing images, and text represent the currently generated image and the text editing instructions expanded by the large language model, respectively. This also obtains the feature encoding of the edited image in two different modalities: image and text.

[0054] Finally, the image-text similarity calculation module 7 uses the cosine distance to measure and f T The difference between the two feature encodings (text) is used as the optimization target in the fine-tuning process to update the generative model parameters again and achieve text-guided controllable editing. The optimization target is defined as: By optimizing the loss function L, the generation process is fine-tuned according to the text editing instructions, and the edited remote sensing image generation result is obtained.

[0055] A controllable remote sensing image editing method and system based on an image denoising diffusion model according to an embodiment of the present invention can perform stable and controllable text-guided remote sensing image editing on a single remote sensing image acquired by a remote sensing image acquisition device. A remote sensing image generation module 3 uses a denoising diffusion model to generate an unedited remote sensing image. A prompt word integration module 4 optimizes text editing instructions to alleviate the problem that the ambiguity of the text itself restricts the further application of image editing methods in the remote sensing field. An image feature extraction module 5 and an editing instruction text feature extraction module 6 are used to perform image and text feature encoding on the generation results of the optical remote sensing image and multiple groups of text editing instructions. By calculating the similarity between the image feature encoding of the optical remote sensing image generation result and the text feature encoding of the text editing instruction, the parameters and generation results of the denoising diffusion model in the remote sensing image generation module 3 are optimized to achieve better remote sensing image editing effects and support for real-world tasks such as disaster assessment.

[0056] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0057] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0058] In order to illustrate the purpose of the present invention, the above embodiments are only for description and not for limiting the scope of the present invention. The scope of the present invention is determined by the claims, and any equivalent substitutions and modifications that do not deviate from the principles and core features of the embodiments of the present invention should be included in the scope of the present invention.

[0059] It is obvious to those skilled in the art that the specific forms of the embodiments of the present invention are not limited to the details in the above exemplary embodiments. Therefore, these embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is limited by the claims rather than the above description. Any figure mark should not be regarded as limiting the claims. In addition, the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units, modules or devices can be implemented by the same unit, module or device through software or hardware.

[0060] Finally, although the above preferred embodiments are described in detail, they are only for describing the technical solutions of the embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art should understand that the technical solutions may be modified or replaced by equivalents, but should not depart from the principles and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A controllable editing method for remote sensing images based on image denoising diffusion model, characterized in that: The following steps are involved: Detect the target area and obtain a single remote sensing scene image of the target area; Using the acquired single remote sensing scene image of the target area, an optical remote sensing image is generated by using an image generation method based on a denoising diffusion model; A text generation method based on a large language model is used to diversify and expand a set of text editing instructions to generate multiple sets of optimized text editing instructions; An image encoder based on remote sensing image pre-training is used to encode image features of the generated results of optical remote sensing images; A text encoder based on remote sensing image pre-training is used to encode text features of multiple sets of text editing instructions, and the encoded results are averaged. The similarity between the image feature encoding of the optical remote sensing image generation result and the text feature encoding of the text editing instruction is calculated. According to the similarity calculation result, the text editing instruction is fine-tuned in the remote sensing image generation process by optimizing the loss function to obtain the edited remote sensing image generation result.

2. The method according to claim 1, characterized in that The target area is detected by using a remote sensing image acquisition device, a single remote sensing scene image of the target area is obtained and sent to a data processor; the data processor includes a remote sensing image generation module, a prompt word integration module, a generated image feature extraction module, an editing instruction text feature extraction module, and a picture-text similarity calculation module.

3. The method according to claim 2, characterized in that The remote sensing image generation module defines a Markov chain with a diffusion step, introduces random noise into the input data x in a progressive manner, and then learns the inverse process of the diffusion process to reconstruct the image from the noise to achieve image generation; the processing process of the remote sensing image generation module includes the following steps: First, construct the image pyramid {x N-1 ,x N-2 ,...,x 0 }, where x N-1 represents the input image itself, x 0 represents the input image after N-1 downsampling. For a scale s in (0,1,…,N-1), the forward process is represented as: where t∈(0,T) represents the number of steps in the forward process of adding noise to the image, is the variance at each step, represents the input image after N-1 downsampling and t-step forward process noise added, represents the standard normal distribution, I represents the variance, and T represents the total number of steps in the forward process; Then, the original data x s Noise image at any step It is expressed as: ∈ represents the added noise, and are the signal rate and noise rate constructed by the reparameterization technique; Secondly, the backward diffusion process of the model is carried out to gradually predict the noise ∈ added in the forward stage: where p θ represents the predicted probability distribution, Represents the mean value of the denoising operation at time t; Finally, in the optimization phase of model training, the following loss function is used to achieve model optimization: where ∈ θ Noise prediction results for the diffusion model.

4. The method according to claim 2, characterized in that: The prompt word integration module uses a large language model, inputs an image editing target in the form of a text, and outputs multiple groups of text prompts with the same semantics but different forms, while avoiding the generation of irrelevant content, forming diversified and high-quality prompt information, and obtaining multiple groups of text editing instructions.

5. The method according to claim 2, characterized in that: The generated image feature extraction module and the edit instruction text feature extraction module use the text-image pre-trained CLIP model based on remote sensing image self-supervision to perform image feature encoding and text feature encoding on the generated results of the optical remote sensing image and multiple groups of text editing instructions respectively. The process is expressed as and f T (text), where f I and f T They represent the image feature encoder and text feature encoder pre-trained using multimodal contrastive learning method on remote sensing images, respectively. and text represent the currently generated image and the text editing instructions expanded by the large language model, respectively.

6. The method according to claim 2, characterized in that The image-text similarity calculation module uses cosine distance to measure and f T The difference between the two feature encodings (text) is used as the optimization target in the fine-tuning process to update the generative model parameters again and achieve text-guided controllable editing; the optimization target is defined as: By optimizing the loss function L, fine-tuning of the text editing instructions in the remote sensing image generation process is achieved, and the edited remote sensing image generation result is obtained.

7. The method according to claim 2, characterized in that The remote sensing image acquisition device is a network camera; the data processor is a computer or an embedded motherboard; the remote sensing image generation module is a denoising diffusion model; the prompt word integration module is a large language model; the generated image feature extraction module and the editing instruction text feature extraction module are self-supervised remote sensing image text-image pre-training models.

8. A controllable editing system for remote sensing images based on image denoising diffusion model, characterized in that: It includes a remote sensing image acquisition device and a data processor; the data processor includes a remote sensing image generation module, a prompt word integration module, a generated image feature extraction module, an editing instruction text feature extraction module and a picture-text similarity calculation module; The remote sensing image acquisition device is used to detect the target area, obtain a single remote sensing scene image of the target area and send it to the data processor; The remote sensing image generation module generates an optical remote sensing image using an image generation method based on a denoising diffusion model; The prompt word integration module adopts a text generation method based on a large language model to diversify and expand a set of text editing instructions to generate multiple sets of optimized text editing instructions; The generated image feature extraction module uses a CLIP image encoder pre-trained based on remote sensing images to perform image feature encoding on the generated result of the optical remote sensing image; The editing instruction text feature extraction module uses a CLIP text encoder pre-trained based on remote sensing images to encode text features of multiple groups of text editing instructions, and performs an averaging operation on the encoded results; The image-text similarity calculation module adopts the cosine similarity calculation method to calculate the similarity between the image feature coding of the optical remote sensing image generation result and the text feature coding of the text editing instruction. According to the similarity calculation result, the text editing instruction is fine-tuned in the remote sensing image generation process by optimizing the loss function to obtain the edited remote sensing image generation result.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images