Remote sensing image super-resolution method and product based on diffusion model and multi-modal large language model

Abstract

The invention provides a remote sensing image super-resolution method based on a diffusion model and a multi-modal large language model, and relates to the technical field of image processing. According to the method, a pre-trained stable diffusion model is adopted, the model is trained on a large-scale data set, the stable and detailed super-resolution effect can be achieved, and the convergence speed is higher. According to the method, a multi-mode large language model is used for enhancing the semantic comprehension ability of the model, and detailed content description and texture description are provided. The invention also introduces an additional classifier which provides global information and a simple signal about the LR. The prior information is adaptively integrated into a diffusion model through a cross attention module. And finally, through combination of the strong generation capability of the stable diffusion model and the semantic understanding capability of the MLLM, the method provided by the invention is excellent in the aspect of remote sensing image reconstruction, and excellent performance is realized in the aspect of quality of a plurality of images on different data sets.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a remote sensing image super-resolution method and product based on a diffusion model and a multimodal large language model. Background Art

[0002] Remote sensing images (RSI) allow the observation of objects on Earth from space, providing a valuable source of information for monitoring the Earth's surface. In recent years, remote sensing images have been widely used in various fields including environmental monitoring, resource exploration, and land cover classification. However, the acquisition of high-resolution (HR) remote sensing images is often hindered by hardware limitations. Image quality and resolution are also compromised due to atmospheric conditions. Therefore, there is a strong demand for effective methods to improve low-resolution (LR) images and enhance human perception to meet the requirements of various downstream practical applications.

[0003] Remote sensing image super-resolution (RSI SR) involves the use of computational methods and techniques to improve the resolution of images. The main goal of super-resolution is to reconstruct high-resolution images with rich texture details using low-resolution images, so as to obtain high-resolution images beyond the limitations of various factors. Therefore, remote sensing image super-resolution has become an active research topic in the field of remote sensing. Due to the rapid development of deep learning technology, super-resolution models based on deep neural networks, such as convolutional neural network (CNNs)-based models, transformer-based models, hybrid Transformer-CNN models, and Mamba-based models, have made significant progress. Although these discriminative methods using feed-forward convolutional networks may be sufficient at low magnifications for super-resolution, they often fail to capture the intricate details required for high magnifications. In order to restore visually convincing details, deep generative adversarial networks (GANs) have been applied to the super-resolution task. Generative adversarial networks (GANs) use adversarial optimization between the generator and the discriminator to exploit the interaction between the two components and push the generator towards restoring realistic images. Generative adversarial networks have demonstrated convincing image generation results. However, existing methods often suffer from several limitations because generative adversarial networks require carefully designed regularization and optimization tricks to control optimization instabilities and mode collapse.

[0004] Recently, denoising diffusion probabilistic models (DDPMs) have attracted great attention and recognition in image-to-image translation. DDPMs consist of a diffusion process and a reverse process. The diffusion process injects noise into the image by shifting the data distribution to the latent variable distribution, while the reverse process generates high-quality images by denoising. Due to its principled and well-defined probability diffusion mechanism, DDPMs not only overcome the training instability and mode collapse problems often encountered in generative adversarial networks (GANs), but also surpass the limitations of the peak signal-to-noise ratio (PSNR)-based loss function commonly used in discriminative models. Recent work introduces a diffusion probabilistic model for efficient remote sensing super-resolution (RSI SR), which trains a diffusion model from scratch on a specific remote sensing super-resolution dataset. Although it generates more realistic details, the generalization ability of the model is still not ideal due to being trained on a specific dataset, and the training time of the model is quite long. Summary of the invention

[0005] The present application provides a remote sensing image super-resolution method and product based on a diffusion model and a multimodal large language model to at least partially solve the above-mentioned problems.

[0006] In a first aspect, the present application provides a remote sensing image super-resolution method based on a diffusion model and a multimodal large language model, the method comprising: Obtain content description information and texture description information of low-resolution remote sensing images based on a multimodal large language model; Obtaining a global category of the low-resolution remote sensing image based on a classifier; The low-resolution remote sensing image, content description information and texture description information of the low-resolution remote sensing image, and the global category of the low-resolution remote sensing image are processed based on a stable diffusion model to obtain a super-resolution remote sensing image.

[0007] Optionally, the low-resolution remote sensing image, the content description information and the texture description information of the low-resolution remote sensing image, and the global category of the low-resolution remote sensing image are processed by a stable diffusion model to obtain a super-resolution remote sensing image, including: Processing the low-resolution remote sensing image based on a variational autoencoder to obtain a potential representation; Encoding the content description information and the texture description information of the low-resolution remote sensing image based on a text encoder to obtain description encoding information, wherein the description encoding information interacts with the latent representation through a cross-attention module; Encoding the global category of the low-resolution remote sensing image based on a category-aware encoder to obtain category encoding information, wherein the category encoding information modulates an intermediate feature map in a residual block of a U-Net architecture of a stable diffusion model through a spatial feature transform; A noise-free latent representation is obtained based on the U-Net architecture, and a super-resolution remote sensing image is obtained based on a variational self-decoder.

[0008] Optionally, the classifier and the stable diffusion model are obtained based on joint training of training samples. During the joint training process, the parameters of the classifier are updated, the parameters of the category-aware encoder to be trained are updated, and other parameters of the pre-trained stable diffusion model are frozen. The training samples are sample low-resolution remote sensing images carrying global category labels.

[0009] Optionally, the joint training process includes the following steps: Encode the sample low-resolution remote sensing image into a sample latent vector through a variational autoencoder , within t steps, the noise is gradually introduced into the sample latent vector through the diffusion process of the stable diffusion model to generate the sample noise latent vector , the stable diffusion model integrates the following parameters to predict the added noise: sample low-resolution remote sensing image , sample noise latent vector , sample content description information and sample texture description information , the parameters of the class perception encoder are updated based on the minimization of the difference between the predicted noise obtained by the stable diffusion model and the actual noise introduced in the diffusion process. The loss function is expressed as follows: ; in, , represents the prediction noise of the stable diffusion model, represents the category-aware encoder to be trained, represents a pre-trained text encoder with frozen parameters, cls Represents sample content description information and sample texture description information; The classifier to be trained is trained based on sample low-resolution remote sensing images with global category labels, and the loss function is expressed as follows: ; Among them, N is the number of sample low-resolution remote sensing images, M is the number of global categories, y ic Indicates i The true labels of the sample low-resolution remote sensing images, p ic Represents the classifier to be trained i The samples belong to c The predicted probability of each category; The loss function of the joint training of the classifier and the stable diffusion model is expressed as follows: ; in, L represents the total loss, L D represents the diffusion loss, L C represents the classification loss, λ Represents the balance coefficient.

[0010] Optionally, encoding the content description information and the texture description information of the low-resolution remote sensing image based on a text encoder to obtain description encoding information includes: Through pre-trained text encoder Extract the content description information of the low-resolution remote sensing image respectively and texture description information Text features: ; Content description information of the low-resolution remote sensing image and texture description information The text features are concatenated to obtain the description encoding information c : c = concat( f con , f tex ).

[0011] Optionally, the class-aware encoder comprises noise time schedule parameters, which provide a signal-to-noise ratio value for each time step during a joint training process.

[0012] The second aspect of the present application provides a remote sensing image super-resolution device based on a diffusion model and a multimodal large language model, and the remote sensing image super-resolution device based on a diffusion model and a multimodal large language model comprises: Description acquisition module, used to obtain content description information and texture description information of low-resolution remote sensing images based on a multimodal large language model; A category acquisition module, used for acquiring the global category of the low-resolution remote sensing image based on a classifier; The super-resolution processing module is used to process the low-resolution remote sensing image, the content description information and the texture description information of the low-resolution remote sensing image, and the global category of the low-resolution remote sensing image based on a stable diffusion model to obtain a super-resolution remote sensing image.

[0013] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the remote sensing image super-resolution method based on a diffusion model and a multimodal large language model as described in the first aspect of the present invention.

[0014] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the remote sensing image super-resolution method based on a diffusion model and a multimodal large language model as described in the first aspect of the present invention.

[0015] The fifth aspect of the present application provides a computer program product, including a computer program / instruction, which is implemented by a processor to implement the steps in the remote sensing image super-resolution method based on a diffusion model and a multimodal large language model as described in the first aspect of the present invention.

[0016] Existing diffusion models are usually trained from scratch on specific remote sensing datasets and only learn the mapping from LR to HR, resulting in limited generalization capabilities of existing diffusion models and only over-smoothed results. In order to solve these problems, this application proposes a remote sensing image super-resolution method based on a diffusion model and a multimodal large language model. The method adopts a pre-trained stable diffusion model, which is trained on a large-scale dataset, and can achieve stable and detailed super-resolution effects with faster convergence speed. In addition, the application also uses a multimodal large language model (MLLM) to enhance the semantic understanding ability of the model and provide detailed content and texture descriptions. In addition, the application also introduces an additional classifier, which provides global information and concise signals about LR. Combining the advantages of MLLM can provide content description information and texture description information about low-resolution remote sensing images, and these prior information are adaptively integrated into the diffusion model through the cross-attention module. Finally, by combining the powerful generation ability of the stable diffusion model with the semantic understanding ability of MLLM, the method proposed in this application performs well in reconstructing remote sensing images and achieves excellent performance in multiple image quality on different datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1It is a schematic diagram of high-level semantic information at different levels of remote sensing image images of a remote sensing image super-resolution method based on a diffusion model and a multimodal large language model provided by the present application; Figure 2 It is a flowchart of the steps of the remote sensing image super-resolution method based on the diffusion model and the multimodal large language model provided by the present application; Figure 3 It is a schematic diagram of the overall architecture of a stable diffusion model in a remote sensing image super-resolution method based on a diffusion model and a multimodal large language model provided in this application; Figure 4 It is a schematic diagram of a model framework of a remote sensing image super-resolution method based on a diffusion model and a multimodal large language model provided in this application; Figure 5 This is a schematic diagram of super-resolution results of representative methods under different categories of remote sensing image super-resolution methods based on diffusion models and multimodal large language models provided in this application. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] StableSR is an image super-resolution technology based on a pre-trained diffusion model, which aims to use the powerful generation ability of the diffusion model to improve the resolution and quality of real-world images. The following are the core principles of StableSR: 1. Using the pre-trained diffusion model, StableSR injects low-resolution image information into the diffusion model by fine-tuning the lightweight time-aware encoder and controllable feature wrapping module (CFW) based on the pre-trained Stable Diffusion model.

[0021] 2. Time-aware encoder, which is responsible for generating time-aware features, which are adaptively modulated in different iterations of the diffusion model to better preserve the generative prior knowledge. This design not only improves training efficiency, but also enhances the performance of super-resolution.

[0022] 3. Controllable feature packaging module, in order to deal with the randomness and information loss problems of the diffusion model, StableSR introduced a controllable feature packaging module. This module adjusts the output of the diffusion model through the residual connection of multi-scale intermediate features, thereby achieving a balance between fidelity and authenticity.

[0023] 4. Progressive aggregation sampling strategy, in order to support super-resolution tasks of arbitrary resolution, StableSR adopts a progressive aggregation sampling strategy. This strategy divides the image into overlapping blocks and uses a Gaussian kernel for fusion in each diffusion iteration, thereby achieving smoother boundary transitions.

[0024] StableSR retains the pre-trained diffusion prior without sacrificing its network structure and weights. It only needs to fine-tune some lightweight modulation layers for super-resolution tasks. Since it inherits the generative power of the stable diffusion model, it does not need to make any assumptions about degradation, thus achieving open domain image super-resolution. Most current methods try to use better and larger models to adapt to the mapping between low-resolution and high-resolution images, treating the image super-resolution problem as a purely low-level visual restoration problem.

[0025] In this application, a diffusion-based remote sensing image super-resolution method guided by a multimodal large language model (MLLM) is introduced, which combines the advantages of the powerful generation ability of the diffusion model and the semantic understanding ability of the MLLM. Previous related studies required training the diffusion model from scratch on a specific remote sensing super-resolution dataset, while the method of this application utilizes the prior knowledge of a stable diffusion model pre-trained on a large dataset. This enables the model of this application to achieve powerful and detailed super-resolution effects with faster convergence speed.

[0026] Furthermore, this application argues that high-level semantic information is crucial for low-level super-resolution tasks, especially in complementing low-resolution images. Figure 1 The high-level semantic information of remote sensing images at different levels is displayed, which provides key clues for super-resolution tasks. Specifically, in this application, category information is extracted by a classifier, and the corresponding prior extractor is a classifier. Content information and texture information are extracted by a multimodal large language model, and the corresponding prior extractor is a multimodal large language model. The multimodal large language model has demonstrated strong capabilities in multimodal understanding. Therefore, this application uses MLLM to generate description information (including content description information and texture description information) of low-resolution images as semantic priors. These description information contain both semantic content and detailed texture information, which are cleverly integrated into the super-resolution model to guide the diffusion model to generate high-fidelity and realistic images. This application uses cross-attention to effectively combine semantic priors with generative priors.

[0027] In addition, the present application further attempts to introduce global category semantic information (specifically, the category to which the main body of the picture belongs) in the super-resolution (SR) process. The present application does not use a pre-trained classifier, but proposes a jointly optimized image classifier to achieve generalization. Global category information, as global semantics, is used as an embedding in the latent space together with the embedding of the low-resolution image. This condition can be generated at multiple scales in the denoising process, thereby improving the overall quality of the output image. The present application uses three public remote sensing datasets: AID, DOTA and DIOR to comprehensively evaluate the effectiveness of the method proposed in the present application. A large number of experiments have proved that the method of the present application is always superior to existing methods.

[0028] Specifically, the diffusion-based remote sensing image super-resolution method guided by a multimodal large language model proposed in this application combines the advantages of generative priors in stable diffusion and semantic priors in a multimodal large language model to improve the quality of super-resolution images. In this application, the advantages of a multimodal large language model and a classifier are used to extract multi-level semantic information. The MLLM provides detailed content and texture descriptions, while the classifier provides valuable global category information. These multi-level semantic information are adaptively integrated in the diffusion model through a cross-attention module. This application also designs a category-aware encoder to insert category information into the U-Net.

[0029] Specifically, the present application provides a flowchart of a remote sensing image super-resolution method based on a diffusion model and a multimodal large language model, such as Figure 2 As shown, specifically, the remote sensing image super-resolution method based on the diffusion model and the multimodal large language model includes the following steps: S201, obtaining content description information and texture description information of the low-resolution remote sensing image based on a multimodal large language model.

[0030] S202: Acquire a global category of the low-resolution remote sensing image based on a classifier.

[0031] S203: Processing the low-resolution remote sensing image, content description information and texture description information of the low-resolution remote sensing image, and the global category of the low-resolution remote sensing image based on a stable diffusion model to obtain a super-resolution remote sensing image.

[0032] The rapid development of deep learning has triggered the development of numerous RSI SR methods. These deep learning-based methods have proven to outperform traditional techniques, marking a significant advancement in the field. These methods can be divided into three categories: discriminant-based methods, GAN-based methods, and diffusion-based methods.

[0033] In the embodiment of the present invention, a stable diffusion model is used as the basic diffusion framework. Compared with the diffusion model, the stable diffusion model has a faster training speed. The traditional diffusion model requires the full-size image to be input into the U-Net during the reverse diffusion process, which will lead to a significant slowdown when processing large-size images and time steps. However, stable diffusion solves this defect by introducing an autoencoder to compress the image into a low-dimensional representation, called a latent representation. This compression enables stable diffusion to achieve faster processing speeds, making it a more efficient alternative.

[0034] Stable Diffusion (SD) is a deep learning architecture based on Latent Diffusion Model (LDM), which is mainly used to generate high-quality images based on text prompts. Its network architecture mainly includes the following three core components: Variational Autoencoder (VAE), U-Net and Text Encoder. Among them, the role of VAE is to compress the input image from the pixel space to the low-dimensional latent space. During the training process, the encoder of VAE converts the image (such as 512×512×3) into a latent representation (such as 64×64×4), which significantly reduces the computational and memory requirements. During the generation process, the decoder of VAE restores the denoised latent representation to the final image. U-Net is the core component of the stable diffusion model, which is responsible for predicting and removing noise in the latent space. It adopts an encoder-decoder structure combined with a residual network (ResNet) and a vision transformer (VisionTransformers). During the generation process, U-Net receives the noisy latent representation and text embedding as input, gradually predicts and removes the noise, and finally generates an image that matches the text description.

[0035] Stable diffusion models usually use a pre-trained CLIP text encoder to process the input text prompts. CLIP converts the text into embedding vectors, which serve as conditional information to guide the denoising process of U-Net, ensuring that the generated image is consistent with the text description.

[0036] In addition, the stable diffusion model also introduces a scheduler to control the process of adding and removing noise. This architecture significantly improves generation efficiency and image quality by performing diffusion and denoising in the latent space.

[0037] The overall structure of the stable diffusion model is as follows Figure 3 As shown, first, the input image Use variational autoencoder (VAE) to encode and obtain the latent representation Then, Gaussian noise is added iteratively.t After one step, we get a noisy latent representation z t The goal of the training process is to t Use text or image based c UNet architecture to estimate Gaussian noise .

[0038] UNet combines residual blocks and criss-cross attention modules at various spatial resolutions to improve its predictions. Text-based cues are processed through the CLIP text encoder to generate multimodal features, which are then integrated into the criss-cross attention layer of UNet. In addition, the time course of the diffusion step is embedded in UNet through residual blocks. The optimization process can be expressed as: (1) in, represents the original Gaussian noise, represents the prediction noise, and iterative refinement leads to a noise-free latent representation, denoted as , followed by a VAE-based decoder D Reconstruct it into the pixel domain and get the output image In order to incorporate the text context into the image synthesis process, the stable diffusion adopts the cross attention module. The latent features of the image, denoted as F, and the text encoding features obtained from the CLIP text encoder, denoted as F c , is transformed through the projection layer. This transformation produces the query Q = W Q .F , key K = W K ·F c , and value W = W V ·F c ,in W Q 、W K 、W V Denote the weight matrices of query, key, and value projection respectively. The attention module aggregates value features through a weighted aggregation process, which is defined as follows: (2) Where d represents the output dimension of key and query features. Then, the latent features are refined based on the results obtained from the attention block.

[0039] In the stable diffusion model, the Cross-Attention Mechanism ensures that the generated image is closely related to the input text description by combining text embedding with the latent representation of the image.

[0040] The working principle of the cross-attention module is as follows: The stable diffusion model uses the pre-trained CLIP model to convert the input text into an embedding vector. These text embeddings interact with the potential representation of the image (generated by the VAE encoder) through the cross-attention layer. In the cross-attention layer, each word of the text embedding calculates an attention score with each position (or block) of the image potential representation, thereby establishing a semantic association between the text and the image. The cross-attention module generates attention scores by calculating the similarity between the text word embedding and the image potential representation. These scores indicate the importance of each word in the text to each position in the image. For example, in the prompt "a woman with green eyes", the cross-attention module associates "green" and "eyes" with the corresponding parts in the image, ensuring that the generated woman has green eyes.

[0041] The cross-attention module guides the image generation process in this way, ensuring that the generated image is semantically consistent with the text description. It not only focuses on the keywords in the text, but also understands the grammatical structure and contextual information. For example, by analyzing the cross-attention map, the researchers found that certain grammatical structures (such as the relationship between adjectives and nouns) affect the effect of image generation.

[0042] The cross-attention module plays a vital role in the stable diffusion model, which enables the model to accurately generate images based on the text description and avoid generating content that is irrelevant to the text. It also provides support for the interpretability of the model. By analyzing the attention map, we can understand how the model maps text to images. In addition, the cross-attention module is also the focus of model fine-tuning (such as LoRA). By adjusting the weights of these layers, stylized generation can be efficiently achieved.

[0043] The cross-attention module ensures that the generated results are highly consistent with the input prompts through the semantic association between text and images.

[0044] When training the diffusion model, the input image is first converted into a latent representation through the VAE encoder, and then Gaussian noise is gradually added to the latent representation, a process called diffusion.

[0045] The denoising process is the core of the stable diffusion model. The goal is to gradually remove the noise through a denoising network (usually U-Net) and restore the original potential representation. U-Net learns the distribution of noise, predicts the noise at each time step, and gradually restores a clear potential representation.

[0046] In an embodiment of the present invention, a remote sensing image super-resolution framework based on a diffusion model and a multimodal large language model is proposed to address the challenge of remote sensing super-resolution. Figure 4 As shown, the remote sensing image super-resolution framework based on the diffusion model and the multimodal large language model consists of three main modules: a stable diffusion module, an MLLM semantic prior module, and a class-aware encoder module. First, this application studies the formulation of the super-resolution problem, aiming to lay a solid foundation for the method of this application. Next, this application outlines the remote sensing image super-resolution framework based on the diffusion model and the multimodal large language model, and explains in detail the implementation details of the stable diffusion module, the MLLM semantic prior module, and the class-aware encoder module.

[0047] Remote sensing image super-resolution is the process of improving the resolution and detail level of remote sensing images. Traditional image super-resolution tasks involve learning to obtain high-resolution images from low-resolution images. x to high resolution images y The mathematical expression of the mapping is as follows: (3) Given a low-resolution input, the conditional probability density function of the high-resolution image is It is usually modeled as a standard Gaussian distribution or a Laplace distribution.

[0048] Among them, the mean is the true super-resolution high-definition image. Since image super-resolution is an ill-posed problem, a single low-resolution image may correspond to multiple high-resolution images, and the model may learn the average of multiple high-resolution images. In addition, changes in imaging features of different sensors may hinder the model from reaching the optimal state, resulting in blurred output. Taking the above problems into consideration and inspired by the multimodal large language model (MLLM), this application proposes to introduce additional prior information to enrich the information of low-resolution images, thereby transforming the one-to-many problem into a relatively deterministic mapping. This application introduces the following modeling methods: (4) in con Indicates additional conditions, i.e., prior knowledge. In this application, the condition conIt includes three aspects of description: category, content, and texture. Image categories describe the content or subject represented by the image. They provide basic information about the content of the image and help understand and classify the image. Image content description involves the presence, location, and characteristics of objects or scenes, including their categories, shapes, sizes, etc. It also includes the depiction of people, actions, or emotional expressions in the image. In addition, it also includes environmental background and scene characteristics, such as indoor or outdoor settings, natural landscapes or urban landscapes, and other relevant features or specific contextual information related to the image. These descriptions help to understand and convey the content of the image more accurately. Image texture description involves capturing surface features, details, and texture patterns, including features such as smoothness, roughness, texture density, and texture direction.

[0049] It also covers the color features within the image, including brightness, saturation, and hue. Texture and color descriptions provide detailed information about the visual features of the image, helping to more accurately depict and express the appearance and visual perception of the image. Obviously, these multi-level prior descriptions play a key role in supplementing the super-resolution model with detailed information. By decomposing the image into category, content, and texture components, this modeling approach provides better guidance for the reconstruction results, resulting in higher fidelity and reasonable diversity.

[0050] Specifically, step S203 includes the following sub-steps: S2031, processing the low-resolution remote sensing image based on a variational autoencoder to obtain a latent representation.

[0051] S2032, encoding the content description information and texture description information of the low-resolution remote sensing image based on a text encoder to obtain description encoding information, and the description encoding information interacts with the latent representation through a cross-attention module.

[0052] S2033, encoding the global category of the low-resolution remote sensing image based on a category-aware encoder to obtain category coding information, wherein the category coding information modulates an intermediate feature map in a residual block of a U-Net architecture of a stable diffusion model through a spatial feature transformation.

[0053] S2034, obtaining a noise-free latent representation based on the U-Net architecture, and obtaining a super-resolution remote sensing image based on a variational self-decoder.

[0054] like Figure 4 As shown, it shows the overall framework of the remote sensing image super-resolution framework based on the diffusion model and the multimodal large language model proposed in this application. The diffusion model and the multimodal large language model are used to meet the existing challenges. The whole framework consists of three main components: a multimodal large language model, a stable diffusion model, and a category-aware encoder.

[0055] The framework integrates multiple priors into image super-resolution, including semantic priors generated by MLLM, pre-trained stable diffusion model (SD) priors and global category priors. The semantic priors and category priors are encoded to extract features, which are then adaptively injected into the denoising process through cross-attention to modulate the latent representation of SD.

[0056] Specifically, this application uses a pre-trained text-to-image (T2I) SD model as the backbone network to improve the stability and convergence speed of training. In addition, in the semantic prior extraction stage, this application uses the most advanced MLLM to obtain content description information and texture description information for low-resolution (LR) images as semantic priors. These priors are then input into the CLIP text encoder, resulting in two types of embeddings.

[0057] In order to effectively combine the semantic priors obtained in the first stage with the well-defined diffusion model, this application designs a cross-attention module. This module promotes the interaction between the semantic priors generated in the first stage and the SD model priors.

[0058] In addition, this application introduces a category-aware encoder to modulate the U-Net features, aiming to enhance the category-awareness of the pre-trained SD model. Finally, the remote sensing image super-resolution framework based on the diffusion model and multimodal large language model outputs high-quality super-resolution images.

[0059] In recent years, significant progress has been made in the field of language understanding and generation through the development of large language models (LLMs). These models are trained on large text corpus datasets, allowing them to generate contextually relevant text. The success of large language models has triggered the exploration of incorporating visual modalities into large language models, resulting in multimodal large language models (MLLMs). In the field of multimodal large language models, images are usually processed using pre-trained visual encoders (such as visual transformers) and combined with text annotation embeddings within the framework of large language models. This fusion of visual and textual information extends the semantic understanding capabilities of large language models, making multimodal large language models excel in generating responses based on visual inputs.

[0060] Therefore, this application uses LLaVA to perceive low-resolution (LR) images and extract semantically meaningful information. LLaVA effectively leverages comprehensive annotations including titles and bounding boxes to generate high-quality captions as well as question-answer pairs directly related to the image. In this application's method, the application aims to enhance the semantic understanding of multimodal language models from the perspective of content and texture by formulating appropriate instructions, such as: The instructions are: #Task: Description of Remote Sensing Image Content Make a concise description describing the main content of the provided remote sensing image. Focus on key features and keep the description within 100 words. #Guidelines: Identify the main subject or object. Describe the scene type and any significant details. Be accurate and concise. #Sample Output: "Satellite imagery shows a densely populated urban area with a central park surrounded by tall buildings and a road network. A river can be seen in the suburbs." The output is: A satellite image showing a suburban neighborhood characterized by single-family homes with clearly defined plots and gardens. The area includes intersecting streets, driveways, and some green areas. Cars are parked along the roads, indicating residential activity.

[0061] Another example: The instructions are: #Task: Remote Sensing Image Texture Description Create a short description that captures the texture of the provided remote sensing image. Highlight visual patterns and surface features in 100 words or less. #Guidelines: - Describe texture patterns and surface features. - Mention any tonal or tonal variations that affect the texture. - Use vivid language to convey the nature of the texture. #Sample Output: "The image is characterized by a mix of textures: a patchwork of farmland, the smooth surface of a lake, and the rough texture of distant mountains." Answer: In this aerial image, a lush courtyard brings soft, irregular shapes. Intersecting paths provide linear contrast, while the tree line under the shade enriches the scene with darker tones, blending man-made order with natural vitality.

[0062] In the RSI SR task, the integration of auxiliary image data is crucial to achieve high-quality output. Therefore, this application adds a novel category-aware encoder to the diffusion architecture. The encoder is designed to provide the diffusion model with important contextual clues about the category of the subject depicted in the low-resolution image.

[0063] The method proposed in this application first uses a classifier to extract relevant features from low-resolution images. In order to improve the accuracy of the prediction, this application uses ResNet-50 as the classifier of this application and uses low-resolution images as input for training to enhance the fidelity and context understanding ability of the super-resolution process. The objective function is the cross entropy loss, as shown below: (5) Among them, N is the number of sample low-resolution remote sensing images, M is the number of global categories, y ic Indicates i The true labels of the sample low-resolution remote sensing images, p ic Represents the classifier to be trained i The samples belong to c The predicted probability of each category; The output of the class-aware encoder is used to modulate the intermediate feature map in the residual block of the U-Net architecture through spatial feature transformation. In addition, the application also integrates temporal information into the encoder to enhance the overall quality of the generated image.

[0064] In the early steps of the process, when the generated images contain limited information and low signal-to-noise ratio, the stronger constraints provided by the class-aware encoder provide more useful guidance to the diffusion process, improving the understanding of the information that needs to be reconstructed. In contrast, as the diffusion process proceeds and more image information is reconstructed, the constraints from the LR images can be adjusted to be less dominant. This flexibility is facilitated by introducing the noise schedule as a hyperparameter that provides the signal-to-noise ratio value at each time step during training. By considering the temporal aspect, we strike a balance between leveraging the guidance of the LR images throughout the diffusion process. Furthermore, introducing temporal conditions in the diffusion process helps to balance the use of low-resolution images. By combining the class-aware encoder and integrating temporal information, our method not only improves the quality of image reconstruction but also surpasses existing state-of-the-art super-resolution models, as demonstrated in our comprehensive experimental evaluation.

[0065] In this application, two consecutive cross-attention layers are deployed to integrate additional semantic priors and enhance the model’s ability to exploit rich contextual information. These priors are generated by MLLM and incorporated into the diffusion model.

[0066] Given an aspect ratio-distorted image x, a multimodal language model (MLLM) generates a description of its content and texture description This application uses the pre-trained text encoder in CLIP To extract text features: (6) For simplicity, this application follows the sequence dimension c = concat( f con ， f tex) directly connects these two types of features. Then the present application enables the interaction between the text features and the latent features of the diffusion model. k+1 layer x k+1 , using cross attention as follows: (7) In this way, after providing sufficient semantic prior description, criss-cross attention adaptively assigns weights to each word. It allows adaptive selection of semantic features. The effectiveness of this setting is further confirmed by ablation studies.

[0067] In the present application, the classifier and the stable diffusion model are obtained based on joint training of training samples. During the joint training process, the parameters of the classifier are updated, the parameters of the category-aware encoder to be trained are updated, and other parameters of the pre-trained stable diffusion model are frozen. The training samples are low-resolution remote sensing images carrying global category labels.

[0068] Specifically, in this application, during the joint training process of the model, first, the LR image is encoded into a latent vector . Then, in Within the range of random numbers t steps, noise is gradually introduced into the vector through the diffusion process to generate the noise potential state The remote sensing image super-resolution framework based on diffusion model and multimodal large language model integrates the following parameters to predict the added noise: low-resolution image , Noise potential state , content cues, and texture cues to predict the added noise.

[0069] The optimization goal of the model is to minimize the difference between the predicted noise and the actual noise introduced during the diffusion process. This goal can be stated as follows: (8) in, and Represents the noisy predictions for the stable diffusion model. is the category-aware encoder to be optimized. is a frozen CLIP encoder. In addition, the final loss function of the model in this application is designed as follows: (9) in, L represents the total loss, L D represents the diffusion loss, L C represents the classification loss, λ Denotes the equilibrium coefficient. To reduce computational overhead and training time, all parameters of the stable diffusion model are frozen during training.

[0070] In the testing phase, this paper adopts a classifier-free guidance strategy to improve the quality of generated images without additional training. The strategy involves using classifier-free negative cues to guide the diffusion model. In each step, predictions are made using cues generated from a multimodal large language model. At the same time, the cues are replaced with negative cues, denoted as c neg The predictions generated using the two cues are fused to obtain the final output.

[0071] In the experiment, a combination of negative words such as "blurry, pointy, noisy, unclear, low resolution, oversmoothed" was used as negative cues to generate higher quality images. This approach enables the application to leverage the power of negative cues without the need for a separate classifier, ultimately improving the entire image generation process.

[0072] In the remote sensing super-resolution task, this application uses three popular datasets to evaluate the performance of the model of this application, namely AID, DOTA and DIOR. According to the experimental setting, the training set contains 3000 images randomly selected from the AID dataset, and the size of each image is 640×640. Specifically, this application selects 100 images from each of the 30 categories of AID to construct the training set. In addition, this application creates a test set for each category, each containing 10 images, ensuring no overlap with the training set, and a total of 300 test images. In addition, this application also uses subsets of the DOTA and DIOR datasets for testing, containing 700 and 1000 images respectively. The resolution of these images is 512×512. Therefore, the test set of this application contains a total of 2000 images. In the simulation experiment of this application, this application uses bicubic interpolation for image degradation.

[0073] In this application, the pre-trained SD-v1.5 model is used as the text-to-image (T2I) model. The remote sensing image super-resolution framework model based on the diffusion model and the multimodal large language model is fine-tuned using the AdamW optimizer with a weight decay of 0.01 and 100,000 iterations. The training parameters include batch size 32 and learning rate 5e-5, and all operations are performed on a single NVIDIA A100 GPU. For image generation, this application uses DDPM sampling over 50 time steps, and the balance coefficient λ is set to 0.05.

[0074] In order to comprehensively evaluate the performance of remote sensing image super-resolution methods, this application uses four metrics commonly used in SRI SR tasks to evaluate the quality of generated images: Fréchet Inception Distance (FID), Learning Perceptual Patch Similarity (LPIPS), Deep Image Structure and Texture Similarity (DISTS), and the widely recognized Peak Signal-to-Noise Ratio (PSNR). Specifically, FID is used to capture the similarity between the generated image and the real image. LPIPS and DISTS are used for perceptual quality assessment. PSNR is used to evaluate the fidelity of image reconstruction. In addition, considering the lack of real images in actual experiments, this application also uses the Natural Image Quality Evaluator (NIQE) as a no-reference metric. This metric enables this application to gain in-depth understanding of the perceptual quality and high-frequency details of the generated images.

[0075] This paper compares the proposed remote sensing image super-resolution framework model based on diffusion model and multimodal large language model with the state-of-the-art (SOTA) single image super-resolution methods, including EDSR, RCAN, HAT-L, MSRGANESRGAN, SPSR, SR3, IRSDE and EDiffSR. These methods are carefully selected to ensure a comprehensive evaluation. Specifically, EDSR, RCAN and HAT-L are discriminative model-based models, which adopt wide convolutional neural networks (CNNs), Transformer architecture and their variants respectively. MSRGAN is a generative adversarial network (GAN) based method, which is a modified version of SRGAN that removes batch normalization (BN) layers to avoid artifacts and uses the same perceptual loss as ESRGAN

[77] . SPSR combines a carefully designed gradient loss to preserve structural details and has shown good performance. On the other hand, SR3 first adopts a denoising diffusion probability model to iteratively improve the noisy output. IRSDE and EDiffSR are recently proposed diffusion methods.

[0076] 1) Quantitative comparison: Based on the performance of different methods on various categories of the AID dataset. This application uses FID as a quantitative evaluation indicator. A lower FID indicates that the difference between the generated image and the real image is smaller, which means that the quality of the generated image is higher and very similar to the real image. Overall, the generation method based on GANs is significantly better than the generation method based on the discriminative model. This is because the adversarial training method of GANs helps to generate more texture details. In addition, the diffusion model has more advantages than the GAN method. The fundamental reason is that the diffusion model has a stronger distribution modeling ability and can better capture the distribution of high-dimensional images. According to the experimental results, it can be determined that the remote sensing image super-resolution framework based on the diffusion model and the multimodal large language model of this application performs best in most categories. It is worth noting that the remote sensing image super-resolution framework based on the diffusion model and the multimodal large language model ranks in the top two in almost all categories, which makes the method of this application reach the SOTA level in terms of the average FID of all categories. Specifically, the method of this application is 1.56 higher than the second-ranked method EDiffSR in terms of FID, which verifies the effectiveness of the method proposed in this application. Although both EDiffSR and the method of this application belong to diffusion models, the reason why the method of this application performs better is that it utilizes more prior information, including pre-trained diffusion models, text and content descriptions of images, and category information. This prior knowledge can supplement the missing information in LR input and provide richer input conditions, thereby improving performance.

[0077] In addition, this application evaluated the FID, LPIPS, DISTS and NIQE indicators of each method on three datasets (AID, DOTA and DIOR). The experimental results show that the method of this application achieves the best performance in all three indicators. It is worth noting that the GAN-based method always achieves the best results in LPIPS. The reason behind this may be that these methods incorporate LPIPS loss during training. Among all diffusion-based methods, the method of this application outperforms the others. DISTS is used to measure the structure and details of an image, and the method of this application achieves the lowest DISTS score. This shows that the method of this application is able to preserve the image structure and restore fine details.

[0078] 2) Qualitative comparison: In order to intuitively evaluate the effectiveness of different methods, this application presents a visual display of super-resolution images. Figure 5 The super-resolution results of representative methods in different categories are shown. Figure 5In the figure, the image samples from the first row to the last row are: railwaystation_17, playground_249, school_69, resort_20, farmland_89, square_156, viaduct_109, storagetanks_93. The images in the columns from left to right are processed by Bicubic, HAT-L, ESRGAN, EDiffSR and the method of the present application. It can be seen that, in general, the method of the present application achieves the best results in preserving structure and restoring texture, and is closer to the real image. For example, in the playground_249 sample, the method of the present application is able to restore the line markings on the football field. In contrast, the discriminative method and the GAN-based method destroy the structure of the image, and the line markings generated by the EDiffSR method based on the diffusion model appear blurred. The same phenomenon is also observed in other categories of the present application. The method of the present invention performs better on images with high structural textures. The present invention attributes this to the fact that the text description provides prior knowledge for the diffusion model, which helps preserve the structure.

[0079] It can be seen that the present application proposes a multimodal large language model remote sensing image super-resolution method based on diffusion. The core idea is to introduce model priors and semantic priors to alleviate the ill-posed one-to-many problem in image super-resolution. In order to enhance the robustness of the model, the present application directly inherits the pre-trained stable diffusion model on a large-scale dataset. The present application only fine-tunes a small number of parameters to adapt the model to the remote sensing super-resolution task. In order to provide more image information, the present application uses a multimodal large language model to generate content and texture descriptions to provide clues for the model to fill in texture details. The present application also combines image category information to enable the model to have a high-level understanding of low-resolution inputs. The present application conducted experiments on three datasets, and the results consistently demonstrated the effectiveness of the proposed method.

[0080] Based on the same inventive concept, the present application also provides a remote sensing image super-resolution device based on a diffusion model and a multimodal large language model, and the remote sensing image super-resolution device based on a diffusion model and a multimodal large language model includes: Description acquisition module, used to obtain content description information and texture description information of low-resolution remote sensing images based on a multimodal large language model; A category acquisition module, used for acquiring the global category of the low-resolution remote sensing image based on a classifier; The super-resolution processing module is used to process the low-resolution remote sensing image, the content description information and the texture description information of the low-resolution remote sensing image, and the global category of the low-resolution remote sensing image based on a stable diffusion model to obtain a super-resolution remote sensing image.

[0081] Optionally, the super-resolution processing module includes: A first processing submodule is used to process the low-resolution remote sensing image based on a variational autoencoder to obtain a potential representation; A first encoding submodule is used to encode the content description information and the texture description information of the low-resolution remote sensing image based on a text encoder to obtain description encoding information, and the description encoding information interacts with the potential representation through a cross attention module; A second encoding submodule is used to encode the global category of the low-resolution remote sensing image based on a category-aware encoder to obtain category encoding information, wherein the category encoding information modulates an intermediate feature map in a residual block of a U-Net architecture of a stable diffusion model through a spatial feature transformation; The second processing submodule is used to obtain a noise-free potential representation based on the U-Net architecture and obtain a super-resolution remote sensing image based on a variational self-decoder.

[0082] Optionally, the classifier and the stable diffusion model are obtained based on joint training of training samples. During the joint training process, the parameters of the classifier are updated, the parameters of the category-aware encoder to be trained are updated, and other parameters of the pre-trained stable diffusion model are frozen. The training samples are sample low-resolution remote sensing images carrying global category labels.

[0083] Optionally, the joint training process includes the following steps: Encode the sample low-resolution remote sensing image into a sample latent vector through a variational autoencoder , within t steps, the noise is gradually introduced into the sample latent vector through the diffusion process of the stable diffusion model to generate the sample noise latent vector , the stable diffusion model integrates the following parameters to predict the added noise: sample low-resolution remote sensing image , sample noise latent vector , sample content description information and sample texture description information , the parameters of the class perception encoder are updated based on the minimization of the difference between the predicted noise obtained by the stable diffusion model and the actual noise introduced in the diffusion process. The loss function is expressed as follows: ; in, , represents the prediction noise of the stable diffusion model, represents the category-aware encoder to be trained, represents a pre-trained text encoder with frozen parameters, cls Represents sample content description information and sample texture description information; The classifier to be trained is trained based on sample low-resolution remote sensing images with global category labels, and the loss function is expressed as follows: ; Among them, N is the number of sample low-resolution remote sensing images, M is the number of global categories, y ic Indicates i The true labels of the sample low-resolution remote sensing images, p ic Represents the classifier to be trained i The samples belong to c The predicted probability of each category; The loss function of the joint training of the classifier and the stable diffusion model is expressed as follows: ; in, L represents the total loss, L D represents the diffusion loss, L C represents the classification loss, λ Represents the balance coefficient.

[0084] Optionally, the second encoding submodule is specifically used to: Through pre-trained text encoder Extract the content description information of the low-resolution remote sensing image respectively and texture description information Text features: ; Content description information of the low-resolution remote sensing image and texture description information The text features are concatenated to obtain the description encoding information c : c = concat( f con , f tex ).

[0085] Optionally, the class-aware encoder comprises noise time schedule parameters, which provide a signal-to-noise ratio value for each time step during a joint training process.

[0086] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the remote sensing image super-resolution method based on a diffusion model and a multimodal large language model as described in any of the above embodiments are implemented.

[0087] Based on the same inventive concept, the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the remote sensing image super-resolution method based on the diffusion model and the multimodal large language model described in any of the above embodiments are implemented.

[0088] Based on the same inventive concept, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the remote sensing image super-resolution method based on a diffusion model and a multimodal large language model described in any of the above embodiments.

[0089] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0090] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (apparatus), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0095] Finally, it should be noted that, in this application, relational terms such as first and second, etc., are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or terminal device including the elements.

[0096] The above is a detailed introduction to the remote sensing image super-resolution method based on a diffusion model and a multimodal large language model provided by the present invention. In this application, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A remote sensing image super-resolution method based on a diffusion model and a multimodal large language model, characterized in that: The method comprises: Obtain content description information and texture description information of low-resolution remote sensing images based on a multimodal large language model; Obtaining a global category of the low-resolution remote sensing image based on a classifier; The low-resolution remote sensing image, content description information and texture description information of the low-resolution remote sensing image, and the global category of the low-resolution remote sensing image are processed based on a stable diffusion model to obtain a super-resolution remote sensing image.

2. The remote sensing image super-resolution method based on diffusion model and multimodal large language model according to claim 1, characterized in that: The low-resolution remote sensing image, the content description information and the texture description information of the low-resolution remote sensing image, and the global category of the low-resolution remote sensing image are processed by a stable diffusion model to obtain a super-resolution remote sensing image, including: Processing the low-resolution remote sensing image based on a variational autoencoder to obtain a potential representation; Encoding the content description information and the texture description information of the low-resolution remote sensing image based on a text encoder to obtain description encoding information, wherein the description encoding information interacts with the latent representation through a cross-attention module; Encoding the global category of the low-resolution remote sensing image based on a category-aware encoder to obtain category encoding information, wherein the category encoding information modulates an intermediate feature map in a residual block of a U-Net architecture of a stable diffusion model through a spatial feature transform; A noise-free latent representation is obtained based on the U-Net architecture, and a super-resolution remote sensing image is obtained based on a variational self-decoder.

3. The remote sensing image super-resolution method based on diffusion model and multimodal large language model according to claim 2, characterized in that: The classifier and the stable diffusion model are obtained through joint training based on training samples. During the joint training process, the parameters of the classifier are updated, the parameters of the category-aware encoder to be trained are updated, and other parameters of the pre-trained stable diffusion model are frozen. The training samples are low-resolution remote sensing images carrying global category labels.

4. The remote sensing image super-resolution method based on diffusion model and multimodal large language model according to claim 3 is characterized in that: The joint training process includes the following steps: Encode the sample low-resolution remote sensing image into a sample latent vector through a variational autoencoder , within t steps, the noise is gradually introduced into the sample latent vector through the diffusion process of the stable diffusion model to generate the sample noise latent vector , the stable diffusion model integrates the following parameters to predict the added noise: sample low-resolution remote sensing image , sample noise latent vector , sample content description information and sample texture description information , the parameters of the class perception encoder are updated based on the minimization of the difference between the predicted noise obtained by the stable diffusion model and the actual noise introduced in the diffusion process. The loss function is expressed as follows: ； in, , represents the prediction noise of the stable diffusion model, represents the category-aware encoder to be trained, represents a pre-trained text encoder with frozen parameters, cls Represents sample content description information and sample texture description information; The classifier to be trained is trained based on sample low-resolution remote sensing images with global category labels, and the loss function is expressed as follows: ； Among them, N is the number of sample low-resolution remote sensing images, M is the number of global categories, y ic Indicates i The true labels of the sample low-resolution remote sensing images, p ic Represents the classifier to be trained i The samples belong to c The predicted probability of each category; The loss function of the joint training of the classifier and the stable diffusion model is expressed as follows: ； in, L represents the total loss, L D represents the diffusion loss, L C represents the classification loss, λ Represents the balance coefficient.

5. The remote sensing image super-resolution method based on diffusion model and multimodal large language model according to claim 2, characterized in that: The content description information and texture description information of the low-resolution remote sensing image are encoded based on a text encoder to obtain description encoding information, including: Through pre-trained text encoder Extract the content description information of the low-resolution remote sensing image respectively and texture description information Text features: ； Content description information of the low-resolution remote sensing image and texture description information The text features are concatenated to obtain the description encoding information c : c = concat( f con , f tex )。 6. The remote sensing image super-resolution method based on diffusion model and multimodal large language model according to claim 2, characterized in that: The class perceptual encoder includes noise schedule parameters that provide a signal-to-noise ratio value for each time step during a joint training process.

7. A remote sensing image super-resolution device based on a diffusion model and a multimodal large language model, characterized in that: The remote sensing image super-resolution device based on the diffusion model and the multimodal large language model includes: Description acquisition module, used to obtain content description information and texture description information of low-resolution remote sensing images based on a multimodal large language model; A category acquisition module, used for acquiring the global category of the low-resolution remote sensing image based on a classifier; The super-resolution processing module is used to process the low-resolution remote sensing image, the content description information and the texture description information of the low-resolution remote sensing image, and the global category of the low-resolution remote sensing image based on a stable diffusion model to obtain a super-resolution remote sensing image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the remote sensing image super-resolution method based on the diffusion model and the multimodal large language model described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the remote sensing image super-resolution method based on a diffusion model and a multimodal large language model as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the remote sensing image super-resolution method based on a diffusion model and a multimodal large language model described in any one of claims 1 to 6 are implemented.

Images