Remote sensing image panchromatic sharpening method and device based on denoising diffusion probability model

Abstract

The application provides a remote sensing image panchromatic sharpening method and device based on a denoising diffusion probability model. The method comprises the following steps: first, difference processing is performed on a high-resolution multispectral HRMS image and an interpolated multispectral IMS image obtained based on a low-resolution multispectral LRMS image, to obtain an original difference DM image; then, forward diffusion processing is performed on the original difference image by using a denoising diffusion probability model, to obtain a corresponding noise image; next, key features are extracted from a panchromatic PAN image and the LRMS image by using a feature extraction model; inverse denoising processing is performed on the noise image based on the key features by using the denoising diffusion probability model, to obtain a reconstructed difference DM image; then, the denoising diffusion probability model and the feature extraction model are trained with the objective of making the data distribution of the reconstructed difference DM image approximate to the data distribution of the original difference DM image; finally, the interpolated multispectral IMS image is added to the reconstructed DM image, to obtain a panchromatic sharpening result for the LRMS image and the panchromatic PAN image.

Description

Technical Field

[0001] This specification relates to the field of remote sensing image fusion technology, and in particular to a method and apparatus for panchromatic sharpening of remote sensing images based on a denoising diffusion probability model. Background Technology

[0002] Panchromatic sharpening is an image fusion method used in remote sensing. It fuses a high spatial resolution panchromatic (PAN) image with a low spatial resolution multispectral (LRMS) image to generate a high spatial resolution multispectral (HRMS) image. High-quality remote sensing images have wide applications in various fields, such as scene classification, semantic segmentation, land use mapping, detailed 3D urban reconstruction, environmental monitoring, and urban planning.

[0003] Panchromatic sharpening methods can be categorized into four main types: component substitution (CS) methods, multiresolution analysis (MRA) methods, variational optimization (VO) methods, and deep learning (DL) methods. Each method has its advantages and limitations. For example, CS methods perform well in terms of spatial quality but may introduce spectral distortion; MRA methods can provide high spectral fidelity but have relatively lower spatial quality. VO methods attempt to balance spatial enhancement and spectral preservation, but they are computationally expensive and may perform poorly in terms of fusion results. In recent years, remarkable achievements have been made using deep learning methods such as Convolutional Neural Networks (CNNs), Transformers, and Generative Adversarial Networks (GANs).

[0004] Traditional methods and deep learning (DL) based methods have some limitations, including the limitations of linear transformation processes, the feature smoothing problem faced by convolutional neural network (CNN) based methods, and the difficulty of stable training for generative adversarial network (GAN) based methods. Summary of the Invention

[0005] This invention proposes a method and apparatus for panchromatic sharpening of remote sensing images based on a denoising diffusion probability model, which solves the above problems by using a denoising diffusion probability model and changing the learning objective of the traditional fusion network.

[0006] According to the first aspect, a panchromatic sharpening method for remote sensing images based on a denoising diffusion probability model is provided. This method includes:

[0007] A differential processing method is applied to a high-resolution multispectral HRMS image and an interpolated multispectral IMS image obtained from a low-resolution multispectral LRMS image to obtain an original differential DM image. A denoising diffusion probability model is then used to perform forward diffusion processing on the original differential image to obtain a corresponding noisy image. A feature extraction model is used to extract key features from the panchromatic PAN image and the low-resolution multispectral LRMS image. The denoising diffusion probability model is then used to perform inverse denoising processing on the noisy image based on the key features to obtain a reconstructed differential DM image. The denoising diffusion probability model and the feature extraction model are trained with the goal of making the data distribution of the reconstructed differential DM image approximate the data distribution of the original differential DM image. Finally, the interpolated multispectral IMS image is added to the reconstructed DM image to obtain a panchromatic sharpening result for the low-resolution multispectral LRMS image and the panchromatic PAN image.

[0008] In one embodiment, the feature extraction model includes a convolution module and a spectral information mutual calibration module; the extraction of key features from the panchromatic PAN image and the low-resolution multispectral LRMS image using the feature extraction model includes:

[0009] The convolution module is used to process the panchromatic PAN image and the low-resolution multispectral LRMS image respectively to obtain the corresponding PAN features I. PAN and LRMS features I MS Using the spatial information mutual calibration module, global pooling and max global pooling of the panchromatic PAN image and the low-resolution multispectral LRMS image are averaged along the channel dimension to obtain multiple spectral information vectors v. p ; to store the multiple spectral information vectors v p Connected in series to form V c And use a multilayer perceptron (MLP) to process the V c , to obtain the I PAN and I MS Channel calibration weight CC PAN and CC MS ; In the channel dimension, the I PA and I MS respectively with CC PAN and CC MSPerform element-wise multiplication and then add them to themselves to obtain spectral cross-calibration information I′. PAN and I′ MS This is categorized under the key features mentioned above.

[0010] In one specific embodiment, the feature extraction model further includes a spatial information mutual calibration module; the step of extracting key features from the panchromatic PAN image and the low-resolution multispectral LRMS image using the feature extraction model further includes:

[0011] Using the spatial information mutual calibration module, the spectral mutual calibration information I′ is... PAN and I′ MS The concatenated vector is then subjected to average global pooling, max global pooling, and 1x1 convolution along the channel dimension, and the results of these three processes are concatenated into V. S ; the V S The information is input into a multilayer perceptron (MLP) to obtain the spectral mutual calibration information I′. PAN and I′ MS Corresponding spatial calibration weights SC PAN and SC MS ; in the channel dimension, the I′ PAN and I′ MS respectively with SC PAN and SC MS Perform element-wise multiplication and then add them to themselves to obtain spatial cross-calibration information I″. PAN and I″ MS This is categorized under the key features mentioned above.

[0012] In one embodiment, the forward diffusion process includes: based on the original difference image, performing image noise processing at each time step to obtain a noisy image corresponding to each time step, wherein the last noisy image is used as the noise image.

[0013] In one embodiment, the inverse denoising process includes: performing image denoising processing at each time step based on the noisy image to obtain a denoised image corresponding to each time step, wherein the last denoised image is used as the reconstructed differential DM image.

[0014] According to the second aspect, a panchromatic sharpening device for remote sensing images based on a denoising diffusion probability model is provided. The device includes:

[0015] The original difference image acquisition unit is configured to perform difference processing on the high-resolution multispectral HRMS image and the interpolated multispectral IMS image obtained based on the low-resolution multispectral LRMS image to obtain the original difference DM image. The noisy image acquisition unit is configured to perform forward diffusion processing on the original difference image using a denoising diffusion probability model to obtain the corresponding noisy image. The feature extraction unit is configured to extract key features from the panchromatic PAN image and the low-resolution multispectral LRMS image using a feature extraction model. The reconstructed difference image acquisition unit is configured to perform inverse denoising processing on the noisy image based on the key features using the denoising diffusion probability model to obtain the reconstructed difference DM image. The model training unit is configured to train the denoising diffusion probability model and the feature extraction model with the goal of making the data distribution of the reconstructed difference DM image approximate the data distribution of the original difference DM image. The sharpening unit is configured to add the interpolated multispectral IMS image to the reconstructed DM image to obtain a panchromatic sharpening result for the low-resolution multispectral LRMS image and the panchromatic PAN image.

[0016] In one embodiment, the feature extraction model includes a convolution module and a spectral information mutual calibration module. The feature extraction unit is specifically configured to: use the convolution module to process the panchromatic PAN image and the low-resolution multispectral LRMS image respectively, to obtain the corresponding PAN features I. PAN and LRMS features I MS Using the spectral information mutual calibration module, global pooling and max global pooling of the panchromatic PAN image and the low-resolution multispectral LRMS image are averaged along the channel dimension to obtain multiple spectral information vectors v. p ; to store the multiple spectral information vectors v p Connected in series to form V c And use a multilayer perceptron (MLP) to process the V c , to obtain the I PAN and I MS Channel calibration weight CC PAN and CC MS ; In the channel dimension, the I PAN and I MS respectively with CC PAN and CC MS Perform element-wise multiplication and then add them to themselves to obtain spectral cross-calibration information I′. PAN and I′ MS This is categorized under the key features mentioned above.

[0017] Furthermore, in one specific embodiment, the feature extraction model further includes a spatial information mutual calibration module; the feature extraction unit is further configured to: utilize the spatial information mutual calibration module to process the spectral mutual calibration information I′ PAN and I′ MS The concatenated vector is then subjected to average global pooling, max global pooling, and 1x1 convolution along the channel dimension, and the results of these three processes are concatenated into V. S ; the V S The information is input into a multilayer perceptron (MLP) to obtain the spectral mutual calibration information I′. PAN and I′ MS Corresponding spatial calibration weights SC PAN and SC MS ; in the channel dimension, the I′ PAN and I′ MS respectively with SC PAN and SC MS Perform element-wise multiplication and then add them to themselves to obtain spatial cross-calibration information I″. PAN and I″ MS This is categorized under the key features mentioned above.

[0018] In one embodiment, the noise image acquisition unit is specifically configured to: perform image noise addition processing at each time step based on the original difference image to obtain a noise-added image corresponding to each time step, wherein the last noise-added image is used as the noise image.

[0019] In one embodiment, the reconstructed differential image acquisition unit is specifically configured to: perform image denoising processing at each time step based on the noisy image to obtain denoised images corresponding to each time step, wherein the last denoised image is used as the reconstructed differential DM image.

[0020] This invention discloses a panchromatic sharpening method for remote sensing images based on a denoising diffusion probability model. The method constructs a forward diffusion component, a conditional injection and modal mutual calibration module, and an inverse denoising component to obtain panchromatic sharpening results. The forward diffusion component acquires the latent states of the differential (DM) image at each time step after denoising. Then, the conditional injection and modal mutual calibration module extracts key spatial and spectral information from panchromatic (PAN) and low-resolution multispectral (LRMS) data of different modalities, and uses spatial and spectral mutual calibration to guide the inverse denoising process. The inverse denoising component aims to optimize the parameters of the neural network under the guidance of the conditional injection information, completing the process of recovering the differential (DM) image from isotropic Gaussian noise. Finally, the interpolated multispectral (IMS) image is added to obtain the panchromatic sharpening result. This method addresses the shortcomings of current methods, provides high-quality image processing results, and is expected to improve the panchromatic sharpening effect of remote sensing images and the spatial-spectral quality of remote sensing images in downstream applications. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This diagram illustrates a flowchart of a remote sensing image panchromatic sharpening method based on a denoising diffusion probability model, as provided in an embodiment of this specification.

[0023] Figure 2 This diagram illustrates the overall structure of a remote sensing image panchromatic sharpening model based on a denoising diffusion probability model, as provided in an embodiment of the present invention.

[0024] Figure 3 This diagram illustrates the structure of the spectral information mutual calibration module provided in an embodiment of the present invention.

[0025] Figure 4 This diagram illustrates the structure of the spatial information mutual calibration module provided in an embodiment of the present invention.

[0026] Figure 5 This invention provides panchromatic sharpening results for different land cover types on Gaofen-2 satellite with reduced resolution, as shown in this embodiment.

[0027] Figure 6 This shows the results of full-color sharpening on a high-resolution CMOS imager provided in an embodiment of the present invention.

[0028] Figure 7 This shows the full-color sharpening display result on a QuickBird with reduced resolution provided by an embodiment of the present invention;

[0029] Figure 8 This shows the full-color sharpening display result on a WorldView3 with reduced resolution provided by an embodiment of the present invention;

[0030] Figure 9 This shows the full-resolution, full-color sharpening display result on Gaofen-2 provided by an embodiment of the present invention;

[0031] Figure 10 This shows the full-color sharpening display result on a full-resolution QuickBird provided by an embodiment of the present invention;

[0032] Figure 11 This shows the full-color sharpening display result on a full-resolution WorldView3 provided by an embodiment of the present invention;

[0033] Figure 12 This shows the generalized full-color sharpening result on a WorldView2 with reduced resolution provided by an embodiment of the present invention;

[0034] Figure 13 This diagram illustrates the structure of a remote sensing image panchromatic sharpening device based on a denoising diffusion probability model, as provided in an embodiment of this specification. Detailed Implementation

[0035] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0037] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0038] In the description of the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more.

[0039] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0040] This invention designs a method and device for panchromatic sharpening of remote sensing images based on a denoising diffusion probability model. It can effectively address the following issues: (1) Traditional panchromatic sharpening methods based on component transformation, multi-resolution analysis, and variational optimization methods have strong physical limitations and are difficult to meet real application scenarios; (2) Panchromatic sharpening methods based on convolutional neural networks tend to smooth features and do not have ideal sharpening effects on boundaries; (3) Based on the unstable training of generative adversarial models, this invention achieves better panchromatic sharpening effects by using a denoising diffusion probability model and changing the learning objectives of traditional fusion networks. It has high practical value and can be effectively used for panchromatic sharpening of remote sensing images and downstream application scenarios based on remote sensing images.

[0041] Figure 1 This diagram illustrates a flowchart of a remote sensing image panchromatic sharpening method based on a denoising diffusion probability model, as provided in an embodiment of this specification. Figure 1 As shown, the method includes the following steps:

[0042] Step S110: Perform differential processing on the high-resolution multispectral HRMS image and the multispectral IMS image interpolated based on the low-resolution multispectral LRMS image to obtain the original differential DM image.

[0043] It should be understood that difference processing and interpolation processing can be implemented using existing related methods, which will not be elaborated upon here.

[0044] Step S120: The original difference image is forward diffused using a denoising diffusion probability model to obtain the corresponding noise image.

[0045] For example, this step includes: performing image noise processing at each time step based on the original difference image to obtain a noisy image corresponding to each time step, wherein the last noisy image is used as the noise image.

[0046] It should be understood that the noisy image corresponding to each time step is actually the hidden state at each time step, or the forward hidden state; in addition, the forward diffusion process does not involve the model parameters that need to be trained.

[0047] Step S130: Use a feature extraction model to extract key features from the panchromatic PAN image and the low-resolution multispectral LRMS image.

[0048] In some embodiments, the feature extraction model can be implemented using a variety of machine learning algorithms.

[0049] In some embodiments, the feature extraction model includes a convolution module and a spectral information mutual calibration module.

[0050] Therefore, the implementation of this step may include:

[0051] The convolution module is used to process the panchromatic PAN image and the low-resolution multispectral LRMS image respectively to obtain the corresponding PAN features I. PAN and LRMS features I MS .

[0052] Furthermore, using the spectral information mutual calibration module, the global pooling and max global pooling of the panchromatic PAN image and the low-resolution multispectral LRMS image are averaged along the channel dimension to obtain multiple spectral information vectors V. P ; to store the multiple spectral information vectors v p Connected in series to form V c And use a multilayer perceptron (MLP) to process the V c , to obtain the I PAM and I MS Channel calibration weight CC PA and CC MS ; In the channel dimension, the I PAN and I MS respectively with CC PAN and CC MS Perform element-wise multiplication and then add them to themselves to obtain spectral cross-calibration information I′. PAN and I′ MS This is categorized under the key features mentioned above.

[0053] In some embodiments, the feature extraction model further includes a spatial information mutual calibration module, and correspondingly, the implementation of this step may also include:

[0054] Using the spatial information mutual calibration module, the spectral mutual calibration information I′ is... PAN and I′ MS The concatenated vector is then subjected to average global pooling, max global pooling, and 1x1 convolution along the channel dimension, and the results of these three processes are concatenated into V. S ; the V S The information is input into a multilayer perceptron (MLP) to obtain the spectral mutual calibration information I′. PAN and I′ MS Corresponding spatial calibration weights SC PAN and SC MS; in the channel dimension, the I′ PAN and I′ MS respectively with SC PAN and SC MS Perform element-wise multiplication, then add the results to themselves to obtain spatial information cross-calibration information I″. PAN and I″ MS This is categorized under the key features mentioned above.

[0055] Step S140: Using the denoising diffusion probability model, the noisy image is subjected to inverse denoising processing based on the key features to obtain a reconstructed differential DM image.

[0056] In some embodiments, this step includes: performing image denoising processing at each time step based on the noisy image to obtain a denoised image corresponding to each time step, wherein the last denoised image is used as the reconstructed differential DM image.

[0057] It should be understood that the denoised image corresponding to each time step is essentially the hidden state at each time step, or the inverse hidden state; in addition, the model parameters θ that need to be trained are used when performing inverse diffusion processing.

[0058] Step S150: With the goal of making the data distribution of the reconstructed differential DM image approximate the data distribution of the original differential DM image, train the denoising diffusion probability model and the feature extraction model.

[0059] In some embodiments, this step may include: calculating the DL divergence between the reconstructed differential DM image data distribution and the original differential DM image data distribution, and training a denoising diffusion probability model and a feature extraction model with the goal of reducing the DL divergence.

[0060] In some embodiments, this step may include: obtaining the forward hidden state and the reverse hidden state at each time step, and training the denoising diffusion probability model and the feature extraction model with the goal of reducing the forward hidden state and the reverse hidden state corresponding to the same time step.

[0061] Step S160: Add the interpolated multispectral IMS image to the reconstructed DM image to obtain the panchromatic sharpening result for the low-resolution multispectral LRMS image and the panchromatic PAN image.

[0062] Figure 2 This diagram illustrates the overall structure of a remote sensing image panchromatic sharpening model based on a denoising diffusion probability model, as provided in an embodiment of the present invention. Figure 2As shown, the model consists of three parts: a forward diffusion part, a conditional injection and modal mutual calibration module, and an inverse denoising part. The forward diffusion part acquires the hidden states of the original difference image at each time step after adding noise. Then, the conditional injection and modal mutual calibration module extracts key spatial and spectral information from panchromatic (PAN) and low-resolution multispectral (LRMS) data of different modalities, and uses spatial and spectral mutual calibration to guide the inverse denoising process. The inverse denoising part aims to optimize the parameters of the neural network under the guidance of the conditional injection information, completing the process of recovering the difference (DM) image from isotropic Gaussian noise. Finally, the interpolated multispectral (IMS) image is added to obtain the panchromatic sharpening result.

[0063] The following section elaborates on the contents of the three parts mentioned above.

[0064] (I) Forward Diffusion

[0065] The forward diffusion process of the denoising diffusion probability model begins with the prior data distribution x0. In this embodiment, the prior data distribution x0 specifically refers to the difference (DM) image between the high-resolution multispectral (HRMS) image and the interpolated multispectral (IMS) image. Then, a known distribution of Gaussian noise is continuously added to the prior data distribution x0 through a Markov chain process to obtain an approximate standard normal distribution. The Gaussian transition process during the forward diffusion of any adjacent hidden states in a Markov chain is shown in Equation (1):

[0066]

[0067] Where, β t Indicates from x t-1 To x t The variance of the Gaussian noise added during the transition, where all variances satisfy β1, ..., β T ∈[0,1).

[0068] The forward diffusion process is determined by the approximate posterior q(x) 1：T |x0) is given, as shown in the following formula (2):

[0069]

[0070] Substituting formula (1) into formula (2), and after reparameterization, we can determine the relationship between x0 and β. t Derive the hidden state x at any time step t t Data distribution q(x) t Equations (3) and (4) below give the hidden state x. t The derivation of .

[0071]

[0072]

[0073] (II) Reverse Denoising Section

[0074] The purpose of the denoising diffusion probability model is to deduce the Gaussian noise x T In the sampling process, samples from a specific data distribution q(x0) are recreated, which requires an inverse denoising process to learn the parameterized Gaussian transition q(x). t-1 |x t It should be noted that if β t If it is small enough, then q(x) t-1 |x t The distribution q(x) will also be Gaussian. However, the estimated q(x) t-1 |x t It is very difficult; this invention uses model p. θ To approximate these conditional probabilities, we fit the mean and variance. The formula for the inverse denoising Gaussian transition of any adjacent states on the Markov chain is as follows:

[0075]

[0076] Where t∈[1,T], μ θ and Σ θ It is p θ (x t-1 |x t Let the mean and variance of ) be...

[0077] Substituting formulas (1) and (3) into the conditional probability In the mean, and using Bayes' theorem, the mean can be... and variance The parameters are as follows: (6), (7) and (8):

[0078]

[0079] α t =1-β t (7)

[0080]

[0081] The optimization objective of the denoising diffusion probability model is to recreate a sampling distribution. To make it as close as possible to the previous data distribution x0, this can be achieved by minimizing the negative log-likelihood (NLL) and optimized using the variational lower bound of equation (9):

[0082]

[0083] Where D KL (·||·) represents the Kullback-Leibler divergence. and These are fixed values ​​after the data distribution x0 and the noise scheme β are determined. Parameterized Substituting q(x) t-1 |x t (x0) and p θ (x t-1 |x t The mean and variance of () are calculated as follows:

[0084]

[0085] Therefore, the optimization objective simplifies to making the prediction distribution p θ (x t-1 |x t ) to be as close as possible to the posterior distribution q(x) at any time t < 1 t-1 |x t (x0).

[0086] (III) Conditional Injection and Modal Mutual Calibration Module

[0087] Conditional injection: Since there is a lot of uncertainty in the denoising diffusion probability model during the reverse process of reconstructing the initial input x0 from Gaussian noise, panchromatic (PAN) images and low-resolution multispectral (LRMS) images are used as conditional injections to guide the reverse process, as shown in equations (11) and (12) below.

[0088]

[0089] cond=Φ(P,MS) (12)

[0090] Where Φ(·) is the encoder branch used to process panchromatic (PAN) images and low-resolution multispectral (LRMS) images as injection conditions. More specifically, embodiments of the present invention use a modal mutual calibration (MIM) module to combine modally calibrated panchromatic (PAN) and low-resolution multispectral (LRMS) features with x. t They are connected to achieve conditional injection. In one implementation, the MIM module corresponds to the feature extraction model mentioned above.

[0091] Modal Mutual Calibration Module: Significant modal differences (i.e., spectral and spatial differences) exist between panchromatic (PAN) and low-resolution multispectral (LRMS) images. This module allows PAN and LRMS images to guide the q(x) function in the neural network by focusing on different aspects. t-1 |x tModeling of PAN and LRMS images. To explicitly utilize modal variations between PAN and LRMS images to enhance injection conditions, this method proposes a modal mutual calibration module (MIM) that considers mutual calibration of channel and spatial information.

[0092] First, the PAN and IMS images are input into the convolutional module to obtain dimension-specific PAN features. PAN ∈R H×W×c and IMS features MS ∈R H×W×c Next, we will analyze feature I separately. PAN and I MS Modal mutual calibration is performed using the Spectral Information Mutual Calibration Module (MIM-Spectral) and the Spatial Information Mutual Calibration Module (MIM-Spatial). MIM-Spectral and MIM-Spatial are described below.

[0093] Figure 3 This diagram illustrates the structure of the spectral information mutual calibration module provided in an embodiment of the present invention, as shown below. Figure 3 As shown, to calibrate the spectral differences between PAN and LRMS images, we obtain sufficient spectral information vector v by averaging global pooling and max global pooling along the channel dimension. p ∈R c Connect them in series to form Next, a multilayer perceptron (MLP) and a sigmoid function σ(·) are used to obtain the channel calibration weights for the PAN and MS features, referred to as CC, respectively. PAN and CC MS Finally, the PAN and MS features are compared with the CC features at the channel dimension. PAN and CC MS Perform element-wise multiplication and then add the results to themselves to obtain the final spectral cross-calibration information I′. PAN and I′ MS The MIM-Spectral process is shown in equations (13), (14), and (15):

[0094] CC MS CC PAN =σ(MLP(V) C (13)

[0095] I' MS =I MS ×CC MS +I MS (14)

[0096] I' PAN =I PAN ×CC PAN+I PAN (15)

[0097] Figure 4 This diagram illustrates the structure of the spatial information mutual calibration module provided in an embodiment of the present invention. Figure 4 As shown, MIM-Spatial will input the spectral mutual calibration information I′ PAN and I′ MS By concatenating these methods and performing average global pooling and max global pooling along the channel dimension, we can obtain... Compared to MIM-Spectral, MIM-Spatial requires processing more complex modeling information; therefore, this method generates a small amount of key information through 1×1 convolution. Then v p and v 1×1 Connected together, forming This is then input into an MLP, where the spatial calibration weights SC are obtained via the sigmoid function. PAN and SC MS The MIM-Spatial process is shown in equations (16), (17), and (18):

[0098] SC MS SC PAN =σ(MLP(V) S (16)

[0099] I″ MS =I′ MS ×SC MS +I′ MS (17)

[0100] I″ PAN =I′ PAN ×SC PAN +I′ PAN (18)

[0101] The above is a combination Figure 2 The method of the present invention will be further described below through a detailed description of its implementation.

[0102] In one embodiment, the neural network architecture follows the main network architecture of U-Net in the Denoising Diffusion Probabilistic Model (DDPM), and adds an information calibration module in the injection conditions PAN and MS to better handle the relationship between different modal data.

[0103] In one embodiment, to enhance sensitivity to time steps to improve p θ (x t-1|x t To improve the prediction accuracy of (cond), this invention introduces time coding technology. The simplified optimization objective clearly states that... Among all time steps t, the mapping has the most direct impact on model prediction performance. Therefore, this invention... The sinusoidal position embedding is applied as shown in formula (19).

[0104]

[0105] Where d is half the dimension of the temporal embedding, and i = 0, 1, ..., d-1. Through series Temporal embedding features can be obtained TE(t) is currently a non-trainable constant value. To address this issue, this invention introduces two Multi-Layer Perceptron (MLP) layers to obtain a final trainable TE(t).

[0106] The following describes the training and inference of the neural network model.

[0107] In one embodiment, the hidden state transitions for each step are learned via a neural network, and the error is backpropagated to train the model. The parameterization in equation (10) is then simplified by adding conditions. Then, the optimization objective in the following equation is given. As shown in formula (20).

[0108]

[0109] Figure 5 This invention provides panchromatic sharpening results for different land cover types on Gaofen-2 satellite with reduced resolution, as shown in this embodiment. Figure 6 This shows the results of full-color sharpening on a high-resolution CMOS imager provided in an embodiment of the present invention. Figure 7 This shows the full-color sharpening display result on a QuickBird with reduced resolution provided by an embodiment of the present invention; Figure 8 This shows the full-color sharpening display result on a WorldView3 with reduced resolution provided by an embodiment of the present invention; Figure 9 This shows the full-resolution, full-color sharpening display result on Gaofen-2 provided by an embodiment of the present invention; Figure 10 This shows the full-color sharpening display result on a full-resolution QuickBird provided by an embodiment of the present invention; Figure 11 This shows the full-color sharpening display result on a full-resolution WorldView3 provided by an embodiment of the present invention; Figure 12 This shows the generalized full-color sharpening result on a WorldView2 with reduced resolution provided by an embodiment of the present invention.

[0110] The specific algorithm for model training is as follows:

[0111]

[0112] Model sampling:

[0113]

[0114]

[0115] This invention has been validated on the PanCollection dataset. See Tables (1)-(6) below:

[0116] Table 1 shows the quantitative evaluation metrics for all comparative methods on the down-resolution Gaofen-2 dataset.

[0117]

[0118] Table 2 shows the quantitative evaluation metrics for all comparative methods on the down-resolution QuickBird dataset.

[0119]

[0120] Table 3 shows the quantitative evaluation metrics for all comparative methods on the down-resolution WorldView-3 dataset.

[0121]

[0122]

[0123] Table 4 shows the quantitative evaluation metrics for all comparison methods on the full-resolution Gaofen-2 dataset.

[0124]

[0125] Table 5 shows the quantitative evaluation metrics for all comparative methods on the full-resolution QuickBird dataset.

[0126]

[0127] Table 6 shows the quantitative evaluation metrics for all comparison methods on the full-resolution WorldView-3 dataset.

[0128]

[0129]

[0130] Corresponding to the method provided by the present invention, the present invention also provides an apparatus. Figure 13This diagram illustrates the structure of a remote sensing image panchromatic sharpening device based on a denoising diffusion probability model, as provided in an embodiment of this specification. Figure 13 As shown, the device 1300 includes:

[0131] The original difference image acquisition unit 1310 is configured to perform difference processing on the high-resolution multispectral HRMS image and the multispectral IMS image interpolated based on the low-resolution multispectral LRMS image to obtain the original difference DM image. The noisy image acquisition unit 1320 is configured to perform forward diffusion processing on the original difference image using a denoising diffusion probability model to obtain the corresponding noisy image. The feature extraction unit 1330 is configured to extract key features from the panchromatic PAN image and the low-resolution multispectral LRMS image using a feature extraction model. The reconstructed difference image acquisition unit 1340 is configured to perform inverse denoising processing on the noisy image based on the key features using the denoising diffusion probability model to obtain the reconstructed difference DM image. The model training unit 1350 is configured to train the denoising diffusion probability model and the feature extraction model with the goal of making the data distribution of the reconstructed difference DM image approximate the data distribution of the original difference DM image. The sharpening unit 1360 is configured to add the interpolated multispectral IMS image to the reconstructed DM image to obtain a panchromatic sharpening result for the low-resolution multispectral LRMS image and the panchromatic PAN image.

[0132] In one embodiment, the feature extraction model includes a convolution module and a spectral information mutual calibration module; the feature extraction unit 1330 is specifically configured to: use the convolution module to process the panchromatic PAN image and the low-resolution multispectral LRMS image respectively, and obtain the corresponding PAN features I. PAN and LRMS features I MS Using the aforementioned spectral information mutual calibration module, global pooling and max global pooling are averaged on the panchromatic PAN image and the low-resolution multispectral LRMS image along the channel dimension to obtain multiple spectral information vectors V. p ; to store the multiple spectral information vectors v p Connected in series to form V c And use a multilayer perceptron (MLP) to process the V c , to obtain the I PAN and I MS Channel calibration weight CC PAN and CC MS ; In the channel dimension, the I PAN and I MS respectively with CC PAN and CC MS Perform element-wise multiplication and then add them to themselves to obtain spectral cross-calibration information I′. PANand I′ MS This is categorized under the key features mentioned above.

[0133] In one specific embodiment, the feature extraction model further includes a spatial information mutual calibration module; the feature extraction unit 1330 is further configured to: utilize the spatial information mutual calibration module to process the spectral mutual calibration information I′ PAN and I′ MS The concatenated vector is then subjected to average global pooling, max global pooling, and 1x1 convolution along the channel dimension, and the results of these three processes are concatenated into V. S ; the V S The information is input into a multilayer perceptron (MLP) to obtain the spectral mutual calibration information I′. PAN and I′ MS Corresponding spatial calibration weights SC PAN and SC MS ; in the channel dimension, the I′ PAN and I′ MS respectively with SC PAN and SC MS Perform element-wise multiplication and then add them to themselves to obtain spatial cross-calibration information I″. PAN and I″ MS This is categorized under the key features mentioned above.

[0134] In one embodiment, the noise image acquisition unit 1320 is specifically configured to: perform image noise addition processing at each time step based on the original difference image to obtain a noise-added image corresponding to each time step, wherein the last noise-added image is used as the noise image.

[0135] In one embodiment, the reconstructed differential image acquisition unit 1340 is specifically configured to: perform image denoising processing at each time step based on the noisy image to obtain denoised images corresponding to each time step, wherein the last denoised image is used as the reconstructed differential DM image.

[0136] It needs to be explained that, for Figure 13 For a description of the apparatus, see also the description of the aforementioned method.

[0137] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 1 The method described.

[0138] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 1The methods described herein. Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0139] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for panchromatic sharpening of remote sensing images based on a denoising diffusion probability model, characterized in that, The method includes: Differential processing is performed on the high-resolution multispectral HRMS image and the multispectral IMS image interpolated based on the low-resolution multispectral LRMS image to obtain the original differential DM image; The original differential DM image is forward diffused using a denoising diffusion probability model to obtain the corresponding noise image; Key features are extracted from the panchromatic PAN image and the low-resolution multispectral LRMS image using a feature extraction model. The feature extraction model includes a convolution module and a spectral information mutual calibration module. The extraction of key features from the panchromatic PAN image and the low-resolution multispectral LRMS image using the feature extraction model includes: using the spectral information mutual calibration module to average the global pooling and max global pooling of the panchromatic PAN image and the low-resolution multispectral LRMS image along the channel dimension to obtain multiple spectral information vectors. ; to generate multiple spectral information vectors Connected into And use a multilayer perceptron (MLP) to process the above. , to obtain the and Channel calibration weights and In the channel dimension, the aforementioned and respectively with and Element-wise multiplication is performed, and then the results are added to themselves to obtain spectral mutual calibration information. and This is categorized under the aforementioned key features; Using the aforementioned denoising diffusion probability model, the noisy image is subjected to inverse denoising processing based on the aforementioned key features to obtain a reconstructed differential DM image; The denoising diffusion probability model and feature extraction model are trained with the goal of making the data distribution of the reconstructed differential DM image approximate the data distribution of the original differential DM image. The interpolated multispectral IMS image is added to the reconstructed differential DM image to obtain the panchromatic sharpening result for the low-resolution multispectral LRMS image and the panchromatic PAN image.

2. The method according to claim 1, characterized in that, The feature extraction model further includes a spatial information mutual calibration module; the extraction of key features from the panchromatic PAN image and the low-resolution multispectral LRMS image using the feature extraction model also includes: Using the spatial information mutual calibration module, the spectral mutual calibration information is... and The concatenated vector is then subjected to average global pooling, max global pooling, and 1x1 convolution along the channel dimension, and the results of these three processes are concatenated. ; will the Input into a multilayer perceptron (MLP) to obtain mutual calibration information with the spectrum. and Corresponding spatial calibration weights and ; in the channel dimension, the aforementioned and respectively with and Element-wise multiplication is performed, and then the results are added to themselves to obtain spatial cross-calibration information. and This is categorized under the key features mentioned above.

3. The method according to claim 1, characterized in that, The forward diffusion process includes: Based on the original differential DM image, image noise processing is performed at each time step to obtain a noisy image corresponding to each time step, wherein the last noisy image is used as the noise image.

4. The method according to claim 1, characterized in that, The reverse denoising process includes: Based on the noisy image, image denoising is performed at each time step to obtain the denoised image corresponding to each time step, wherein the last denoised image is used as the reconstructed differential DM image.

5. A remote sensing image panchromatic sharpening device based on a denoising diffusion probability model, characterized in that, The device includes: The original difference image acquisition unit is configured to perform differential processing on the high-resolution multispectral HRMS image and the multispectral IMS image interpolated based on the low-resolution multispectral LRMS image to obtain the original difference DM image. The noise image acquisition unit is configured to perform forward diffusion processing on the original differential DM image using a denoising diffusion probability model to obtain the corresponding noise image; The feature extraction unit is configured to extract key features from the panchromatic PAN image and the low-resolution multispectral LRMS image using a feature extraction model; the feature extraction model includes a convolution module and a spectral information mutual calibration module; the feature extraction unit is specifically configured to: process the panchromatic PAN image and the low-resolution multispectral LRMS image respectively using the convolution module to obtain the corresponding PAN features. and LRMS features Using the spectral information mutual calibration module, the global pooling and max global pooling of the panchromatic PAN image and the low-resolution multispectral LRMS image are averaged along the channel dimension to obtain multiple spectral information vectors. ; to generate multiple spectral information vectors Connected into And use a multilayer perceptron (MLP) to process the above. , to obtain the and Channel calibration weights and In the channel dimension, the aforementioned and respectively with and Element-wise multiplication is performed, and then the results are added to themselves to obtain spectral mutual calibration information. and This is categorized under the aforementioned key features; The reconstructed difference image acquisition unit is configured to use the denoising diffusion probability model to perform inverse denoising processing on the noise image based on the key features to obtain a reconstructed difference DM image; The model training unit is configured to train the denoising diffusion probability model and the feature extraction model with the goal of making the data distribution of the reconstructed differential DM image approximate the data distribution of the original differential DM image; The sharpening unit is configured to add the interpolated multispectral IMS image to the reconstructed differential DM image to obtain a panchromatic sharpening result for the low-resolution multispectral LRMS image and the panchromatic PAN image.

6. The apparatus according to claim 5, characterized in that, The feature extraction model further includes a spatial information mutual calibration module; the feature extraction unit is further configured as follows: Using the spatial information mutual calibration module, the spectral mutual calibration information is... and The concatenated vector is then subjected to average global pooling, max global pooling, and 1x1 convolution along the channel dimension, and the results of these three processes are concatenated. ; will the Input into a multilayer perceptron (MLP) to obtain mutual calibration information with the spectrum. and Corresponding spatial calibration weights and ; in the channel dimension, the aforementioned and respectively with and Element-wise multiplication is performed, and then the results are added to themselves to obtain spatial cross-calibration information. and This is categorized under the key features mentioned above.

7. The apparatus according to claim 5, characterized in that, The noise image acquisition unit is specifically configured as follows: Based on the original differential DM image, image noise processing is performed at each time step to obtain a noisy image corresponding to each time step, wherein the last noisy image is used as the noise image.

8. The apparatus according to claim 5, characterized in that, The specific configuration of the reconstructed difference map acquisition unit is as follows: Based on the noisy image, image denoising is performed at each time step to obtain the denoised image corresponding to each time step, wherein the last denoised image is used as the reconstructed differential DM image.

Images