Unsupervised spectral super-resolution method based on physical prior guided diffusion model

Abstract

The present application relates to an unsupervised spectral super-resolution method based on a physical prior guided conditional diffusion model, and belongs to the technical field of remote sensing image processing. The present application solves the problems of low accuracy of existing inference process, inability to ensure the authenticity of satellite spectral recovery, and often neglecting the inherent physical properties of remote sensing data, and easily leading to spectral distortion. The process is as follows: 1: construct a multi-hyperspectral data pair, which serves as a training set; 2: construct a physical prior guided conditional diffusion model; 3: train the physical prior guided conditional diffusion model based on the training set to obtain a trained physical prior guided conditional diffusion model; 4: obtain a multi-spectral image with high spatial resolution to be measured, and the trained physical prior guided conditional diffusion model outputs high spatial resolution hyperspectral data.

Description

Technical Field

[0001] This invention relates to an unsupervised spectral super-resolution method based on a physical prior-guided conditional diffusion model, belonging to the field of remote sensing image processing technology. Background Technology

[0002] Hyperspectral images contain both spatial ground feature information and rich spectral information, possessing high spectral resolution. They can capture subtle differences in reflectivity between different and similar ground features, thereby improving the ability to identify different ground features. However, constrained by the law of conservation of sensor energy and hardware limitations, spaceborne hyperspectral imaging equipment faces challenges such as low spatial resolution, narrow swath width, and weak revisit capability while pursuing high spectral resolution. Multispectral images have much lower spectral resolution than hyperspectral images, so they only have preliminary feature recognition and classification capabilities. However, they possess high spatial resolution and a large swath width, as well as a short revisit period. Hyperspectral and multispectral images each have their own advantages and disadvantages in terms of temporal, spatial, spectral resolution, and imaging swath width. Therefore, utilizing the complementary advantages of multispectral and hyperspectral images to reconstruct hyperspectral images with wide swath width, high revisit frequency, and high spatial resolution from multispectral images—i.e., spectral super-resolution—has become a major research direction.

[0003] The goal of spectral super-resolution is to improve the spectral resolution of multispectral images to match that of hyperspectral images by introducing spectral information. This is an ill-posed inverse problem of solving the spectral degradation process. Without sufficient spatial-spectral constraints as input, a single multispectral image may correspond to multiple possible high-spatial-resolution hyperspectral solution spaces. Therefore, the core challenge is how to accurately fit the complex nonlinear mapping relationship between the spectra of multispectral and hyperspectral images and effectively improve the physical accuracy and reliability of the reconstructed spectrum. Dictionary learning based on physical models and data-driven deep learning methods are currently two mainstream spectral super-resolution methods. Dictionary learning methods comprehensively characterize the spectral information of the visible and near-infrared regions (400-1000 nm) and the short-wave infrared region (1000-2500 nm) through linear mixture equations. However, it only models ideal imaging conditions and struggles to capture nonlinear effects such as shadows and multiple scattering in real imaging processes. It also struggles to model spatial context information and the deep semantic information of the image. Deep learning methods learn the nonlinear mapping relationship between multispectral and hyperspectral data end-to-end using a large amount of multispectral and hyperspectral data. Since diffusion models, as generative models, probabilistically model the uncertainty of generated spectra, they can provide multiple possible solutions during training, avoiding the optimization process from getting trapped in local optima. Therefore, diffusion models have become a cutting-edge method in the field of spectral super-resolution. Although deep learning methods based on diffusion models have strong nonlinear relationship modeling capabilities and often achieve higher reconstruction metrics in applications, they are essentially still data-driven and lack a physical theoretical foundation. When faced with remote sensing data that heavily relies on precise physical information, the reliability of the reconstruction results is not high. Furthermore, since the imaging spectral range of multispectral cameras is 400-1000 nm, while the usable wavelength range of hyperspectral sensors is 400-2500 nm, deep learning methods lack corresponding training multispectral data in the 1000-2500 nm band as spectral guidance. This may lead to spectral curves that violate physical principles due to an excessive pursuit of reconstruction metrics. Summary of the Invention

[0004] This invention addresses the problems of existing diffusion model-based spectral super-resolution methods, which typically rely on real high spatial resolution hyperspectral images from the ground for training, leading to low accuracy when extended to ground-valued real-world imaging scenarios during inference; the lack of design for covering the entire 400-2500nm band in current mainstream methods, failing to ensure the authenticity of satellite spectra recovery in the 1000-2500nm region and often ignoring the inherent physical characteristics of remote sensing data; and the tendency to cause spectral distortion by using only data amounts such as MSE as end-to-end constraints. Therefore, this invention proposes an unsupervised spectral super-resolution method based on a physically prior-guided diffusion model.

[0005] The specific process of the unsupervised spectral super-resolution method based on the physical prior guided conditional diffusion model is as follows:

[0006] Step 1: Obtain real hyperspectral data, and use the spectral response function to perform spectral downsampling on the real hyperspectral data to obtain multispectral data;

[0007] Multispectral-hyperspectral data pairs are constructed based on multispectral data and real hyperspectral data, and these multispectral-hyperspectral data pairs are used as the training set;

[0008] Step 2: Construct a diffusion model based on physical prior guidance conditions;

[0009] The physical prior-guided conditional diffusion model includes a mask-dictionary-guided spectral diffusion prior module, a physical condition encoding and conditional fusion module, and a space-spectral cooperative conditional diffusion module based on the U-net structure.

[0010] The mask-dictionary guided spectral diffusion prior module includes a noise addition process and a noise reduction process;

[0011] The physical condition encoding and condition fusion module includes an SRF encoder, an MSI encoder, an attention fusion module, and a condition fusion module;

[0012] The SRF encoder comprises, in sequence, a flattening layer, a fully connected layer, a ReLU activation function layer, and another fully connected layer;

[0013] The MSI encoder includes, in sequence, the following: Convolutional layer, residual module 1, residual module 2, residual module 3;

[0014] Residual module 1 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer;

[0015] Residual module 2 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer;

[0016] Residual module 3 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer;

[0017] The attention fusion module is a channel attention module;

[0018] The condition fusion module includes Depthwise separable convolutional layer, first linear layer, second linear layer, third linear layer, Convolutional layers, GELU activation function layers, Convolutional layers, Sigmoid activation function layers, weighted fusion Convolutional layer;

[0019] The space-spectral coordinated conditional diffusion module based on the U-net structure includes a noise addition process and a noise removal process;

[0020] The denoising process includes a first encoder, a first conditional injection module, a second encoder, a second conditional injection module, a third encoder, a third conditional injection module, a bottleneck layer, a first decoder, a fourth conditional injection module, a second decoder, a fifth conditional injection module, and a two-dimensional convolutional layer;

[0021] The bottleneck layer includes spatial attention mechanisms, spectral attention mechanisms, Convolutional layer;

[0022] Step 3: Train the physical prior guided conditional diffusion model based on the training set until the loss function of the physical prior guided conditional diffusion model converges, and obtain the trained physical prior guided conditional diffusion model.

[0023] Step 4: Acquire the high spatial resolution multispectral image to be tested, input the high spatial resolution multispectral image to be tested into the trained physical prior guided conditional diffusion model, and the trained physical prior guided conditional diffusion model outputs high spatial resolution hyperspectral data.

[0024] The beneficial effects of this invention are as follows:

[0025] This invention proposes a spectral super-resolution method for unsupervised spectral super-resolution tasks in the 400-2500 nm wavelength range, guided by mask-dictionary spectral priors and physical encoding. The method consists of three modules: a mask-dictionary-constrained spectral diffusion prior module, a physical conditional encoding and conditional fusion module, and a multi-level conditional injection and diffusion module based on U-net. The innovation lies in the random masking of the spectral response function in the prior module to train the model's spectral extrapolation ability. Simultaneously, a hyperspectral dictionary is used to extract deep sparse features within the 400-2500 nm range. The consistency loss of the sparse coefficients is used as the optimization objective to address the problem that the spectral response function and the input multispectral image cannot fully cover the 400-2500 nm wavelength range, leading to insufficient recovery of the short-wave infrared region, and the over-smoothing problem caused by using only MSE loss for optimization. Subsequently, the physical conditional encoder and conditional fusion module, through precise physical information encoding and fusion, can comprehensively extract spatial-spectral prior information, further improving the fidelity of spectral reconstruction. The U-net-based multi-level conditional injection and conditional diffusion module deeply fits spatial correlation information and spectral cross-correlation relationships through multi-level physical mechanism embedding and spatial-spectral attention mechanisms, thereby gradually recovering high-fidelity spectral information through the diffusion process. Its beneficial effects are:

[0026] From a methodological perspective, this invention aims to address three major problems in high spatial resolution hyperspectral image reconstruction methods based on diffusion models: the need for supervised images, inaccurate reconstruction in the short-wave infrared region, and weak physical consistency of the reconstructed image. 1) This method is specifically designed for real satellite remote sensing images. Because it learns a pure spectral mapping independent of spatial resolution, it is suitable for unsupervised spectral super-resolution tasks. 2) This method aims to achieve accurate spectral recovery in the short-wave infrared 1000–2500 nm region. 3) This method fully utilizes the spatial features of the input multispectral image and the prior sensor spectral response as diffusion condition inputs, further enhancing the physical consistency of the reconstructed image. Compared with traditional dictionary learning methods and deep learning methods based on non-generative models, this method combines physical priors with data-driven learning, accurately capturing complex nonlinear mapping relationships across the entire near-infrared and short-infrared spectral range, and explicitly modeling spectral uncertainties through the diffusion process. This achieves an effective balance between physical fidelity and data consistency in the generated results. Compared with advanced conditional diffusion spectral reconstruction methods, this method is innovatively and specifically designed for scenarios with real satellite remote sensing images but without supervised high-resolution hyperspectral image references. Unlike image fusion-based methods, this method can accurately reconstruct high-resolution hyperspectral images from large-scale high-resolution multispectral images using only small-scale low-resolution hyperspectral images and spectral response functions, and innovatively achieves highly consistent spectral reconstruction in the spectral range of 400-2500 nm.

[0027] From an application perspective, this invention can effectively address the problem that the limited amount and swath width of existing satellite hyperspectral data restrict the practical application of spectral super-resolution tasks in multiple regions. It provides a large and widely covered source of high spatial resolution hyperspectral data for various downstream tasks in the field of remote sensing. Furthermore, this invention can significantly improve the physical consistency of reconstructed images and has extremely high classification accuracy in land cover classification tasks compared to existing methods, thus possessing great practical application value. Attached Figure Description

[0028] Figure 1 This is an overall flowchart of the present invention;

[0029] Figure 2 The specific model structure diagram constructed for this invention is as follows:

[0030] Figure 3 Here is a detailed structural diagram of the denoising network in the space-spectral cooperative conditional diffusion module based on the U-net structure constructed in this invention:

[0031] Figure 4 Here is a detailed structural diagram of the conditional injection module in the denoising network constructed in this invention:

[0032] Figure 5 Here is a detailed structural diagram of the MSI encoder in the physical condition coding and condition fusion module constructed in this invention:

[0033] Figure 6 Here is a detailed structural diagram of the SRF encoder in the physical condition coding and condition fusion module constructed in this invention:

[0034] Figure 7 This is a structural diagram of the condition fusion module in the physical condition encoding and condition fusion module constructed in this invention. Detailed Implementation

[0035] Specific Implementation Method 1: The specific process of this implementation method based on the physical prior guided conditional diffusion model for unsupervised spectral super-resolution is as follows:

[0036] Step 1: Obtain real low spatial resolution hyperspectral data, and use the spectral response function to perform spectral downsampling on the real low spatial resolution hyperspectral data to obtain low spatial resolution multispectral data;

[0037] Multi-hyperspectral data pairs are constructed based on low spatial resolution multispectral data and real low spatial resolution hyperspectral data, and these multi-hyperspectral data pairs are used as training sets.

[0038] Step 2: Construct a diffusion model based on physical prior guidance conditions;

[0039] The physical prior-guided conditional diffusion model includes a mask-dictionary-guided spectral diffusion prior module, a physical condition encoding and conditional fusion module, and a space-spectral cooperative conditional diffusion module based on the U-net structure.

[0040] The mask-dictionary guided spectral diffusion prior module includes a noise addition process and a noise reduction process;

[0041] The physical condition encoding and condition fusion module includes an SRF encoder, an MSI encoder, an attention fusion module, and a condition fusion module;

[0042] The SRF encoder comprises, in sequence, a flattening layer, a fully connected layer, a ReLU activation function layer, and another fully connected layer;

[0043] The MSI encoder includes, in sequence, the following: Convolutional layer, residual module 1, residual module 2, residual module 3;

[0044] Residual module 1 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer;

[0045] Residual module 2 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer;

[0046] Residual module 3 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer;

[0047] The attention fusion module is a channel attention module;

[0048] The condition fusion module includes Depthwise separable convolutional layer, first linear layer, second linear layer, third linear layer, Convolutional layers, GELU activation function layers, Convolutional layers, Sigmoid activation function layers, weighted fusion Convolutional layer;

[0049] The space-spectral coordinated conditional diffusion module based on the U-net structure includes a noise addition process and a noise removal process;

[0050] The denoising process includes a first encoder, a first conditional injection module, a second encoder, a second conditional injection module, a third encoder, a third conditional injection module, a bottleneck layer, a first decoder, a fourth conditional injection module, a second decoder, a fifth conditional injection module, and a two-dimensional convolutional layer;

[0051] The bottleneck layer includes spatial attention mechanisms, spectral attention mechanisms, Convolutional layer;

[0052] Step 3: Train the physical prior guided conditional diffusion model based on the training set until the loss function of the physical prior guided conditional diffusion model converges, and obtain the trained physical prior guided conditional diffusion model.

[0053] Step 4: Acquire the high spatial resolution multispectral image to be tested, input the high spatial resolution multispectral image to be tested into the trained physical prior guided conditional diffusion model, and the trained physical prior guided conditional diffusion model outputs high spatial resolution hyperspectral data.

[0054] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that: Step 1: Obtain real low spatial resolution hyperspectral data, and use the spectral response function to perform spectral downsampling on the real low spatial resolution hyperspectral data to obtain low spatial resolution multispectral data;

[0055] Multi-hyperspectral data pairs are constructed based on low spatial resolution multispectral data and real low spatial resolution hyperspectral data, and these multi-hyperspectral data pairs are used as training sets.

[0056] (1)

[0057] in, This refers to low spatial resolution hyperspectral data (image data obtained by flattening three-dimensional low spatial resolution hyperspectral data into two dimensions). Represents the spectral response function. This represents low spatial resolution multispectral image data.

[0058] The other steps and parameters are the same as in Specific Implementation Method 1.

[0059] Specific Implementation Method 3: This implementation method differs from Specific Implementation Method 1 or 2 in that: Step 3: The physical prior guided conditional diffusion model is trained based on the training set until the loss function of the physical prior guided conditional diffusion model converges, and the trained physical prior guided conditional diffusion model is obtained.

[0060] The specific process is as follows:

[0061] Step 31: Based on real low spatial resolution hyperspectral data in the training set, extract semantic features of the spectral dimension pixel by pixel using the alternating direction multiplier method, and construct a complete hyperspectral dictionary. ;

[0062] Step 32: Combine the low spatial resolution multi-hyperspectral data pairs, spectral response functions, and the pre-trained hyperspectral dictionary from the training set. Input a mask-dictionary-guided spectral diffusion prior module, and the mask-dictionary-guided spectral diffusion prior module outputs the corresponding pixel-by-pixel spectral prior prediction result. Train the mask-dictionary-guided spectral diffusion prior module until the loss function of the mask-dictionary-guided spectral diffusion prior module converges, and obtain the pre-trained mask-dictionary-guided spectral diffusion prior module.

[0063] Step 33: Construct a physical prior guidance model;

[0064] The pixel-wise spectral prior prediction results are output based on low spatial resolution multi-hyperspectral data pairs in the training set and a pre-trained mask-dictionary guided spectral diffusion prior module. The physical prior model is trained until its loss function converges, thus obtaining a pre-trained physical prior model.

[0065] Other steps and parameters are the same as in specific implementation method one or two.

[0066] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in step 31: based on real low spatial resolution hyperspectral data in the training set, semantic features of the pixel-by-pixel spectral dimension are extracted using the alternating direction multiplier method, thus constructing a complete hyperspectral dictionary. ;

[0067] The specific process is as follows:

[0068] Step 311: Let This represents the hyperspectral dictionary to be learned. Sparse representation coefficients for low spatial resolution hyperspectral images;

[0069] Step 312: Based on the hyperspectral dictionary to be learned Sparse representation coefficients of low spatial resolution hyperspectral images The low spatial resolution hyperspectral image and low spatial resolution multispectral image data pairs constructed in step 1 are represented as follows:

[0070] (2)

[0071] Step 313: Construct the optimization model, represented as:

[0072] (3)

[0073] in, Denotes the square of the F-norm; This represents the weighting coefficient of the multispectral dictionary constraint term, which was set to 0.5 in the experiment; The weights of the L1 sparse regularization term are used to control the sparsity coefficients. The sparsity was set to 0.1 in the experiment; Represents the 1-norm;

[0074] The first term is the full image reconstruction constraint, the second term is the spectral response function consistency constraint, and the third term is the sparsity constraint of the sparse coefficients.

[0075] Step 314: To solve this optimization equation, let Separate the first and second terms, and let the third term in equation (4) be... As shown in formula (4):

[0076] (4)

[0077] in, Indicates parameters;

[0078] Step 315: Add to the optimization model As a penalty term, construct the augmented Lagrangian function. As shown in equation (5):

[0079] (5)

[0080] in, This represents the penalty term coefficient, which measures the model's performance on the given conditions. and The rigor of the approximation degree; Indicates parameters;

[0081] Step 316: Based on the augmented Lagrangian function, and combined with the alternating direction multiplier method for each pixel, extract the semantic features of the spectral dimension for each pixel, and construct a complete hyperspectral dictionary;

[0082] The specific process is as follows:

[0083] Step 3161: Randomly initialize the dictionary using a Gaussian distribution. ;

[0084] Initialize sparse coefficients ,parameter ,parameter It is a zero matrix;

[0085] Step 3162: Set the maximum number of rounds , Set the maximum number of iterations. , ;

[0086] Step 3163: Order of Rounds Let the number of iterations be... ;

[0087] It is a hyperspectral dictionary constructed from the spectral curve of each pixel, and solved pixel by pixel using alternating ADMM to achieve the dictionary. The optimization process, in which each atom undergoes... Normalization is performed to ensure that its norm is 1;

[0088] Step 3164: Sparsity Coefficient The optimization method is shown in formula (6):

[0089] (6)

[0090] in, Indicates the number of iterations The sparsity coefficient at time, Indicates the number of iterations Parameters at time; Indicates the number of iterations Parameters at time;

[0091] make The derivative is 0, so we can solve for... The update method is as follows:

[0092] (7)

[0093] in, This represents the identity matrix, with the superscript T indicating transpose.

[0094] Step 3165: Parameters The optimization formula is shown in formula (8):

[0095] (8)

[0096] in, Indicates the number of iterations Parameters at time;

[0097] Solving equation (8) using the soft threshold operator yields the following result. The update method is as follows:

[0098] (9)

[0099] in,

[0100] This represents the soft threshold operator. );

[0101] Step 3166: Parameters The optimization formula is shown in formula (10):

[0102] (10)

[0103] in, Indicates the number of iterations Parameters at time;

[0104] Step 3167: Determine the number of iterations Is it less than ;

[0105] If it is less than, let the number of iterations... Repeat steps 3164 to 3167 until... The number of iterations obtained sparsity coefficient at time As Execute step 3168;

[0106] Otherwise, the number of iterations obtained sparsity coefficient at time As Execute step 3168;

[0107] Step 3168: Adjust the sparsity coefficients and the hyperspectral dictionary to be optimized Substitute into formula (11) to update the hyperspectral dictionary. ;

[0108] Represented as:

[0109] (11)

[0110] in, Indicates round Hyperspectral dictionary ;

[0111] make With the derivative being 0, the closed-form solution of formula (11) is obtained, as shown in formulas (12) and (13):

[0112] (12)

[0113] (13)

[0114] in, Represents the current round (number) The round is based on the optimal solution of the data fit, rather than the dictionary of the final output;

[0115] To prevent Singularity, add a regularization term to formula (13) To obtain intermediate variables and hyperspectral dictionary The update process is shown in (14) and (15):

[0116] (14)

[0117] (15)

[0118] in, Represents the regularization parameter; Represents the identity matrix; This represents the relaxation factor. To improve the algorithm's robustness and reduce iterative oscillations, the dictionary from the previous round is used. For intermediate variables Perform weighted smoothing;

[0119] Step 3169: Let Repeat steps 3164 to 3169 until... This yields a complete hyperspectral dictionary.

[0120] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0121] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that step 32 involves combining the low spatial resolution multi-hyperspectral data pairs, spectral response functions, and the pre-trained hyperspectral dictionary from the training set. Input a mask-dictionary-guided spectral diffusion prior module, and the mask-dictionary-guided spectral diffusion prior module outputs the corresponding pixel-by-pixel spectral prior prediction result. Train the mask-dictionary-guided spectral diffusion prior module until the loss function of the mask-dictionary-guided spectral diffusion prior module converges, and obtain the pre-trained mask-dictionary-guided spectral diffusion prior module.

[0122] The specific process is as follows:

[0123] Step 321: Response to the spectral response function After implementing a 20% random masking strategy, the features are encoded using an MLP. The process is as follows:

[0124] Spectral response function When 20% of the elements in the matrix are set to 0, the transformed spectral response function... Input to a Multilayer Perceptron (MLP), output features of the Multilayer Perceptron (MLP) ;

[0125] Step 322: Input the low spatial resolution hyperspectral image into a two-dimensional convolutional layer, and the two-dimensional convolutional layer outputs spatial features. ;

[0126] Features and characteristics By adding elements one by one, we can obtain the features. (As input conditions for the spectral diffusion prior module);

[0127] Step 323: For the conditional diffusion model in the spectral dimension, use the currently captured real reference low spatial resolution hyperspectral image. In this process, spectral curves are extracted pixel-by-pixel from low spatial resolution hyperspectral images. The extracted spectral curves are then used... express;

[0128] The forward process of the spectral diffusion prior model is progressively advanced to each time step. By adding Gaussian noise of a certain intensity, the ideal Gaussian noise spectrum curve can be achieved. By deriving each time step The noise addition process is derived in time. Intermediate spectral curve after adding noise As shown in formulas (16) to (18):

[0129] (16)

[0130] (17)

[0131] (18)

[0132] in, Indicates from State noise added State transition probabilities, This represents the spectrum without added noise (because of the time step). It starts from 1, and also from... (Start adding noise) Indicates time step The spectrum after adding noise Indicates time step The spectrum after adding noise Indicates time step The noise scheduling parameters control the proportion of information retained from the previous moment in each step; Represents the identity matrix. Indicates the first One time step; express Depends on Gaussian conditional probability;

[0133] This indicates that the mean is 0 and the variance is... Gaussian noise, Indicates at time step Added obedience Standard Gaussian noise, This indicates that the mean is 0 and the variance is... Gaussian noise;

[0134] Represents noise scheduling parameters The cumulative product, ; Indicates time step Noise scheduling parameters;

[0135] Indicates starting from the origin To the finish line The final equivalent result of all accumulated noise during the process, This indicates traversing from time step 1 to time step 2. Intermediate variables at each time step between them ;

[0136] The inverse denoising process of the spectral diffusion prior model from the pure Gaussian noise spectral curve Begin by gradually using probability Sampling allows for the reconstruction of the spectral structure of the data, ultimately yielding the target spectral curve. The specific process is as follows:

[0137] pure Gaussian noise spectrum The input is fed into the spectral self-attention mechanism, and the spectral self-attention mechanism outputs features. ;

[0138] feature and the features obtained in step 321 Add them together and add time-step embeddings to obtain the features. ;

[0139] feature The target spectral curve is obtained by sequentially passing the data through a three-layer MLP network. This achieves denoising through conditional diffusion;

[0140] Represented as:

[0141] (19)

[0142] in, For denoising networks (including removing pure Gaussian noise from the spectrum) The input is fed into the spectral self-attention mechanism, and the spectral self-attention mechanism outputs features. ;feature and the features obtained in step 31 Add them together and add time-step embeddings to obtain the features. ;feature The target spectral curve is obtained by sequentially passing the data through a three-layer MLP network. ); Denoising Network The goal is to optimize and obtain a pure pixel-by-pixel spectral curve of the target;

[0143] In the mask-dictionary constrained spectral prior module, a denoising network is used. To predict probability distribution To fit the true posterior probability distribution ;

[0144] The true posterior distribution can be obtained using formula (20):

[0145] (20)

[0146] in, express arrive The actual transition probability during the process;

[0147] express Obey For the mean, The variance is a Gaussian distribution. It represents the mean of the probability distribution. Represents the variance of the probability distribution;

[0148] The mean and variance of the true posterior probability distribution can be calculated using formulas (21) and (22):

[0149] (twenty one)

[0150] (twenty two)

[0151] in, Indicates noise scheduling parameters The cumulative product, ;

[0152] Therefore, in the proposed research scheme, after using a network to predict the probability distribution to fit the true probability distribution, from... arrive Denoising process It is expressed as follows:

[0153] (twenty three)

[0154] in, This represents the mean of Gaussian noise predicted by the network and used for backsampling. This indicates the level of noise added in each denoising process and is set in the model. To achieve a deterministic sampling process; Represents the variance of the probability distribution; Indicates the predicted spectral curve;

[0155] Use a linear scheduling strategy in the program to adjust The value. (To be) The value increases linearly from 0.0001 to 0.02 to balance sampling quality and running speed;

[0156] Step 324: Construct the loss function for the mask-dictionary-guided spectral diffusion prior module; expressed as:

[0157]

[0158] in,

[0159] First loss The second term represents the overall difference constraint between the original low spatial resolution hyperspectral image and the reconstructed low spatial resolution hyperspectral image. The third term represents the MSE-weighted error of the 20% spectral response function mask region. The fourth term represents the consistency constraint of the spectral response function. Representing dictionary sparsity constraints, the fifth loss term Indicates spatial texture feature constraints;

[0160] and These represent the original spectral curve and the predicted spectral curve, respectively.

[0161] This represents the spectral values ​​of the original spectral curve within the preset mask region (20% region) of the spectral response function; This represents the spectral value of the predicted spectral curve within the preset mask region (20% region) of the spectral response function; both are obtained by traversing the preset SRF mask bands and extracting the spectral values ​​of the corresponding bands. Represents the square of the 2-norm;

[0162] This represents the spectral response value after the original spectral curve has been sampled using the spectral response function (SRF). This represents the spectral response value after the predicted spectral curve is downsampled using the spectral response function (SRF); the response value can be obtained by performing matrix multiplication between the spectral curve and the spectral response function matrix.

[0163] The true value of the coefficients indicates that the spectrum is sparse; The predicted values ​​of the spectral sparsity coefficients are indicated.

[0164] This represents the spatial gradient of the original spectral curve; This represents the spatial gradient of the reconstructed spectral curve; both are obtained by performing a first-order difference operation on the spatial dimensions (horizontal and vertical directions) of the spectral image, and are used to extract the spatial edge and texture information of the image.

[0165] The second term applies additional constraint weights to the regions of the spectral response mask to enhance the model's generalization ability and predictive power for unknown bands. The spectral response function consistency constraint ensures the consistency between the reconstructed low-spatial-resolution hyperspectral image after spectral downsampling and the input low-spatial-resolution multispectral image. Furthermore, the dictionary sparse coefficient constraint is optimized using the alternating direction multiplier method to sparsely represent the original and reconstructed spectra, and utilizes a pre-trained hyperspectral dictionary to constrain their sparse coefficients to maintain consistency, thereby enhancing the model's physical interpretability and its ability to fit the true spectral distribution.

[0166] The spectral sparsity representation coefficients are calculated as follows:

[0167]

[0168] in, This represents the weight coefficient of the regularization term, which is set to 0.1 in the experiment; Represents the 1-norm; Represents the square of the 2-norm;

[0169] Step 325: Train the mask-dictionary guided spectral diffusion prior module until the loss function of the mask-dictionary guided spectral diffusion prior module converges, and obtain the pre-trained mask-dictionary guided spectral diffusion prior module.

[0170] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0171] Specific Implementation Method Six: This implementation method differs from one of Specific Implementation Methods One to Five in that step 33: constructing a physical prior guidance model;

[0172] The pixel-wise spectral prior prediction results are output based on low spatial resolution multi-hyperspectral data pairs in the training set and a pre-trained mask-dictionary guided spectral diffusion prior module. Train the physics prior model until the loss function of the physics prior model converges to obtain the pre-trained physics prior model.

[0173] The specific process is as follows:

[0174] Step 331: Input the low spatial resolution multispectral images and spectral response functions from the training set into the pre-trained mask-dictionary guided spectral diffusion prior module. The pre-trained mask-dictionary guided spectral diffusion prior module outputs pixel-by-pixel spectral prior prediction results. ;

[0175] Step 332: Output pixel-by-pixel spectral prior prediction results from the low spatial resolution multispectral images, spectral response functions, and the pre-trained mask-dictionary guided spectral diffusion prior module in the training set. Input the physical condition encoding and condition fusion module, and the physical condition encoding and condition fusion module outputs the fused result;

[0176] Step 333: Input the low spatial resolution multispectral image, spectral response function, and physical condition encoding and conditional fusion module output from the training set into the U-net-based spatial-spectral co-conditional diffusion module. The U-net-based spatial-spectral co-conditional diffusion module outputs the predicted low spatial resolution hyperspectral image.

[0177] Step 334: Repeat steps 331 to 334 to train the physical prior guided model until the loss function of the physical prior guided model converges, and obtain the pre-trained physical prior guided model.

[0178] Therefore, through the above steps, it is possible to achieve the following: during the model training process, only a small range of real-world low-spatial-resolution hyperspectral images are required as input; during the testing process, combined with the already trained model, only a large range of real high-spatial-resolution multispectral images and the fixed feature of sensor spectral response are needed as input to reconstruct a large range of high-spatial-resolution hyperspectral images. This greatly addresses the problem of insufficient hyperspectral data and small swath width leading to difficulties in spectral super-resolution of large-scene high-spatial-resolution multispectral images. Furthermore, through innovative model design, high-precision reconstruction of the spectrum in the 400-2500nm range is achieved, especially in the short-wave infrared 1000-2500nm range, which adds strong physical constraints to the model and fully ensures the spectral consistency and physical rationality of the generated images.

[0179] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0180] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One to Six in that step 331 involves inputting the low spatial resolution multispectral images and spectral response functions from the training set into a pre-trained mask-dictionary-guided spectral diffusion prior module, and the pre-trained mask-dictionary-guided spectral diffusion prior module outputting pixel-by-pixel spectral prior prediction results. ;

[0181] The specific process is as follows:

[0182] First, execute steps 321 and 322 (step 321 originally had a 20% mask, but this time the mask is not applied, and step 322 remains unchanged) to obtain the features. ;

[0183] Then from the pure Gaussian noise spectrum curve Starting from this point, we directly execute formula (23) in step 323 to gradually denoise the data, and finally obtain the pixel-by-pixel spectral prior prediction results. .

[0184] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0185] Specific Implementation Method Eight: This implementation method differs from any of Specific Implementation Methods One to Seven in that step 332 involves outputting pixel-by-pixel spectral prior prediction results from the low spatial resolution multispectral images, spectral response functions, and the pre-trained mask-dictionary-guided spectral diffusion prior module in the training set. Input the physical condition encoding and condition fusion module, and the physical condition encoding and condition fusion module outputs the fused result;

[0186] The specific process is as follows:

[0187] Step 3321: Input the spectral response functions from the training set into the SRF encoder, and the SRF encoder outputs the features. ;

[0188] The specific process is as follows:

[0189] The spectral response functions from the training set are sequentially input into a flattening layer, a fully connected layer, a ReLU activation function layer, and another fully connected layer. The fully connected layer outputs the features. ;

[0190] Step 3322: Input the low spatial resolution multispectral images from the training set into the MSI encoder. The MSI encoder outputs features. ;

[0191] The specific process is as follows:

[0192] Input the low spatial resolution multispectral images from the training set Convolutional layer Convolutional layer output features ;

[0193] Will Convolutional layer output features Input residual module 1, output residual module 1 features The specific process is as follows:

[0194] Will Convolutional layer output features Enter in sequence Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, batch normalization layer output features ;

[0195] Output features of batch normalization layer and Convolutional layer output features By adding elements one by one, we can obtain the features. ; Features Input ReLU activation function layer, output features of ReLU activation function layer ;

[0196] Features Input residual module 2, output residual module 2 features The specific process is as follows:

[0197] Features Enter in sequence Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, batch normalization layer output features ;

[0198] Output features of batch normalization layer With features By adding elements one by one, we can obtain the features. ;

[0199] Features Input ReLU activation function layer, output features of ReLU activation function layer ;

[0200] Features Input residual module 3, output residual module 3 features ,feature As an output feature of the MSI encoder ;

[0201] The specific process is as follows:

[0202] Features Enter in sequence Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, batch normalization layer output features ;

[0203] Output features of batch normalization layer With features By adding elements one by one, we can obtain the features. ;

[0204] Features Input ReLU activation function layer, output features of ReLU activation function layer ;

[0205] feature As an output feature of the MSI encoder ;

[0206] Step 3323: Output features from the SRF encoder and MSI encoder output characteristics Input to the attention fusion module, output features from the attention fusion module. The specific process is as follows:

[0207] SRF encoder output features and MSI encoder output characteristics Input channel attention module, output features of channel attention module ;

[0208] Step 3324: Add features The pre-trained mask-dictionary-guided spectral diffusion prior module outputs pixel-by-pixel spectral prior predictions. Input the conditional fusion module, and the conditional fusion module outputs the fused result;

[0209] The specific process is as follows:

[0210] Step 33241: Output pixel-by-pixel spectral prior prediction results from the pre-trained mask-dictionary guided spectral diffusion prior module. enter Depth-separable convolutional layers The output features of the depthwise separable convolutional layer are input into the first linear layer (linear) and the second linear layer (linear), respectively. The first linear layer (linear) outputs the features. The second linear layer output features ;

[0211] Step 33242: Add features Input to the third linear layer (linear), the third linear layer (linear) outputs features. ;

[0212] Step 33243: Features and characteristics Element-wise multiplication yields the characteristics. ;

[0213] Step 33244: Features and characteristics Element-wise multiplication yields the characteristics. ;

[0214] Step 33245: Add features and characteristics By splicing the data, we can obtain the features. ;

[0215] Step 33246: Add features Enter in sequence Convolutional layers, GELU activation function layers, Convolutional layer, Sigmoid activation function layer, Sigmoid activation function layer output features ;

[0216] Step 33247: Features and characteristics Perform weighted fusion to obtain the fused features. ;

[0217] Step 33248: Merge the features enter Convolutional layer Convolutional layer output features ;

[0218] Step 33249: Add features and characteristics Element-by-element addition is performed to obtain the fused features output by the conditional fusion module. ;

[0219] feature The physical condition encoding and condition fusion module outputs the fused result.

[0220] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0221] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One to Eight in that step 333: the low spatial resolution multispectral image, spectral response function, and physical condition encoding and conditional fusion module output fusion result from the training set are input into the U-net-based spatial-spectral co-conditional diffusion module, and the U-net-based spatial-spectral co-conditional diffusion module outputs the predicted low spatial resolution hyperspectral image.

[0222] The specific process is as follows:

[0223] Step 3331: During the forward pass, gradually transfer the data from the low spatial resolution hyperspectral images in the training set. Start moving to each time step By adding Gaussian noise of a certain intensity, an ideal Gaussian noise image can be achieved. ;

[0224] At time step The image after adding noise is represented by the following formula:

[0225]

[0226] Step 3332: In the reverse process, gradually transform the image from the ideal Gaussian noise image. Start moving to each time step Denoising is performed to ultimately achieve the predicted ideal low spatial resolution hyperspectral image. ;

[0227] At time step The noise reduction process is as follows:

[0228] 1) Input to the first encoder, output feature of the first encoder The specific process is as follows:

[0229] Will Enter in sequence Convolutional layer, ReLU activation function layer, ReLU activation function layer output features ;

[0230] 2) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the first conditional injection module, the first conditional injection module outputs features The specific process is as follows:

[0231] Features enter Convolutional layer Convolutional layer output features;

[0232] The convolutional layer output features, physical conditional encoding, and conditional fusion module output the fused result features. Features are obtained by adding elements one by one. ;

[0233] Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ;

[0234] feature and characteristics Features are obtained by adding elements one by one. ;

[0235] 3) Output features of the first condition injection module Input to the second encoder, output feature of the second encoder The specific process is as follows:

[0236] Will Enter in sequence Convolutional layer, ReLU activation function layer, ReLU activation function layer output features ;

[0237] 4) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the second conditional injection module, the second conditional injection module outputs features The specific process is as follows:

[0238] Features enter Convolutional layer Convolutional layer output features;

[0239] The convolutional layer output features, physical conditional encoding, and conditional fusion module outputs the fused result features. Features are obtained by adding elements one by one. ;

[0240] Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ;

[0241] feature and characteristics Features are obtained by adding elements one by one. ;

[0242] 5) Output features of the second condition injection module Input to the third encoder, output features from the third encoder The specific process is as follows:

[0243] Will Enter in sequence Convolutional layer, ReLU activation function layer, ReLU activation function layer output features ;

[0244] 6) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the third conditional injection module, the third conditional injection module outputs features The specific process is as follows:

[0245] Features enter Convolutional layer Convolutional layer output features;

[0246] Will The convolutional layer outputs features and physical condition encoding, while the conditional fusion module outputs the fused feature result. Features are obtained by adding elements one by one. ;

[0247] Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ;

[0248] feature and characteristics Features are obtained by adding elements one by one. ;

[0249] 7) Inject the third condition into the module output features Input bottleneck layer, bottleneck layer output features The specific process is as follows:

[0250] The third condition injection module outputs features Input spatial attention mechanism, output features of spatial attention mechanism ;

[0251] The third condition injection module outputs features Input spectral attention mechanism, output features of spectral attention mechanism ;

[0252] Features and characteristics enter Convolutional layer Convolutional layer output features ;

[0253] 8) Features Input to the first decoder, output features of the first decoder The specific process is as follows:

[0254] feature Enter in sequence Transposed convolutional layer, ReLU activation function layer, ReLU activation function layer output features ;

[0255] feature Output features of the third conditional injection module Channel dimensions are concatenated to output features. ;

[0256] 9) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the fourth conditional injection module, the fourth conditional injection module outputs features The specific process is as follows:

[0257] Features enter Convolutional layer Convolutional layer output features;

[0258] Will The convolutional layer outputs features and the physical condition encoding and conditional fusion module outputs the fused result features. Features are obtained by adding elements one by one. ;

[0259] Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ;

[0260] feature and characteristics Features are obtained by adding elements one by one. ;

[0261] 10) Features The input is to the second decoder, and the output of the first decoder is the feature. The specific process is as follows:

[0262] feature Enter in sequence Transposed convolutional layer, ReLU activation function layer, ReLU activation function layer output features ;

[0263] feature Output features of the second conditional injection module Channel dimensions are concatenated to output features. ;

[0264] 11) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the fifth conditional injection module, the fifth conditional injection module outputs features The specific process is as follows:

[0265] Features enter Convolutional layer Convolutional layer output features;

[0266] Will The convolutional layer outputs features and the physical condition encoding and conditional fusion module outputs the fused result features. Features are obtained by adding elements one by one. ;

[0267] Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ;

[0268] feature and characteristics Features are obtained by adding elements one by one. ;

[0269] 12) Features Input a 2D convolutional layer, output a 2D convolutional layer. ;feature Predicted values ​​for low spatial resolution hyperspectral images ;

[0270] 13) Based on Get time step hyperspectral image at the location ; indicates the following:

[0271]

[0272] in, Indicates time step Hyperspectral image at the location; This represents the original, noise-free hyperspectral image;

[0273] This represents the mean of Gaussian noise predicted by the network and used for backsampling.

[0274] Represents predicted values ​​from low spatial resolution hyperspectral images This represents a generalized form of the denoising network structure described in 1) to 12) above, whose optimization objective is to output predicted values ​​for low spatial resolution hyperspectral images. ;

[0275] In the reverse denoising process of the diffusion model, it is assumed that... The proposed denoising network in the space-spectral joint conditional diffusion model differs from traditional conditional diffusion methods. It features two innovations: a multi-level injection mechanism for physical condition information and a parallel spatial-spectral attention mechanism. Specifically, in Each level of encoder and decoder is followed by a conditional fusion module to realize conditional information. In the multi-level injection process, the training target of the denoising network at each time step is a clean, low-spatial-resolution hyperspectral image. Therefore, in The inverse denoising network at each time step is represented as follows:

[0276]

[0277] network The specific structure and training process are as follows: First, the conditional input of the diffusion model... The output of the physical condition encoding and prior fusion, along with the temporal embedding module, are jointly determined. The temporal embedding module employs a sine-cosine encoding method and extracts features through an MLP, broadcasting them to the same spatial-spectral feature dimension. The conditional input is shown below:

[0278]

[0279] Denoising Network It consists of three main parts: an encoder, a spatial-spectral joint attention module, and a decoder.

[0280] Conditional input The testing process was performed on a U-net structure denoising network that had been trained with multi-level conditional injection. arrive The testing process is as follows:

[0281]

[0282] Among the proposed solutions, The value is set to decrease linearly from 0.3 to 0 to increase the randomness of the model's predictions at the beginning of training, avoiding getting trapped in local optima. Then, in the later training phase, deterministic sampling is used to ensure that the reconstruction task produces accurate and unique results. Additionally, the value is set... The value is linearly increased from 0.0001 to 0.02 to balance sampling quality and running speed. In the program, [the value is then...]. Set to 50, when hour, ;

[0283] The loss function of the physical prior guided model in step 334 Represented as:

[0284] The loss function guides the joint training of the conditional diffusion module with the physical feature encoding and prior fusion modules, while only the sampling process of the spectral prior module is used;

[0285]

[0286] in,

[0287] First loss This represents the reconstruction loss of the entire image;

[0288] Second loss This represents the difference loss between the result of predicting a low spatial resolution hyperspectral image using spectral downsampling and the input low spatial resolution multispectral image.

[0289] Third loss This represents the loss due to spectral dimension variation. It calculates the absolute value of the difference between the pixel values ​​of two adjacent channels in the spectral dimension of the predicted hyperspectral image, thus constraining the variation of the spectral dimension and making the spectral trend closer to the actual scene.

[0290] Fourth loss This represents the spatial texture feature consistency loss, which constrains the spatial texture feature consistency.

[0291] Fifth loss This represents the loss of spectral correlation, constraining spectral correlation.

[0292] This represents the original low spatial resolution hyperspectral image as input. This represents a predicted low spatial resolution hyperspectral image; This represents low spatial resolution multispectral images in the training set. This represents the spectral response result of the predicted image after spectral downsampling using the spectral response function (SRF). Indicates the first Pixel values ​​of a hyperspectral image within a band channel. Indicates the first Pixel values ​​of the hyperspectral image within the band channel; Represents the spatial gradient of the original image. It represents the spatial gradient of the predicted image; both are obtained by performing a first-order difference operation on the spatial dimensions (horizontal and vertical directions) of the spectral image, and are used to extract the spatial edge and texture information of the image. This represents the correlation coefficient between the original image and the predicted image. Band indexes representing the original and predicted low spatial resolution hyperspectral images; This indicates the total number of bands contained in a low spatial resolution hyperspectral image.

[0293] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0294] Specific Implementation Method 10: This implementation method differs from Specific Implementation Methods 1 to 9 in that step 4: acquire the high spatial resolution multispectral image to be tested, input the high spatial resolution multispectral image to be tested into the trained physical prior guided conditional diffusion model, and the trained physical prior guided conditional diffusion model outputs high spatial resolution hyperspectral data.

[0295] The specific process is as follows:

[0296] Step 41: Input the high spatial resolution multispectral image and spectral response function (SRF) into the pre-trained mask-dictionary guided spectral diffusion prior module, execute step 331, and output the pixel-by-pixel spectral prior prediction results. ;

[0297] Step 42: Output pixel-by-pixel spectral prior prediction results from the high spatial resolution multispectral image, the spectral response function (SRF), and the pre-trained mask-dictionary guided spectral diffusion prior module. The physical condition encoding and condition fusion modules are input together, and the fused result is output as the condition input for the space-spectrum cooperative conditional diffusion module based on the U-net structure.

[0298] Step 43: Based on the high spatial resolution multispectral image to be tested, the spectral response function, and the physical condition coding and conditional fusion module, the fused result is output, gradually transforming the image from an ideal Gaussian noise image. Start moving to each time step Denoising until Ultimately, the predicted values ​​of the high spatial resolution hyperspectral image of the target are obtained. .

[0299] The specific process is the same as step 3312.

[0300] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0301] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. An unsupervised super-resolution spectral method based on a physical prior-guided diffusion model, characterized in that: The specific process of the method is as follows: Step 1: Obtain real hyperspectral data, and use the spectral response function to perform spectral downsampling on the real hyperspectral data to obtain multispectral data; Multispectral-hyperspectral data pairs are constructed based on multispectral data and real hyperspectral data, and these multispectral-hyperspectral data pairs are used as the training set; Step 2: Construct a diffusion model based on physical prior guidance conditions; The physical prior-guided conditional diffusion model includes a mask-dictionary-guided spectral diffusion prior module, a physical condition encoding and conditional fusion module, and a space-spectral cooperative conditional diffusion module based on the U-net structure. The mask-dictionary guided spectral diffusion prior module includes a noise addition process and a noise reduction process; The physical condition encoding and condition fusion module includes an SRF encoder, an MSI encoder, an attention fusion module, and a condition fusion module; The SRF encoder comprises, in sequence, a flattening layer, a fully connected layer, a ReLU activation function layer, and another fully connected layer; The MSI encoder includes, in sequence, the following: Convolutional layer, residual module 1, residual module 2, residual module 3; Residual module 1 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer; Residual module 2 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer; Residual module 3 includes Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer; The attention fusion module is a channel attention module; The condition fusion module includes Depthwise separable convolutional layer, first linear layer, second linear layer, third linear layer, Convolutional layers, GELU activation function layers, Convolutional layers, Sigmoid activation function layers, weighted fusion Convolutional layer; The space-spectral coordinated conditional diffusion module based on the U-net structure includes a noise addition process and a noise removal process; The denoising process includes a first encoder, a first conditional injection module, a second encoder, a second conditional injection module, a third encoder, a third conditional injection module, a bottleneck layer, a first decoder, a fourth conditional injection module, a second decoder, a fifth conditional injection module, and a two-dimensional convolutional layer; The bottleneck layer includes spatial attention mechanisms, spectral attention mechanisms, Convolutional layer; Step 3: Train the physical prior guided conditional diffusion model based on the training set until the loss function of the physical prior guided conditional diffusion model converges, and obtain the trained physical prior guided conditional diffusion model. Step 4: Acquire the high spatial resolution multispectral image to be tested, input the high spatial resolution multispectral image to be tested into the trained physical prior guided conditional diffusion model, and the trained physical prior guided conditional diffusion model outputs high spatial resolution hyperspectral data.

2. The unsupervised spectral super-resolution method based on a physically prior guided diffusion model according to claim 1, characterized in that: Step 1: Obtain real hyperspectral data, and use the spectral response function to perform spectral downsampling on the real hyperspectral data to obtain multispectral data; Multispectral-hyperspectral data pairs are constructed based on multispectral data and real hyperspectral data, and these multispectral-hyperspectral data pairs are used as the training set; (1) in, This represents low spatial resolution hyperspectral data; Represents the spectral response function. This represents multispectral image data.

3. The unsupervised spectral super-resolution method based on a physically prior guided diffusion model according to claim 2, characterized in that: Step 3: Train the physical prior guided conditional diffusion model based on the training set until the loss function of the physical prior guided conditional diffusion model converges, thus obtaining the trained physical prior guided conditional diffusion model; the specific process is as follows: Step 31: Based on real hyperspectral data in the training set, extract semantic features of the spectral dimension pixel by pixel using the alternating direction multiplier method, and construct a complete hyperspectral dictionary. ; Step 32: Combine the multi-hyperspectral data pairs, spectral response functions, and the pre-trained hyperspectral dictionary from the training set. Input a mask-dictionary-guided spectral diffusion prior module, and the mask-dictionary-guided spectral diffusion prior module outputs the corresponding pixel-by-pixel spectral prior prediction result. Train the mask-dictionary-guided spectral diffusion prior module until the loss function of the mask-dictionary-guided spectral diffusion prior module converges, and obtain the pre-trained mask-dictionary-guided spectral diffusion prior module. Step 33: Construct a physical prior guidance model; The spectral diffusion prior module, based on multi-hyperspectral data pairs from the training set and a pre-trained mask-dictionary guided module, outputs pixel-by-pixel spectral prior predictions. The physical prior model is trained until its loss function converges, thus obtaining a pre-trained physical prior model.

4. The unsupervised spectral super-resolution method based on a physically prior guided conditional diffusion model according to claim 3, characterized in that: Step 31: Based on real hyperspectral data in the training set, extract semantic features of the spectral dimension pixel by pixel using the alternating direction multiplier method, and construct a complete hyperspectral dictionary. ; The specific process is as follows: Step 311: Let This represents the hyperspectral dictionary to be learned. The sparse representation coefficients of a hyperspectral image; Step 312: Based on the hyperspectral dictionary to be learned sparse representation coefficients of hyperspectral images The hyperspectral and multispectral image data pairs constructed in step 1 are represented as follows: (2) Step 313: Construct the optimization model, represented as: (3) in, Denotes the square of the F-norm; Represents the weight coefficients of the multispectral dictionary constraint terms; Represents the weight coefficients of the L1 sparse regularization term; Represents the 1-norm; Step 314: Let Separate the first and second terms, and let the third term in equation (4) be... As shown in formula (4): (4) in, Indicates parameters; Step 315: Add to the optimization model As a penalty term, construct the augmented Lagrangian function. As shown in equation (5): (5) in, Indicates the coefficient of the penalty term; Indicates parameters; Step 316: Based on the augmented Lagrangian function, and combined with the alternating direction multiplier method for each pixel, extract the semantic features of the spectral dimension for each pixel, and construct a complete hyperspectral dictionary; The specific process is as follows: Step 3161: Randomly initialize the dictionary using a Gaussian distribution. ; Initialize sparse coefficients ,parameter ,parameter It is a zero matrix; Step 3162: Set the maximum number of rounds Set the maximum number of iterations. ; Step 3163: Order of Rounds Let the number of iterations be... ; Step 3164: Sparsity Coefficient The optimization method is shown in formula (6): (6) in, Indicates the number of iterations The sparsity coefficient at time, Indicates the number of iterations Parameters at time; Indicates the number of iterations Parameters at time; make The derivative is 0, so we can solve for... The update method is as follows: (7) in, This represents the identity matrix, with the superscript T indicating transpose. Step 3165: Parameters The optimization formula is shown in formula (8): (8) in, Indicates the number of iterations Parameters at time; Solving equation (8) using the soft threshold operator yields the following result. The update method is as follows: (9) in, This represents the soft threshold operator. ); Step 3166: Parameters The optimization formula is shown in formula (10): (10) in, Indicates the number of iterations Parameters at time; Step 3167: Determine the number of iterations Is it less than ; If it is less than, let the number of iterations... Repeat steps 3164 to 3167 until... The number of iterations obtained sparsity coefficient at time As Execute step 3168; Otherwise, the number of iterations obtained sparsity coefficient at time As Execute step 3168; Step 3168: Adjust the sparsity coefficients and the hyperspectral dictionary to be optimized Substitute into formula (11) to update the hyperspectral dictionary. ; Represented as: (11) in, Indicates round Hyperspectral dictionary ; make With the derivative being 0, the closed-form solution of formula (11) is obtained, as shown in formulas (12) and (13): (12) (13) in, This represents the optimal solution; To prevent Singularity, add a regularization term to formula (13) To obtain intermediate variables and hyperspectral dictionary The update process is shown in (14) and (15): (14) (15) in, Represents the regularization parameter; Represents the identity matrix; Indicates the relaxation factor; Step 3169: Let Repeat steps 3164 to 3169 until... This yields a complete hyperspectral dictionary.

5. The unsupervised spectral super-resolution method based on a physically prior guided conditional diffusion model according to claim 4, characterized in that: Step 32: Combine the multi-hyperspectral data pairs, spectral response functions, and pre-trained hyperspectral dictionary from the training set. The input is a mask-dictionary-guided spectral diffusion prior module, which outputs corresponding pixel-by-pixel spectral prior prediction results. This module is trained until its loss function converges, resulting in a pre-trained mask-dictionary-guided spectral diffusion prior module. The specific process is as follows: Step 321: Response to the spectral response function After implementing a 20% random masking strategy, the features are encoded using an MLP. The process is as follows: Spectral response function When 20% of the elements in the matrix are set to 0, the transformed spectral response function... Input to a Multilayer Perceptron (MLP), output features of the Multilayer Perceptron (MLP) ; Step 322: Input the hyperspectral image into a two-dimensional convolutional layer, and the two-dimensional convolutional layer outputs spatial features. ; Features and characteristics By adding elements one by one, we can obtain the features. ; Step 323: Extract the spectral curves of the hyperspectral image pixel by pixel from the currently captured real reference hyperspectral image. The extracted spectral curves are then used... express; The forward process of the spectral diffusion prior model is progressively advanced to each time step. By adding Gaussian noise of a certain intensity, the ideal Gaussian noise spectrum curve can be achieved. By deriving each time step The noise addition process is derived in time. Intermediate spectral curve after adding noise As shown in formulas (16) to (18): (16) (17) (18) in, Indicates from State noise added State transition probabilities, This represents the spectrum without added noise; Indicates time step The spectrum after adding noise Indicates time step The spectrum after adding noise Indicates time step Noise scheduling parameters; Represents the identity matrix. Indicates the first One time step; express Depends on Gaussian conditional probability; This indicates that the mean is 0 and the variance is... Gaussian noise, Indicates at time step Added obedience Standard Gaussian noise, This indicates that the mean is 0 and the variance is... Gaussian noise; Represents noise scheduling parameters The cumulative product, ; Indicates time step Noise scheduling parameters; Indicates starting from the origin To the finish line The final equivalent result of all accumulated noise during the process, This indicates traversing from time step 1 to time step 2. Intermediate variables at each time step between them ; The inverse denoising process of the spectral diffusion prior model from the pure Gaussian noise spectral curve To begin, and finally obtain the target spectral curve. The specific process is as follows: pure Gaussian noise spectrum The input is fed into the spectral self-attention mechanism, and the spectral self-attention mechanism outputs features. ; feature and the features obtained in step 321 Add them together and add time-step embeddings to obtain the features. ; feature The target spectral curve is obtained by sequentially passing the data through a three-layer MLP network. ; Represented as: (19) in, For denoising networks; from arrive Denoising process It is expressed as follows: (23) in, This represents the mean of Gaussian noise; This indicates the level of noise added in each noise reduction process; Represents the variance of the probability distribution; Indicates the predicted spectral curve; Step 324: Construct the loss function for the mask-dictionary-guided spectral diffusion prior module; expressed as: in, First loss The second term represents the overall difference constraint between the original low spatial resolution hyperspectral image and the reconstructed low spatial resolution hyperspectral image. The third term represents the MSE-weighted error of the 20% spectral response function mask region. The fourth term represents the consistency constraint of the spectral response function. Representing dictionary sparsity constraints, the fifth loss term Indicates spatial texture feature constraints; and These represent the original spectral curve and the predicted spectral curve, respectively. This represents the spectral values ​​of the original spectral curve within the preset mask region of the spectral response function; This represents the spectral values ​​of the predicted spectral curve within the preset mask region of the spectral response function; Represents the square of the 2-norm; This represents the spectral response value after the original spectral curve has been sampled using the spectral response function (SRF). This represents the spectral response value of the predicted spectral curve after spectral downsampling using the spectral response function (SRF). The spectral sparsity indicates the true value of the coefficients; The predicted value of the spectral sparsity coefficient is indicated; This represents the spatial gradient of the original spectral curve; This represents the spatial gradient of the reconstructed spectral curve; The spectral sparsity representation coefficients are calculated as follows: in, Represents the weight coefficient of the regularization term; Represents the 1-norm; Represents the square of the 2-norm; Step 325: Train the mask-dictionary guided spectral diffusion prior module until the loss function of the mask-dictionary guided spectral diffusion prior module converges, and obtain the pre-trained mask-dictionary guided spectral diffusion prior module.

6. The unsupervised spectral super-resolution method based on a physically prior guided conditional diffusion model according to claim 5, characterized in that: Step 33: Construct a physical prior guidance model; The spectral diffusion prior module, based on multi-hyperspectral data pairs from the training set and a pre-trained mask-dictionary guided module, outputs pixel-by-pixel spectral prior predictions. Train the physics prior model until the loss function of the physics prior model converges to obtain the pre-trained physics prior model. The specific process is as follows: Step 331: Input the multispectral images and spectral response functions from the training set into the pre-trained mask-dictionary guided spectral diffusion prior module. The pre-trained mask-dictionary guided spectral diffusion prior module outputs pixel-by-pixel spectral prior prediction results. ; Step 332: Output pixel-by-pixel spectral prior prediction results from the multispectral images, spectral response functions, and the pre-trained mask-dictionary-guided spectral diffusion prior module in the training set. Input the physical condition encoding and condition fusion module, and the physical condition encoding and condition fusion module outputs the fused result; Step 333: Input the multispectral images, spectral response functions, and physical condition encoding and conditional fusion module outputs from the training set into the U-net-based spatial-spectral co-conditional diffusion module. The U-net-based spatial-spectral co-conditional diffusion module outputs the predicted hyperspectral image. Step 334: Repeat steps 331 to 334 to train the physical prior guided model until the loss function of the physical prior guided model converges, and obtain the pre-trained physical prior guided model.

7. The unsupervised spectral super-resolution method based on a physically prior guided diffusion model according to claim 6, characterized in that: Step 331: Input the multispectral images and spectral response functions from the training set into the pre-trained mask-dictionary guided spectral diffusion prior module, and the pre-trained mask-dictionary guided spectral diffusion prior module outputs pixel-by-pixel spectral prior prediction results. The specific process is as follows: First, execute steps 321 and 322 to obtain the features. ; Then from the pure Gaussian noise spectrum curve Starting from this point, we directly execute formula (23) in step 323 to gradually denoise the data, and finally obtain the pixel-by-pixel spectral prior prediction results. .

8. The unsupervised spectral super-resolution method based on a physically prior guided diffusion model according to claim 7, characterized in that: Step 332: Output pixel-by-pixel spectral prior prediction results from the multispectral images, spectral response functions, and pre-trained mask-dictionary guided spectral diffusion prior modules in the training set. Input the physical condition encoding and condition fusion module, and the physical condition encoding and condition fusion module outputs the fused result; The specific process is as follows: Step 3321: Input the spectral response functions from the training set into the SRF encoder, and the SRF encoder outputs the features. ; The specific process is as follows: The spectral response functions from the training set are sequentially input into a flattening layer, a fully connected layer, a ReLU activation function layer, and another fully connected layer. The fully connected layer outputs the features. ; Step 3322: Input the multispectral images from the training set into the MSI encoder, and the MSI encoder outputs features. ; The specific process is as follows: Input the multispectral images from the training set Convolutional layer Convolutional layer output features ; Will Convolutional layer output features Input residual module 1, output residual module 1 features The specific process is as follows: Will Convolutional layer output features Enter in sequence Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, batch normalization layer output features ; Output features of batch normalization layer and Convolutional layer output features By adding elements one by one, we can obtain the features. ; Features Input ReLU activation function layer, output features of ReLU activation function layer ; Features Input residual module 2, output residual module 2 features ; The specific process is as follows: Features Enter in sequence Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, batch normalization layer output features ; Output features of batch normalization layer With features By adding elements one by one, we can obtain the features. ; Features Input ReLU activation function layer, output features of ReLU activation function layer ; Features Input residual module 3, output residual module 3 features ,feature As an output feature of the MSI encoder ; The specific process is as follows: Features Enter in sequence Two-dimensional convolutional layer, batch normalization layer, ReLU activation function layer, Two-dimensional convolutional layer, batch normalization layer, batch normalization layer output features ; Output features of batch normalization layer With features By adding elements one by one, we can obtain the features. ; Features Input ReLU activation function layer, output features of ReLU activation function layer ; feature As an output feature of the MSI encoder ; Step 3323: Output features of the SRF encoder and MSI encoder output characteristics Input to the attention fusion module, output features from the attention fusion module. The specific process is as follows: SRF encoder output features and MSI encoder output characteristics Input channel attention module, output features of channel attention module ; Step 3324: Add features The pre-trained mask-dictionary-guided spectral diffusion prior module outputs pixel-by-pixel spectral prior predictions. Input the conditional fusion module, and the conditional fusion module outputs the fused result; The specific process is as follows: Step 33241: Output pixel-by-pixel spectral prior prediction results from the pre-trained mask-dictionary guided spectral diffusion prior module. enter Depth-separable convolutional layers The output features of the depthwise separable convolutional layer are input into the first linear layer (linear) and the second linear layer (linear), respectively. The first linear layer (linear) outputs the features. The second linear layer output features ; Step 33242: Add features Input to the third linear layer (linear), the third linear layer (linear) outputs features. ; Step 33243: Features and characteristics Element-wise multiplication yields the characteristics. ; Step 33244: Features and characteristics Element-wise multiplication yields the characteristics. ; Step 33245: Add features and characteristics By splicing the data, we can obtain the features. ; Step 33246: Add features Enter in sequence Convolutional layers, GELU activation function layers, Convolutional layer, Sigmoid activation function layer, Sigmoid activation function layer output features ; Step 33247: Features and characteristics Perform weighted fusion to obtain the fused features. ; Step 33248: Merge the features enter Convolutional layer Convolutional layer output features ; Step 33249: Add features and characteristics Element-by-element addition is performed to obtain the fused features output by the conditional fusion module. ; feature The physical condition encoding and condition fusion module outputs the fused result.

9. The unsupervised spectral super-resolution method based on a physically prior guided conditional diffusion model according to claim 8, characterized in that: Step 333: The multispectral images, spectral response functions, and physical condition encoding and conditional fusion module outputs the fused results from the training set and inputs them into the U-net-based spatial-spectral co-conditional diffusion module. The U-net-based spatial-spectral co-conditional diffusion module outputs the predicted hyperspectral image. The specific process is as follows: Step 3331: During the forward pass, gradually extract hyperspectral images from the training set. Start moving to each time step By adding Gaussian noise, the desired Gaussian noise image can be achieved. ; At time step The image after adding noise is represented by the following formula: Step 3332: In the reverse process, gradually transform the image from the ideal Gaussian noise image. Start moving to each time step Denoising is performed to ultimately achieve the predicted ideal hyperspectral image. ; At time step The noise reduction process is as follows: 1) Input to the first encoder, output feature of the first encoder The specific process is as follows: Will Enter in sequence Convolutional layer, ReLU activation function layer, ReLU activation function layer output features ; 2) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the first conditional injection module, the first conditional injection module outputs features The specific process is as follows: Features enter Convolutional layer Convolutional layer output features; The convolutional layer output features, physical conditional encoding, and conditional fusion module output the fused result features. Features are obtained by adding elements one by one. ; Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ; feature and characteristics Features are obtained by adding elements one by one. ; 3) Output features of the first condition injection module Input to the second encoder, output feature of the second encoder ; The specific process is as follows: Will Enter in sequence Convolutional layer, ReLU activation function layer, ReLU activation function layer output features ; 4) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the second conditional injection module, the second conditional injection module outputs features ; The specific process is as follows: Features enter Convolutional layer Convolutional layer output features; The convolutional layer output features, physical conditional encoding, and conditional fusion module outputs the fused result features. Features are obtained by adding elements one by one. ; Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ; feature and characteristics Features are obtained by adding elements one by one. ; 5) Output features of the second condition injection module Input to the third encoder, output features from the third encoder The specific process is as follows: Will Enter in sequence Convolutional layer, ReLU activation function layer, ReLU activation function layer output features ; 6) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the third conditional injection module, the third conditional injection module outputs features ; The specific process is as follows: Features enter Convolutional layer Convolutional layer output features; Will The convolutional layer outputs features and physical condition encoding, while the conditional fusion module outputs the fused feature result. Features are obtained by adding elements one by one. ; Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ; feature and characteristics Features are obtained by adding elements one by one. ; 7) Inject the third condition into the module output features Input bottleneck layer, bottleneck layer output features The specific process is as follows: The third condition injection module outputs features Input spatial attention mechanism, output features of spatial attention mechanism ; The third condition injection module outputs features Input spectral attention mechanism, output features of spectral attention mechanism ; Features and characteristics enter Convolutional layer Convolutional layer output features ; 8) Features Input to the first decoder, output features The specific process is as follows: feature Enter in sequence Transposed convolutional layer, ReLU activation function layer, ReLU activation function layer output features ; feature Output features of the third conditional injection module Channel dimensions are concatenated to output features. ; 9) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the fourth conditional injection module, the fourth conditional injection module outputs features The specific process is as follows: Features enter Convolutional layer Convolutional layer output features; Will The convolutional layer outputs features and physical condition encoding, while the conditional fusion module outputs the fused feature result. Features are obtained by adding elements one by one. ; Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ; feature and characteristics Features are obtained by adding elements one by one. ; 10) Features The input is to the second decoder, and the output of the first decoder is the feature. ; The specific process is as follows: feature Enter in sequence Transposed convolutional layer, ReLU activation function layer, ReLU activation function layer output features ; feature Output features of the second conditional injection module Channel dimensions are concatenated to output features. ; 11) Features Features of the fused result output by the physical condition encoding and condition fusion modules Input to the fifth conditional injection module, the fifth conditional injection module outputs features ; The specific process is as follows: Features enter Convolutional layer Convolutional layer output features; Will The convolutional layer outputs features and physical condition encoding, while the conditional fusion module outputs the fused feature result. Features are obtained by adding elements one by one. ; Features The input layers are sequentially a global average pooling layer, a fully connected layer, a ReLU activation function layer, and a sigmoid activation function layer. The sigmoid activation function layer outputs features. ; feature and characteristics Features are obtained by adding elements one by one. ; 12) Features Input a 2D convolutional layer, output a 2D convolutional layer. ;feature Predicted values ​​for hyperspectral images ; 13) Based on Get time step hyperspectral image at the location ; indicates the following: in, Indicates time step Hyperspectral image at the location; This represents the original, noise-free hyperspectral image; This represents the mean of Gaussian noise; Represents the predicted value of the hyperspectral image. ; The loss function of the physical prior guided model in step 334 Represented as: in, First loss This represents the reconstruction loss of the entire image; Second loss This represents the difference loss between the result of predicting a low spatial resolution hyperspectral image using spectral downsampling and the input low spatial resolution multispectral image. Third loss This indicates the loss due to changes in spectral dimensions; Fourth loss This represents the spatial texture feature consistency loss; Fifth loss This indicates spectral correlation loss; This represents the original hyperspectral image as input. Represents the predicted hyperspectral image; This represents the multispectral images in the training set. This represents the spectral response result of the predicted image after spectral downsampling using the spectral response function (SRF). Indicates the first Pixel values ​​of a hyperspectral image within a band channel. Indicates the first Pixel values ​​of the hyperspectral image within the band channel; Represents the spatial gradient of the original image. Represents the spatial gradient of the predicted image; This represents the correlation coefficient between the original image and the predicted image. Band indexes representing the original and predicted hyperspectral images; This indicates the total number of bands contained in the hyperspectral image.

10. The unsupervised spectral super-resolution method based on a physically prior guided conditional diffusion model according to claim 9, characterized in that: Step 4: Obtain the high spatial resolution multispectral image to be tested, input the high spatial resolution multispectral image to be tested into the trained physical prior guided conditional diffusion model, and the trained physical prior guided conditional diffusion model outputs high spatial resolution hyperspectral data. The specific process is as follows: Step 41: Input the high spatial resolution multispectral image and spectral response function (SRF) into the pre-trained mask-dictionary guided spectral diffusion prior module, execute step 331, and output the pixel-by-pixel spectral prior prediction results. ; Step 42: Output pixel-by-pixel spectral prior prediction results from the high spatial resolution multispectral image, the spectral response function (SRF), and the pre-trained mask-dictionary guided spectral diffusion prior module. Input both the physical condition encoding and condition fusion modules together, and output the fused result. Step 43: Based on the high spatial resolution multispectral image to be tested, the spectral response function, and the physical condition coding and conditional fusion module, the fused result is output, gradually transforming the image from an ideal Gaussian noise image. Start moving to each time step Denoising until Ultimately, the predicted values ​​of the high spatial resolution hyperspectral image of the target are obtained. .

Images