A hyperspectral image super-resolution method based on global guided conditional diffusion model

Abstract

The present invention relates to a hyperspectral image super-resolution method based on a global guided conditional diffusion model. Compared with the existing diffusion model-based technology, this method solves the problems of unbalanced modeling of global and local information and insufficient recovery of spatial details, especially when processing high-frequency features, which makes the reconstructed image not clear enough in details. The present invention includes the following steps: acquisition and preprocessing of low-resolution hyperspectral images, construction of a hyperspectral image super-resolution model based on a global guided conditional diffusion model, training of a hyperspectral image super-resolution model based on a global guided conditional diffusion model, and acquisition of hyperspectral super-resolution results. The present invention utilizes the Diffusion Transformer diffusion model to dynamically adjust the attention distribution to enhance the model's ability to handle complex and diverse noise patterns, ensure that high-quality images can be restored under different noise levels, better maintain details and spectral consistency, and achieve less computational overhead.

Description

Technical Field

[0001] The invention relates to the technical field of optical remote sensing image processing, and in particular to a hyperspectral image super-resolution method based on a global guided conditional diffusion model. Background Art

[0002] Hyperspectral images can capture rich spectral information from visible light to infrared bands, and therefore show wide application potential in many fields such as remote sensing, medical diagnosis and food safety. However, due to certain limitations of sensor technology in the spectral imaging process, the spatial resolution of hyperspectral images is relatively low. In order to solve this problem and achieve images with both high spatial and high spectral resolution, researchers have developed hyperspectral super-resolution technology to break through the limitations of traditional imaging technology.

[0003] Traditional image super-resolution algorithms can be mainly divided into interpolation methods and reconstruction methods. Interpolation methods generate high-resolution images by interpolating the pixel values ​​around the current pixel, such as bicubic interpolation and nearest neighbor interpolation. However, these methods often have limited effects on detail recovery and may cause blur or distortion. Reconstruction methods are based on degradation models to extract key information from low-resolution images to improve reconstruction effects. However, reconstruction methods are usually sensitive to noise and blur, and have high computational complexity, which limits their use in real-time applications. These traditional methods face challenges in dealing with complex scenes and are difficult to meet the needs of high-quality image restoration. With the rapid development of deep learning technology, neural networks have greatly improved the performance of super-resolution methods.

[0004] In recent years, deep learning technology has attracted the attention of researchers. Super-resolution networks based on deep convolutional neural networks avoid complex artificial priors and are significantly better than traditional algorithms. Transformer-based networks can capture long-range dependencies in images, which is very important for extracting complex spectral features in hyperspectral images. However, traditional Transformers may not be able to effectively generate detailed hyperspectral images, especially when dealing with complex scenes, the realism and details of the generated images may be insufficient. Diffusion models perform well in image generation tasks, can generate high-quality and diverse images, and are suitable for super-resolution tasks. As an emerging generative model, Diffusion Transformer combines the advantages of diffusion models and Transformers, and can better capture long-range dependencies and complex feature structures in images. This makes the model more expressive and flexible when generating high-quality images. However, diffusion models usually require a long training time and the training process is relatively complicated. Summary of the invention

[0005] The purpose of the present invention is to propose a hyperspectral image super-resolution method based on a global guided conditional diffusion model to address the deficiencies of the prior art. The present invention uses the Diffusion Transformer diffusion model and the pre-trained CLIP image encoder to ensure that the global structure of the image remains consistent during the reconstruction process and avoid distortion of local details. By dynamically adjusting the attention distribution, the model's ability to handle complex and diverse noise patterns is enhanced, ensuring that high-quality images can be restored under different noise levels, better maintaining details and spectral consistency, and achieving less computational overhead.

[0006] This hyperspectral image super-resolution method based on the global guided conditional diffusion model specifically includes the following steps:

[0007] 11) Low-resolution hyperspectral imaging Acquisition and preprocessing of

[0008] 12) Construct a hyperspectral image super-resolution model based on the global guided conditional diffusion model, which consists of four parts: encoder, decoder, global local guidance module, and conditional denoising Transformer module;

[0009] 13) Training a hyperspectral image super-resolution model based on a global guided conditional diffusion model. The training process includes: forward diffusion execution: low-resolution hyperspectral images Gradually add Gaussian noise until it becomes a completely Gaussian noise image; generate adaptive attention features as conditions and embed them into the conditional denoising Transformer module; perform reverse denoising: continuously remove noise from the Gaussian noise image until a high-resolution hyperspectral image is restored ;

[0010] 14) Obtaining super-resolution results of hyperspectral images: Convert low-resolution hyperspectral images Input into the trained hyperspectral image super-resolution model based on the global guided conditional diffusion model to obtain a high-resolution hyperspectral image .

[0011] Furthermore, the hyperspectral image super-resolution method based on the global guided conditional diffusion model is characterized in that the low-resolution hyperspectral image The acquisition and preprocessing of include the following steps:

[0012] 21) Select Pavia University and Urban public hyperspectral image data as reference images After removing the water absorption bands from the Pavia University and Urban hyperspectral image data, there are 103 bands left in the Pavia University hyperspectral image and 162 bands left in the Urban hyperspectral image, which are further used to divide the training set and the test set. A 128×128 block is cropped from the central area as the test set, and the remaining area is used for training. The training area is randomly cropped to a size of 128×128 to facilitate training in each iteration; the overlapped parts of the randomly divided training set and the test set are filled with 0 pixels;

[0013] 22) Producing low-resolution hyperspectral images :For reference image The low spatial resolution hyperspectral image for training was obtained by performing Gaussian blur kernel filtering with a window size of 9×9, a mean of 0, and a standard deviation of 2 and four downsampling. .

[0014] Furthermore, the hyperspectral image super-resolution method based on the global guided conditional diffusion model is characterized in that the construction of the hyperspectral image super-resolution model based on the global guided conditional diffusion model comprises the following steps:

[0015] 31) Construct an encoder to map the hyperspectral image to the latent space. Its structure is as follows:

[0016] First, there is a 3×3 convolutional block with LeakyReLU units and maximum pooling, followed by a Linear layer, a LeakyReLU layer, and a Linear layer. The first Linear layer receives the residual connection from the encoder input.

[0017] 32) Construct a decoder, which is responsible for mapping the representation in the latent space back to the original dimension of the hyperspectral image. Its structure is as follows:

[0018] First there is a Linear layer, a LeakyReLU layer, and a Linear layer, followed by a 3×3 convolution block, and finally a 1×1 convolution block with a LeakyReLU unit, and receives a residual connection from the decoder input;

[0019] 33) Construct a global-local guidance module, which consists of a CLIP image encoder and an adaptive attention feature generation part. The CLIP image encoder is a pre-trained CLIP image encoder obtained from the open source CLIP project.

[0020] 331) First, use the pre-trained CLIP image encoder to process the low-resolution hyperspectral image Extracting global semantic information as value V can be expressed as:

[0021] ,

[0022] Where R represents the global semantic information obtained, represents the bicubic interpolation algorithm, represents the pre-trained CLIP image encoder;

[0023] 332) The process of adaptive attention feature generation is as follows:

[0024] For a given time step t, the feature The query Q, key value K, value V, use They are scaled, where d represents the feature dimension of each attention head, which can be expressed as:

[0025] ,

[0026] ,

[0027] Among them, R represents the global semantic information, represents the generated adaptive attention feature, represents the normalized exponential function softmax, It is a feature The linear projection value of Represents query, key and value projection respectively, is a randomly initialized learnable matrix, T represents the transpose operation;

[0028] Finally, the obtained adaptive attention features are combined with the features at time step t Spliced ​​together and embedded into the conditional denoising Transformer module as conditions;

[0029] 34) Construct a conditional denoising Transformer module, which consists of four stacked conditional restoration structures. At each time step t, the model injects noise of different intensities according to the current noise level and uses conditional information to finely restore the details of the hyperspectral image. The conditional restoration structure includes: blocker, diffusion Transformer block, and Transformer decoder;

[0030] 341) The blocker divides the conditional input features into small blocks, and then converts them into tokens through linear embedding as the input of the diffusion Transformer block;

[0031] 342) The diffusion transformer block consists of an adaptive normalization layer, axial multi-head self-attention, and residual connections, regressing scaling and shifting from the noise time step and the conditional embedding vector; regressing the dimension scaling parameters applied before the residual connection within the diffusion transformer block;

[0032] 3421) Axial multi-head self-attention consists of horizontal decomposition and vertical decomposition branches. For input feature x, query Q, key value K, value V are first obtained from x and The average value of the query features in the horizontal direction is taken to achieve horizontal squeezing, and the average value of the query features in the vertical direction is taken to achieve vertical squeezing, and the result is and , after multi-head attention, we get the attention features in the horizontal and vertical directions, then add the features in the horizontal and vertical directions, and finally pass a layer of 1×1 convolution;

[0033] 343) The Transformer decoder consists of a standard linear decoder that decodes the sequence of tokens into noise prediction results.

[0034] Furthermore, the hyperspectral image super-resolution method based on the global guided conditional diffusion model is characterized in that the training of the hyperspectral image super-resolution model based on the global guided conditional diffusion model comprises the following steps:

[0035] 41) Input low-resolution hyperspectral image After being mapped to the latent space by the encoder, Gaussian noise is gradually added from time step t=0 to t=T to obtain the gradually noisy hyperspectral sequence sample X, which can be expressed as:

[0036] ,

[0037] ,

[0038] in represents the probability distribution of all noisy hyperspectral sequence samples, represents the hyperspectral sequence sample after adding noise for the tth time, represents the hyperspectral sequence sample after adding noise for the t-1th time, Represents a sample of a hyperspectral sequence without adding noise, pure noise It obeys Gaussian distribution, and the identity matrix I indicates that each dimension is independent;

[0039] 42) Input low-resolution hyperspectral image The pre-trained CLIP image encoder obtains the global semantic information as the value V and compares it with the noisy hyperspectral sequence sample at each time step t. Perform attention calculation, expressed as:

[0040] ,

[0041] ,

[0042] ,

[0043] Where R represents global semantic information, represents the bicubic interpolation algorithm, represents the pre-trained CLIP image encoder, represents the adaptive attention feature, represents the normalized exponential function softmax, It is a feature The linear projection value of Represents query, key and value projection respectively, is a randomly initialized learnable matrix, T represents the transpose operation; the obtained adaptive attention features are embedded as conditions into the conditional denoising Transformer module;

[0044] 43) The adaptive attention features at time step t and the noisy hyperspectral sequence samples are concatenated as conditional inputs to the conditional denoising Transformer module. According to the reverse denoising model, the noise added in the random noise adding process is predicted, and the noise in the noisy hyperspectral sequence samples is removed accordingly to complete the denoising process and obtain the target hyperspectral sequence, which can be expressed as:

[0045] ,

[0046] ,

[0047] ,

[0048] Among them, pure noise It obeys Gaussian distribution. The unit matrix I indicates that each dimension is independent and the variance of each dimension is 1. Parameterized by θ, A represents the adaptive attention feature; the conditional denoising Transformer module provides the prediction of the noise, and each step is refined as follows:

[0049] ,

[0050] in, and Represents the conditional denoising Transformer module;

[0051] 44) Training and optimization of model parameters: Model loading low-resolution hyperspectral images , according to the mean absolute error term and gradient loss The training loss function composed of the above two algorithms updates the network parameters through the gradient back propagation optimization algorithm to reduce the error, so that the high-resolution hyperspectral image is constantly close to the reference image. ;

[0052] 441) Absolute value error term The loss is expressed as:

[0053] ,

[0054] in, and is the nth reconstructed high-resolution hyperspectral image and reference image, N represents the number of images in a training batch;

[0055] 442) Gradient Loss , expressed as:

[0056] ,

[0057] Among them, M calculates the gradient values ​​along the horizontal, vertical and spectral dimensions of the image, and are the nth reconstructed high-resolution hyperspectral image and reference image, and N represents the number of images in a training batch;

[0058] 443) Overall loss function: The overall loss function is composed of the mean absolute error term and gradient loss The weighted sum composition can be expressed as,

[0059] ,

[0060] in represents the mean absolute error term loss, represents the gradient loss, is a regularization hyperparameter, fixed to 0.1;

[0061] 45) Determine whether the set training time step is reached. If so, complete the hyperspectral image super-resolution model based on the global guided conditional diffusion model, otherwise continue training.

[0062] Beneficial Effects

[0063] The present invention relates to a hyperspectral image super-resolution method based on a global guided conditional diffusion model, which fully improves the shortcomings of existing fusion methods. In view of the problem that the existing methods are unbalanced in modeling global and local information, resulting in the inability of super-resolution results to show the details and textures of the original hyperspectral image, a global-local guided module is designed, and the pre-trained CLIP image encoder and attention mechanism are used to fully learn the global semantic features and local features of the hyperspectral image, enhance the influence of specific areas or features, and improve the naturalness and coherence of the reconstruction results; in view of the problem of high computational overhead of the diffusion model, a more efficient encoder and decoder structure is designed, and the computational burden is significantly reduced by mapping high-dimensional hyperspectral data to a low-dimensional latent space. The introduction of the latent space can reduce noise interference and enhance the model's attention to local details, thereby improving the quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flow chart of the method of the present invention;

[0065] Figure 2 It is a structural diagram of a hyperspectral image super-resolution model based on a global guided conditional diffusion model involved in the present invention;

[0066] Figure 3 A structural diagram of a conditional recovery structure according to the present invention;

[0067] Figure 4 It is a structural diagram of the axial multi-head attention involved in the present invention;

[0068] Figure 5 It is an effect diagram of the implementation of the present invention; DETAILED DESCRIPTION

[0069] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0070] like Figure 1 As shown, the hyperspectral image super-resolution method based on the global guided conditional diffusion model described in the present invention is characterized by comprising the following steps:

[0071] The first step is low-resolution hyperspectral imaging The specific steps of acquisition and preprocessing are as follows:

[0072] (1) Pavia University and Urban public hyperspectral image data were selected as reference images. After removing the water absorption bands from the Pavia University and Urban data, 103 and 162 bands remained, respectively.

[0073] Division of training set and test set: a 128×128 block is cropped from the central area of ​​the reference image as the test set, and the remaining area is used for training. The training area is randomly cropped to a size of 128×128 for training in each iteration;

[0074] The overlapping parts of the randomly divided training set and test set are filled with 0 pixels;

[0075] (2) Producing low-resolution hyperspectral images: The reference image is filtered with a Gaussian blur kernel with a window size of 9×9, a mean of 0, and a standard deviation of 2, and downsampled four times to obtain a low spatial resolution hyperspectral image for training.

[0076] The second step is to build a hyperspectral image super-resolution model based on the global guided conditional diffusion model. The model consists of four parts: encoder, decoder, global-local guidance module, and conditional denoising Transformer module. The specific steps are as follows:

[0077] (1) If Figure 2 As shown, an encoder is constructed to map the hyperspectral image to the latent space, and its structure is as follows:

[0078] First, there is a 3×3 convolutional block with LeakyReLU units and maximum pooling, followed by a Linear layer, a LeakyReLU layer, and a Linear layer. The first Linear layer receives the residual connection from the encoder input.

[0079] (2) If Figure 2 As shown, a decoder is constructed to map the representation in the latent space back to the original dimension of the hyperspectral image. Its structure is as follows:

[0080] First there is a Linear layer, a LeakyReLU layer, and a Linear layer, followed by a 3×3 convolution block, and finally a 1×1 convolution block with a LeakyReLU unit, and receives a residual connection from the decoder input;

[0081] (3) If Figure 2 As shown, a global-local guidance module is constructed, which consists of a CLIP image encoder and an adaptive attention feature generation part:

[0082] (3-1) First, the pre-trained CLIP image encoder is used to extract the global semantic embedding from the low-resolution hyperspectral image to be processed, as the value V, which can be expressed as:

[0083] ,

[0084] Where R represents the obtained global semantic embedding, represents the bicubic interpolation algorithm, represents the pre-trained CLIP image encoder;

[0085] Then, the extracted global semantic information is multiplied by the adaptive attention map and embedded into the conditional denoising Transformer module;

[0086] (3-2) The generation process of adaptive attention features is as follows:

[0087] For features Given a query Q, key K, and value V at time step t, use They are scaled, where d represents the feature dimension of each attention head, which can be expressed as:

[0088] ,

[0089] ,

[0090] in, represents the generated adaptive attention feature, represents the normalized exponential function softmax, It is a feature The linear projection value of Represents query, key and value projection respectively, is a randomly initialized learnable matrix, T represents the transpose operation;

[0091] Finally, the obtained adaptive attention features are combined with the features at time step t Spliced ​​together and embedded into the conditional denoising Transformer module as conditions;

[0092] (4) If Figure 3 As shown in the figure, a conditional denoising Transformer module is constructed, which consists of four stacked conditional restoration structures. At each time step t, the model injects noise of different intensities according to the current noise level and uses conditional information to finely restore image details. The conditional restoration structure includes: blocker, diffusion Transformer block, and Transformer decoder.

[0093] (4-1) The blocker divides the input into small blocks, which are then converted into tokens through linear embedding as input to the diffusion Transformer block;

[0094] (4-2) If Figure 3As shown, the diffusion transformer block consists of an adaptive normalization layer, axial multi-head self-attention, and residual connections, regressing scaling and shifting from the noise time step and the conditional embedding vector; the regression calculates the dimension scaling parameters applied before the residual connection within the diffusion transformer block;

[0095] (4-2-1) Figure 4 As shown in the figure, the axial multi-head self-attention consists of horizontal decomposition and vertical decomposition branches. For the input feature x, the query Q, key value K, and value V are first obtained from x and The average value of the query features in the horizontal direction is taken to achieve horizontal squeezing, and the average value of the query features in the vertical direction is taken to achieve vertical squeezing, and the result is and , after multi-head attention, we get the attention features in the horizontal and vertical directions, then add the features in the horizontal and vertical directions, and finally pass a layer of 1×1 convolution;

[0096] (4-3) The Transformer decoder consists of a standard linear decoder that decodes the sequence of tokens and outputs the noise prediction result;

[0097] The third step is to train the hyperspectral image super-resolution model based on the global guided conditional diffusion model, which includes the following steps:

[0098] (1) After the input low-resolution hyperspectral remote sensing image is mapped to the latent space through the encoder, Gaussian noise is gradually added from time step t = 0 to t = T to obtain a gradually noisy hyperspectral sequence, which can be expressed as:

[0099] ,

[0100] ,

[0101] in represents the probability distribution of all noised data samples, represents the sample after adding noise for the tth time, represents the sample after adding noise for the t-1th time, Represents hyperspectral data without added noise, pure noise It obeys Gaussian distribution, and the identity matrix I indicates that each dimension is independent;

[0102] (2) The input low-resolution hyperspectral remote sensing image is passed through the pre-trained CLIP image encoder to obtain the global semantic information as the value V, which is then compared with the noisy hyperspectral sequence at each time step. The generated adaptive attention map is multiplied and expressed as:

[0103] ,

[0104] ,

[0105] ,

[0106] Where R represents the obtained global semantic embedding, represents the bicubic interpolation algorithm, represents the pre-trained CLIP image encoder, represents the adaptive attention feature, represents the normalized exponential function softmax, It is a feature The linear projection value of Represents query, key and value projection respectively, is a randomly initialized learnable matrix, T represents the transpose operation; the obtained adaptive attention features are embedded as conditions into the conditional denoising Transformer module;

[0107] (3) The adaptive attention features at time step t and the noisy hyperspectral sequence samples are concatenated as conditional inputs to the conditional denoising Transformer module. According to the reverse denoising model, the noise added in the random noise adding process is predicted, and the noise in the noisy hyperspectral sequence is removed accordingly to complete the denoising process and obtain the target hyperspectral sequence, which can be expressed as:

[0108] ,

[0109] ,

[0110] ,

[0111] Among them, pure noise It obeys Gaussian distribution. The unit matrix I indicates that each dimension is independent and the variance of each dimension is 1. Parameterized by θ, Represents the adaptive attention feature; the conditional denoising Transformer module provides the prediction of the noise, and each step is described in detail as follows:

[0112] ,

[0113] in, and Represents the conditional denoising Transformer module.

[0114] (4) Training and optimization of model parameters: The model loads low-resolution hyperspectral remote sensing images and uses the mean absolute error term and gradient loss The training loss function composed of the above two algorithms updates the network parameters through the gradient back propagation optimization algorithm to reduce the error, so that the high-resolution hyperspectral image is constantly close to the reference image.

[0115] (4-1) Absolute value error term The loss is expressed as:

[0116] ,

[0117] in, and is the nth reconstructed hyperspectral image and reference image, N represents the number of images in a training batch;

[0118] (4-2) Gradient loss , expressed as:

[0119] ,

[0120] Among them, M calculates the gradient values ​​along the horizontal, vertical and spectral dimensions of the image, and is the nth reconstructed hyperspectral image and reference image, N represents the number of images in a training batch;

[0121] (4-3) Overall loss function: The overall loss function is composed of the mean absolute error term and gradient loss Weighted and composed:

[0122] ,

[0123] in represents the mean absolute error term loss, represents the gradient loss, is a regularization hyperparameter, fixed to 0.1;

[0124] (5) Determine whether the set training time step is reached. If so, complete the hyperspectral image super-resolution model based on the global guided conditional diffusion model; otherwise, continue training.

[0125] The fourth step is to obtain low-resolution hyperspectral remote sensing images.

[0126] The fifth step is to input the low-resolution hyperspectral remote sensing image into the trained hyperspectral image super-resolution model based on the global guided conditional diffusion model to obtain a high-resolution hyperspectral remote sensing image.

[0127] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A hyperspectral image super-resolution method based on a global guided conditional diffusion model, characterized in that: The following steps are involved: 11) Low-resolution hyperspectral image HSI lr Acquisition and preprocessing of 12) Construct a hyperspectral image super-resolution model based on the global guided conditional diffusion model. The model consists of four parts: encoder, decoder, global local guidance module, and conditional denoising Transformer module, including the following steps: 121) Construct an encoder to map the hyperspectral image to the latent space, and its structure is as follows: First, there is a 3×3 convolutional block with LeakyReLU units and maximum pooling, followed by a Linear layer, a LeakyReLU layer, and a Linear layer. The first Linear layer receives the residual connection from the encoder input. 122) Construct a decoder, which is responsible for mapping the representation in the latent space back to the original dimension of the hyperspectral image. Its structure is as follows: First there is a Linear layer, a LeakyReLU layer, and a Linear layer, followed by a 3×3 convolution block, and finally a 1×1 convolution block with a LeakyReLU unit, and receives a residual connection from the decoder input; 123) constructing a global local guidance module, which consists of a CLIP image encoder and an adaptive attention feature generation part, wherein the CLIP image encoder is a pre-trained CLIP image encoder obtained from the open source CLIP project; 1231) First, the pre-trained CLIP image encoder is used to process the low-resolution hyperspectral image HSI lr Extracting global semantic information as value V can be expressed as: R=CLIP(BiCubic(HSI lr )), Where R represents the global semantic information obtained, BiCubic() represents the bicubic interpolation algorithm, and CLIP() represents the pre-trained CLIP image encoder; 1232) The process of adaptive attention feature generation is as follows: For a given time step t, feature X t The query Q, key value K, value V, use They are scaled, where d represents the feature dimension of each attention head, which can be expressed as: Q=W Q X t ,K=W K X t ,V=W V R, Among them, R represents global semantic information, A represents the generated adaptive attention features, and softmax() represents the normalized exponential function softmax. It is feature X t The linear projection value, Q = W Q X t , K=W K X t , V=W V R represents query, key and value projection respectively, and W Q , W K and W V is a randomly initialized learnable matrix, T represents the transpose operation; Finally, the obtained adaptive attention feature is combined with the feature X at time step t t Spliced ​​together and embedded into the conditional denoising Transformer module as conditions; 124) Construct a conditional denoising Transformer module, which consists of four stacked conditional restoration structures. At each time step t, the model injects noise of different intensities according to the current noise level and uses conditional information to finely restore the details of the hyperspectral image. The conditional restoration structure includes: blocker, diffusion Transformer block, and Transformer decoder; 1241) The blocker divides the conditional input features into small blocks, and then converts them into tokens through linear embedding as the input of the diffusion Transformer block; 1242) The diffusion transformer block consists of an adaptive normalization layer, axial multi-head self-attention, and residual connections, regressing scaling and shifting from the noise time step and the conditional embedding vector; regressing the dimension scaling parameters applied before the residual connection within the diffusion transformer block; 12421) Axial multi-head self-attention consists of horizontal decomposition and vertical decomposition branches. For input feature x, query Q, key value K, value V are first obtained from x and W. Q , W K and W V The average value of the query features in the horizontal direction is taken to achieve horizontal squeezing, and the average value of the query features in the vertical direction is taken to achieve vertical squeezing, and q is obtained. h , k h , v h and q v , k v , v v , after multi-head attention, we get the attention features in the horizontal and vertical directions, then add the features in the horizontal and vertical directions, and finally pass a layer of 1×1 convolution; 1243) The Transformer decoder consists of a standard linear decoder that decodes the sequence of tokens and outputs the noise prediction result; 13) Training a hyperspectral image super-resolution model based on a global guided conditional diffusion model. The training process includes: forward diffusion execution: low-resolution hyperspectral image HSI lr Gradually add Gaussian noise until it becomes a completely Gaussian noise image; generate adaptive attention features as conditions and embed them into the conditional denoising Transformer module; perform reverse denoising: continuously remove noise from the Gaussian noise image until a high-resolution hyperspectral image HSI is restored hr ; 14) Obtaining super-resolution results of hyperspectral images: Convert low-resolution hyperspectral images HSI lr Input into the trained hyperspectral image super-resolution model based on the global guided conditional diffusion model to obtain a high-resolution hyperspectral image HSI. hr .

2. The hyperspectral image super-resolution method based on the global guided conditional diffusion model according to claim 1, characterized in that: The low-resolution hyperspectral image HSI lr The acquisition and preprocessing of include the following steps: 21) Select Pavia University and Urban public hyperspectral image data as reference image HSI ref After removing the water absorption bands from the Pavia University and Urban hyperspectral image data, there are 103 bands remaining in the Pavia University hyperspectral image and 162 bands remaining in the Urban hyperspectral image, which are further used to divide the training set and the test set; in the reference image HSI ref A 128×128 block is cropped from the central area as the test set, and the remaining area is used for training. The training area is randomly cropped to a size of 128×128 to facilitate training in each iteration; the overlapped parts of the randomly divided training set and the test set are filled with 0 pixels; 22) Produce low-resolution hyperspectral images (HSI) lr :For the reference image HSI ref A Gaussian blur kernel filter with a window size of 9×9, a mean of 0, and a standard deviation of 2 and four downsamplings were performed to obtain a low spatial resolution hyperspectral image HSI for training. lr .

3. The hyperspectral image super-resolution method based on the global guided conditional diffusion model according to claim 1, characterized in that: The training of the hyperspectral image super-resolution model based on the global guided conditional diffusion model comprises the following steps: 31) Input low-resolution hyperspectral image HSI lr After being mapped to the latent space by the encoder, Gaussian noise is gradually added from time step t = 0 to t = T to obtain the gradually noisy hyperspectral sequence sample X, which can be expressed as: q(X t |X t-1 )=N(X t ;Xt -1 ,IN), Where q(X t：T |X0) represents the probability distribution of all noised hyperspectral sequence samples, X t represents the hyperspectral sequence sample after adding noise for the tth time, X t-1 represents the hyperspectral sequence sample after adding noise for the t-1th time, X0 represents the hyperspectral sequence sample without adding noise, and pure noise X T ~N(0,I) follows a Gaussian distribution, and the identity matrix I indicates that each dimension is independent; 32) Input low-resolution hyperspectral image HSI lr The pre-trained CLIP image encoder obtains the global semantic information as the value V and compares it with the noisy hyperspectral sequence sample X at each time step t. t Perform attention calculation, expressed as: R=CLIP(BiCubic(X Lr )), Q=W Q X t ,K=W K X t ,V=W V R, Where R represents global semantic information, BiCubic() represents the bicubic interpolation algorithm, CLIP() represents the pre-trained CLIP image encoder, A represents the adaptive attention feature, and softmax() represents the normalized exponential function softmax. It is feature X t The linear projection value, Q = W Q X t , K=W K X t , V=W V R represents query, key and value projection respectively, and W Q , W K and W V is a randomly initialized learnable matrix, T represents the transpose operation; the obtained adaptive attention features are embedded into the conditional denoising Transformer module as conditions; 33) The adaptive attention features at time step t and the noised hyperspectral sequence sample X t splicing, as the conditional input conditional denoising Transformer module, according to the reverse denoising model, predict the noise added in the random noise adding process, and remove the noise in the noisy hyperspectral sequence sample accordingly to complete the denoising process and obtain the target hyperspectral sequence, which can be expressed as: p(X T )=N(X T ;0,I), p θ (X t-1 |X t ,A)=N(X t-1 ;(X t ,A),I), Among them, pure noise X T ~N(0,I) obeys Gaussian distribution, the unit matrix I indicates that each dimension is independent, and the variance of each dimension is 1, and the distribution p θ (X 0：T |A) is parameterized by θ, where A represents the adaptive attention feature; the conditional denoising Transformer module provides the prediction of the noise, and each step is described in detail as follows: X t-1 =f θ (X t ,A)+X t +ε, Among them, ε~N(0,I) and f θ Represents the conditional denoising Transformer module; 34) Training and optimization of model parameters: Model loading low-resolution hyperspectral image HSI lr , according to the mean absolute error term L1 and the gradient loss L g The training loss function composed of the above is used to update the network parameters through the gradient back propagation optimization algorithm to reduce the error and make the high-resolution hyperspectral image HSI lr Continuously approaching the reference image HSI ref ; 341) The absolute value error term L1 loss is expressed as: in, and are the nth reconstructed high-resolution hyperspectral image and reference image, and N represents the number of images in a training batch; 342) Gradient loss L g , expressed as: Among them, M calculates the gradient values ​​along the horizontal, vertical and spectral dimensions of the image, and are the nth reconstructed high-resolution hyperspectral image and reference image, and N represents the number of images in a training batch; 343) Overall loss function: The overall loss function consists of the mean absolute error term L1 and the gradient loss L g The weighted sum composition can be expressed as, Where L1 represents the mean absolute error term loss, L g Represents the gradient loss, λ is a regular hyperparameter, fixed at 0.1; 35) Determine whether the set training time step is reached, if so, complete the hyperspectral image super-resolution model based on the global guided conditional diffusion model, otherwise continue training.

Images