Hyperspectral image super-division method and system based on hidden space diffusion model

Abstract

The invention discloses a hyperspectral image super-division method and system based on a hidden space diffusion model, and the method comprises the steps: carrying out the up-sampling of a low-resolution hyperspectral image, grouping, obtaining low-dimensional prior hidden space information through the network dimension reduction coding of an encoder, inputting the information into a multi-scale condition encoder, and carrying out the coding of a multi-scale condition encoder to obtain condition information, after the low-dimensional prior hidden space information and a Gaussian noise channel are spliced, Gaussian noise is converted into high-frequency residual hidden space information under the guidance of condition information by utilizing a diffusion model, and the high-frequency residual hidden space information and the low-dimensional prior hidden space information are added to obtain low-dimensional hidden space information with high-frequency details; and decoding and restoring through a pre-trained decoder network to obtain a final high-resolution hyperspectral super-resolution image. The invention aims to solve the problems that a traditional hyperspectral image super-resolution method based on a convolutional neural network is too smooth and lacks texture detail information during image reconstruction, and challenges are faced when a diffusion model is applied to hyperspectral image super-resolution.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a hyperspectral image super-resolution method and system based on a latent space diffusion model. Background Art

[0002] In recent years, with the rapid development of remote sensing technology and sensors, it has become easier to obtain remote sensing image data. In particular, hyperspectral images have shown great application potential in the fields of land object classification, environmental monitoring, mineral identification and agricultural analysis due to their rich spectral information. However, hyperspectral images usually face the problem of insufficient spatial resolution, which limits their application in applications such as land object identification.

[0003] To overcome this shortcoming, many research works are devoted to the super-resolution technology of hyperspectral images. The current super-resolution methods of hyperspectral images can be divided into two categories: hyperspectral image super-resolution based on image fusion and hyperspectral image super-resolution for single images. Methods based on image fusion can usually obtain high-quality high-resolution hyperspectral images, but these methods rely on auxiliary images with high spatial resolution, which significantly increases the complexity of the imaging system and seriously restricts its practical application. Single image super-resolution methods include interpolation methods, sparse representation-based methods, and deep learning-based methods. Although the interpolation method is simple and fast, it can easily cause hyperspectral image blurring and affect image quality. The calculation based on sparse representation is relatively complex, and the algorithm robustness is poor.

[0004] Deep learning methods are currently the mainstream hyperspectral image super-resolution methods, such as VDSR, 3DFCNN, EDSR, MCNet and EUNet.Among them, the VDSR method references Kim J, Lee JK, Lee K M. Accurate imagesuper-resolution using very deep convolutional networks[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1646-1654; the 3DFCNN method references Mei S, Yuan X, Ji J, et al. Hyperspectral imagespatial super-resolution via 3D full convolutional neural network[J]. RemoteSensing, 2017, 9(11): 1139, EDSR method reference Lim B, Son S, Kim H, et al.Enhanced deep residual networks for single image super-resolution[C] / / Proceedings of the IEEE conference on computer vision and pattern recognitionworkshops. 2017: 136-144; MCNet method reference Li Q, Wang Q, Li X. Mixed 2D / 3Dconvolutional network for hyperspectral image super-resolution[J]. Remotesensing, 2020, 12(10): 1660; EUNet method references Liu D, Li J, Yuan Q, et al. Anefficient unfolding network with disentangled spatial-spectral representationfor hyperspectral image super-resolution[J]. Information Fusion, 2023, 94:92-111. However, although these methods have achieved relatively good super-resolution effects, the images reconstructed by existing methods are often relatively smooth and lack texture details.

[0005] As an emerging generative modeling method, the diffusion model has attracted much attention due to its remarkable performance in image generation. The model generates high-quality images by gradually removing noise and has the ability to handle high-dimensional data and complex distributions, making it very suitable for hyperspectral image super-resolution tasks. However, there are some challenges in applying the diffusion model to hyperspectral image super-resolution. The high dimensionality and scarcity of hyperspectral data often lead to difficulties in model training and convergence. In addition, the diffusion model is computationally intensive and its iterative inference process leads to long generation time. These factors seriously hinder the application of the diffusion model in hyperspectral image super-resolution. Summary of the invention

[0006] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a hyperspectral image super-resolution method and system based on a latent space diffusion model are provided. The present invention aims to solve the problems of over-smoothing and lack of texture detail information in the reconstructed image of the traditional hyperspectral image super-resolution method based on convolutional neural network, as well as the challenges faced when applying the diffusion model to the super-resolution of hyperspectral images.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: A hyperspectral image super-resolution method based on a latent space diffusion model comprises the following steps: S1, upsampling the low-resolution hyperspectral image and grouping it according to the number of bands; S2, reducing the dimensionality of each group of images in the spectral dimension through a pre-trained encoder network in turn to obtain low-dimensional prior latent space information and encoding it through a multi-scale conditional encoder to obtain conditional information; S3, concatenating the low-dimensional prior latent space information with a Gaussian noise channel and converting the Gaussian noise into high-frequency residual latent space information through a diffusion model under the guidance of the conditional information; S4, adding the high-frequency residual latent space information to the low-dimensional prior latent space information to obtain low-dimensional latent space information with high-frequency details; S5, decoding and restoring each group of low-dimensional latent space information through a pre-trained decoder network to obtain the final high-resolution hyperspectral super-resolution image.

[0008] Optionally, the encoder network in step S2 includes a 5×5 convolutional layer, a plurality of residual modules and a 5×5 convolutional layer connected in sequence, the data input to the encoder network first passes through a 5×5 convolutional layer to obtain preliminary features, the preliminary features pass through a plurality of residual modules in sequence and then are added to the preliminary features, and finally pass through a 5×5 convolutional layer to obtain the final output of the encoder network; the decoder network includes a 5×5 convolutional layer, a first group of multiple residual modules, a 5×5 convolutional layer, a 5×5 convolutional layer, a second group of multiple residual modules and a 5×5 convolutional layer connected in sequence, the data input to the decoder network first passes through a 5×5 convolutional layer to obtain preliminary features, the preliminary features pass through a first group of multiple residual modules in sequence and then are added to the preliminary features, and finally pass through a 5× 5 convolution layer to obtain deep feature information, multiple deep feature information are channel-joined to obtain deep image features, the deep image features are first passed through a 5×5 convolution layer to obtain reconstructed image features, the reconstructed image features are sequentially passed through a second group of multiple residual modules and then added to the reconstructed image features, and finally passed through a 5×5 convolution layer to obtain reconstructed image information, and the reconstructed image information is added to the deep image features to obtain the output of the decoder network; the residual module includes a 5×5 convolution layer, a LeakyReLU activation function and a 5×5 convolution layer, and the data input to the residual module is sequentially processed by a 5×5 convolution layer, a LeakyReLU activation function and a 5×5 convolution layer, and then added to the data input to the residual module to obtain the output data of the residual module.

[0009] Optionally, the multi-scale conditional encoder in step S2 includes a 3×3 convolutional layer and multiple spatial spectral prior modules connected in sequence, and the data input to the multi-scale conditional encoder first passes through the 3×3 convolutional layer to extract preliminary features, and then the preliminary features pass through multiple spatial spectral prior modules in sequence to obtain conditional information of multiple scales; the spatial spectral prior module includes a 3×3 convolutional layer, multiple spatial modules, multiple spatial spectral modules and a 3×3 downsampling convolutional layer connected in sequence, and the data input to the spatial spectral prior module first passes through the 3×3 convolutional layer to extract initial depth features. The initial depth feature passes through three spatial modules in sequence and is added to the initial depth feature to generate an intermediate depth feature; the intermediate depth feature is then processed by multiple spatial spectrum modules and added to the intermediate depth feature and the initial depth feature to obtain an enhanced depth feature, and finally the enhanced depth feature is passed through a 3×3 downsampling convolution layer to obtain the output conditional information; the spatial module includes a 3×3 convolution layer, a LeakyReLU activation function, a 3×3 convolution layer, another 3×3 convolution layer, a LeakyReLU activation function and a final 3×3 convolution layer connected in sequence, and the output The data input to the spatial module first passes through the first 3×3 convolution layer and the LeakyReLU activation function in sequence, then passes through the second 3×3 convolution layer, and is added to the initial input data to obtain preliminary processing features. Subsequently, these preliminary processing features are processed by the third 3×3 convolution layer, the LeakyReLU activation function and the final 3×3 convolution layer in sequence, and then added to the preliminary processing features to form the final output features of the spatial module; the spatial spectrum module includes a 3×3 convolution layer, a LeakyReLU activation function, a 3×3 convolution layer connected in sequence , 1 1×1 convolution layer, LeakyReLU activation function, 1 1×1 convolution layer and 1 channel attention module. The data input to the spatial spectral module first passes through the first 3×3 convolution layer and the LeakyReLU activation function, and then passes through the second 3×3 convolution layer. It is added to the initial input data to generate preliminary features. This preliminary feature is then processed by the first 1×1 convolution layer, the LeakyReLU activation function, the second 1×1 convolution layer and the channel attention module, and then added to the preliminary feature to form the final output feature of the spatial spectral module.

[0010] Optionally, the diffusion model in step S3 includes a first 3×3 convolutional layer, a U-shaped network, a second 3×3 convolutional layer and a time encoder, the first 3×3 convolutional layer, the U-shaped network and the second 3×3 convolutional layer are connected in sequence, the time encoder is connected in parallel with the U-shaped network, and the network structure of the U-shaped network includes three parts: an encoder, a bottom bottleneck layer and a decoder. The encoder part consists of 4 consecutive conditional denoising modules and a downsampling layer, which is used to extract features layer by layer and gradually reduce the spatial dimension of the feature map to complete feature compression. The bottom bottleneck layer consists of 1 conditional denoising module, which is used to connect the encoder and the decoder. The features compressed by the encoder are deeply processed. The decoder consists of 4 consecutive conditional denoising modules and upsampling layers, which are used to restore the spatial dimensions of the feature map layer by layer. The decoder is directly connected to the corresponding encoder layer through jump connections. The time encoder includes a sinusoidal encoding operation, a multi-layer perceptron MLP composed of two linear layers and a SiLU activation function, and a reshaping operation. The time step t input to the time encoder is first sinusoidally encoded, and then the encoded result is input into the multi-layer perceptron for processing. Finally, the shape of the time embedding is adjusted through the reshaping operation to achieve the effective fusion of time information and image features in the subsequent implementation.

[0011] Optionally, the processing of the input features by the conditional denoising module includes: the features of the input conditional denoising module are sequentially subjected to layer normalization, 1×1 convolution layer, 3×3 deep convolution layer, and simple gating unit to generate first-level features; the first-level features are subjected to a lightweight channel attention module and then to a 1×1 convolution layer to obtain second-level features; the second-level features and the features of the input conditional denoising module are added to obtain intermediate features; the intermediate features are further subjected to layer normalization, 1×1 convolution layer, simple gating unit, and 1×1 convolution layer, and then added to the intermediate features to generate output features of the conditional denoising module; The simple gating unit processes the input features by: dividing the input data into two parts along the channel direction, and performing element-by-element multiplication on the two divided sub-data; the lightweight channel attention module processes the input features by: applying global average pooling to the input features with a dimension of H×W×C and then transforming the dimensions to convert them into 1×C, then using a one-dimensional convolutional layer with a length of 3 to perform convolution in the dimension of C, transforming the weights obtained after the convolution and then multiplying them element-by-element with the input features of the lightweight channel attention module to obtain the output features of the lightweight channel attention module.

[0012] Optionally, before step S1, the method further includes training a hyperspectral image super-resolution network: S101, pre-training an autoencoder network consisting of an encoder network and a decoder network; S102, combining the encoder network and the decoder network in the pre-trained autoencoder network, using a multi-scale conditional encoder to obtain conditional information, using the conditional information in the low-dimensional residual latent space to guide the diffusion model training, and finally completing the training of the entire hyperspectral image super-resolution network.

[0013] Optionally, in step S101, when pre-training the autoencoder network composed of the encoder network and the decoder network, the step includes training the autoencoder network using the training set, and determining the optimal network parameters of the autoencoder network when the loss function is minimized by optimizing a preset first loss function, wherein the function expression of the first loss function is: , , , , in, is the preset first loss function, is the absolute value error loss, is the spectral angle loss, is the spectral gradient loss, and is the weight constant, are the total height, total width and total number of bands of high-resolution hyperspectral images, represents the high-resolution hyperspectral reference image in the training set, Indicates that Input the high-resolution hyperspectral reconstructed image obtained by the autoencoder network, For high-resolution hyperspectral reconstruction The pixel value at For high-resolution hyperspectral reference images The pixel value at are the height coordinate, width coordinate and band coordinate of the high-resolution hyperspectral image, For high-resolution hyperspectral reconstruction The spectral gradient at For high-resolution hyperspectral reference images spectral gradient at; in step S102, when the multi-scale conditional encoder is used to obtain conditional information and the conditional information is used to guide the diffusion model training in the low-dimensional residual latent space, the expression of the second loss function used is: , In the above formula, is the second loss function, The high-resolution hyperspectral reference image predicted by the diffusion model at the i-th time step The noise at The high-resolution hyperspectral reference image added for the i-th time step The noise at time step i is between 1 and 2000.

[0014] In addition, the present invention also provides a hyperspectral image super-resolution system based on a latent space diffusion model, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the hyperspectral image super-resolution method based on the latent space diffusion model.

[0015] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the hyperspectral image super-resolution method based on the latent space diffusion model through a processor.

[0016] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the hyperspectral image super-resolution method based on the latent space diffusion model through a processor.

[0017] Compared with the prior art, the present invention has the following advantages: 1. In view of the problems of large computing resource consumption and slow reasoning speed in the current diffusion model during training and reasoning, the present invention designs a simple and efficient diffusion model, which greatly reduces resource requirements, significantly improves reasoning speed while maintaining high generation quality, breaks through the limitations of traditional models in resource utilization and reasoning efficiency, and improves the practicality of the system. 2. The present invention proposes a multi-scale conditional encoder, which can make full use of prior knowledge and enhance the accuracy of the generation process. The multi-scale conditional encoder integrates prior knowledge across multiple scales to more accurately guide the diffusion process, significantly improves the authenticity and consistency of the generated image, and effectively avoids common model defects such as over-smoothing. 3. In view of the possible loss of details in the latent space conversion process, the present invention designs a simple and efficient grouped autoencoder, which effectively minimizes the loss that may occur in the latent space conversion process, ensuring that the texture and edge details of the generated image are rich and natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0019] Figure 2 Schematic diagram of the network structure of the hyperspectral image super-resolution network in an embodiment of the present invention.

[0020] Figure 3 Schematic diagram of the encoder network structure of the autoencoder in an embodiment of the present invention.

[0021] Figure 4 Schematic diagram of the decoder network structure of the autoencoder in an embodiment of the present invention.

[0022] Figure 5 Schematic diagram of the network structure of the spatial-spectral prior module in an embodiment of the present invention. The multi-scale conditional encoder is composed of four spatial-spectral prior modules connected in series.

[0023] Figure 6 Schematic diagram of the network structure of the diffusion model in an embodiment of the present invention.

[0024] Figure 7 Schematic diagram of the network structure of the conditional denoising module in an embodiment of the present invention.

[0025] Figure 8 : This is a schematic diagram of the super-resolution results of the Chikusei dataset image using different super-resolution methods in the experiment of an embodiment of the present invention, wherein (a) is a low-resolution four-fold double-triple interpolation hyperspectral image, (b) is a four-fold super-resolution image using the 3DFCNN method, (c) is a four-fold super-resolution image using the VDSR method, (d) is a four-fold super-resolution image using the EDSR method, (e) is a four-fold super-resolution image using the MCNet method, (f) is a four-fold super-resolution image using the EUNet method, (g) is a four-fold super-resolution image using the method of this embodiment, and (h) is a high-resolution hyperspectral reference image.

[0026] Fig. 9 : This is a schematic diagram of the super-resolution results of the Houton dataset image using different super-resolution methods in the experiment of an embodiment of the present invention, wherein (a) is a low-resolution four-fold double-triple interpolation hyperspectral image, (b) is a four-fold super-resolution image using the 3DFCNN method, (c) is a four-fold super-resolution image using the VDSR method, (d) is a four-fold super-resolution image using the EDSR method, (e) is a four-fold super-resolution image using the MCNet method, (f) is a four-fold super-resolution image using the EUNet method, (g) is a four-fold super-resolution image using the method of this embodiment, and (h) is a high-resolution hyperspectral reference image.

[0027] Fig.10 : This is a schematic diagram of the super-resolution results of the Pavia Center dataset image using different super-resolution methods in the experiment of an embodiment of the present invention, wherein (a) is a low-resolution four-fold double-triple interpolation hyperspectral image, (b) is a four-fold super-resolution image using the 3DFCNN method, (c) is a four-fold super-resolution image using the VDSR method, (d) is a four-fold super-resolution image using the EDSR method, (e) is a four-fold super-resolution image using the MCNet method, (f) is a four-fold super-resolution image using the EUNet method, (g) is a four-fold super-resolution image using the method of this embodiment, and (h) is a high-resolution hyperspectral reference image. DETAILED DESCRIPTION

[0028] The present invention aims to solve the problems of over-smoothing and lack of texture detail information in the reconstructed images of traditional hyperspectral image super-resolution methods based on convolutional neural networks, as well as the challenges faced when applying diffusion models to hyperspectral image super-resolution. The present invention uses a lightweight diffusion model to efficiently generate high-frequency residual detail information, speed up image generation, reduce computing resource requirements, and improve the practicality of the diffusion model to obtain excellent super-resolution effects.

[0029] like Figure 1 and Figure 2 As shown, the hyperspectral image super-resolution method based on the latent space diffusion model of this embodiment includes using a hyperspectral image super-resolution network composed of an encoder network, a multi-scale conditional encoder, a diffusion model and a decoder network to obtain a final high-resolution hyperspectral super-resolution image: S1, the low-resolution hyperspectral image is up-sampled and grouped according to the number of bands; S2, each group of images is sequentially reduced in dimension in the spectral dimension through a pre-trained encoder network to obtain low-dimensional prior latent space information and encoded using a multi-scale conditional encoder to obtain conditional information; S3, the low-dimensional prior latent space information is concatenated with the Gaussian noise channel and then the Gaussian noise is converted into high-frequency residual latent space information using a diffusion model under the guidance of the conditional information; S4, the high-frequency residual latent space information is added to the low-dimensional prior latent space information to obtain low-dimensional latent space information with high-frequency details; S5, each group of low-dimensional latent space information is decoded and restored through a pre-trained decoder network to obtain the final high-resolution hyperspectral super-resolution image.

[0030] When upsampling the low-resolution hyperspectral image in step S1, the required upsampling method can be selected as needed. For example, as an optional implementation, the double-triple interpolation algorithm is used for interpolation upsampling in this embodiment. When grouping according to the number of bands after upsampling the low-resolution hyperspectral image in step S1, the grouping strategy used divides the hyperspectral image into several groups according to the number of bands, and there is some overlap between the groups; Figure 2 As shown, the number of groups in this embodiment is N, and a hyperspectral image obtains group 1 to group N groups. After the high-resolution hyperspectral reference image and the double-three interpolation hyperspectral image are processed by the encoder, the two are subtracted to obtain low-dimensional residual latent space information. Since the denoising and denoising process of the diffusion model depends on the noise with the same dimension as the original data, it is inefficient to train directly on the original data. Therefore, in this embodiment, an autoencoder is used to map the pixel space of the hyperspectral image to a low-dimensional latent space, which effectively reduces the demand for computing resources. The complex data distribution increases the difficulty of training the diffusion model, and the residual latent space is more in line with the Gaussian distribution, which is conducive to reducing the difficulty of training the diffusion model, so that it can focus more on learning spatial spectral detail information. The training and reasoning process of the model in this low-dimensional residual latent space can significantly reduce the learning complexity and computational complexity.

[0031] like Figure 3 As shown, the encoder network in step S2 includes a 5×5 convolution layer, multiple residual modules (which can be selected according to actual needs, specifically 3 in this embodiment) and a 5×5 convolution layer connected in sequence. The data input to the encoder network first passes through a 5×5 convolution layer to obtain preliminary features. The preliminary features pass through multiple residual modules in sequence and then are added to the preliminary features. Finally, the output of the final encoder network is obtained by passing through a 5×5 convolution layer. like Figure 4 As shown, the decoder network includes a 5×5 convolution layer, a first group of multiple (can be selected according to actual needs, specifically 3 in this embodiment) residual modules, a 5×5 convolution layer, a 5×5 convolution layer, a second group of multiple (can be selected according to actual needs, specifically 3 in this embodiment) residual modules and a 5×5 convolution layer connected in sequence. The data input to the decoder network first passes through a 5×5 convolution layer to obtain preliminary features, the preliminary features pass through the first group of multiple residual modules in sequence and then are added to the preliminary features, and finally pass through a 5×5 convolution layer to obtain deep feature information, and the multiple deep feature information are channel-joined to obtain deep image features. The deep image features first pass through a 5×5 convolution layer to obtain reconstructed image features, and the reconstructed image features pass through the second group of multiple residual modules in sequence and then are added to the reconstructed image features. Finally, a 5×5 convolution layer is passed to obtain reconstructed image information, and the reconstructed image information is added to the deep image features to obtain the output of the decoder network.

[0032] In this embodiment, the residual module includes a 5×5 convolution layer, a LeakyReLU activation function and a 5×5 convolution layer. The data input to the residual module is processed by a 5×5 convolution layer, a LeakyReLU activation function and a 5×5 convolution layer in sequence, and then added to the data input to the residual module to obtain the output data of the residual module.

[0033] like Figure 5 As shown, the multi-scale conditional encoder in step S2 includes a 3×3 convolutional layer and multiple spatial spectral prior modules (which can be selected according to actual needs, specifically 4 in this embodiment) connected in sequence. The data input to the multi-scale conditional encoder is first extracted with preliminary features through the 3×3 convolutional layer, and then the preliminary features are sequentially passed through multiple spatial spectral prior modules to obtain conditional information of multiple scales.

[0034] like Figure 5As shown, the spatial spectrum prior module includes a 3×3 convolutional layer, multiple (can be selected according to actual needs, specifically 3 in this embodiment) spatial modules, multiple (can be selected according to actual needs, specifically 3 in this embodiment) spatial spectrum modules and a 3×3 downsampling convolutional layer connected in sequence. The data input to the spatial spectrum prior module first passes through the 3×3 convolutional layer to extract the initial depth feature; the initial depth feature passes through the three spatial modules in sequence and is added to the initial depth feature to generate an intermediate depth feature; the intermediate depth feature is then processed by multiple spatial spectrum modules and added to the intermediate depth feature and the initial depth feature to obtain an enhanced depth feature. Finally, the enhanced depth feature passes through the 3×3 downsampling convolutional layer to obtain the output conditional information.

[0035] like Figure 5 As shown in the figure, the spatial module includes a 3×3 convolutional layer, a LeakyReLU activation function, a 3×3 convolutional layer, another 3×3 convolutional layer, a LeakyReLU activation function and a final 3×3 convolutional layer connected in sequence. The data input to the spatial module first passes through the first 3×3 convolutional layer and the LeakyReLU activation function in sequence, and then passes through the second 3×3 convolutional layer for processing, and is added to the initial input data to obtain preliminary processing features. Subsequently, these preliminary processing features are processed by the third 3×3 convolutional layer, the LeakyReLU activation function and the final 3×3 convolutional layer in sequence, and then added to the preliminary processing features to form the final output features of the spatial module.

[0036] like Figure 5 As shown in the figure, the spatial spectrum module includes a 3×3 convolution layer, a LeakyReLU activation function, a 3×3 convolution layer, a 1×1 convolution layer, a LeakyReLU activation function, a 1×1 convolution layer and a channel attention module connected in sequence. The data input to the spatial spectrum module first passes through the first 3×3 convolution layer and the LeakyReLU activation function in sequence, and then passes through the second 3×3 convolution layer. It is processed and added to the initial input data to generate preliminary features. This preliminary feature is then processed by the first 1×1 convolution layer, the LeakyReLU activation function, the second 1×1 convolution layer and the channel attention module, and then added to the preliminary feature to form the final output feature of the spatial spectrum module.

[0037] like Figure 6As shown, the diffusion model in step S3 includes a first 3×3 convolutional layer, a U-shaped network, a second 3×3 convolutional layer and a time encoder. The first 3×3 convolutional layer, the U-shaped network and the second 3×3 convolutional layer are connected in sequence, and the time encoder is connected to the U-shaped network in parallel. The U-shaped network is composed of 9 conditional denoising blocks. Due to the use of conditional denoising modules, the model has fewer parameters and fast training and inference speeds. In this embodiment, the network structure of the U-shaped network includes three parts: an encoder, a bottom bottleneck layer and a decoder. The encoder part consists of 4 consecutive conditional denoising modules and a downsampling layer, which is used to extract features layer by layer and gradually reduce the spatial dimension of the feature map to complete feature compression. The bottom bottleneck layer consists of 1 conditional denoising module, which connects the encoder and the decoder to perform deep processing on the features compressed by the encoder. The decoder consists of 4 consecutive conditional denoising modules and an upsampling layer, which is used to restore the spatial dimension of the feature map layer by layer. The decoder is directly connected to the corresponding encoder layer through a skip connection (used to achieve feature fusion and effective use of multi-scale information, thereby significantly improving the accuracy and robustness of the model), see Figure 6 It can be seen that the output ends of the 1st to 4th conditional denoising modules are all connected to a downsampling module, the output ends of the 5th to 8th conditional denoising modules are all connected to an upsampling module, and the output of the 5th conditional denoising module is added to the output element of the 4th conditional denoising module as the input of the 6th conditional denoising module, the output of the 6th conditional denoising module is added to the output element of the 3rd conditional denoising module as the input of the 7th conditional denoising module, the output of the 7th conditional denoising module is added to the output element of the 2nd conditional denoising module as the input of the 8th conditional denoising module, and the output of the 8th conditional denoising module is added to the output element of the 1st conditional denoising module as the input of the 9th conditional denoising module.

[0038] like Figure 6 As shown in the figure, the temporal encoder includes a sinusoidal encoding operation, a multi-layer perceptron MLP composed of two linear layers and a SiLU activation function, and a reshaping operation (Reshape). The time step t input to the temporal encoder is first sinusoidally encoded, and then the encoded result is input into the multi-layer perceptron for processing. Finally, the shape of the temporal embedding is adjusted through the reshaping operation (Reshape) to achieve the effective fusion of temporal information and image features.

[0039] like Figure 7As shown in Figure 1, the processing of the input features by the conditional denoising module includes: the features of the input conditional denoising module are sequentially processed by layer normalization, 1×1 convolution layer, 3×3 deep convolution layer, simple gating unit (Simple Gate) generates the first-level features; the first-level features pass through the lightweight channel attention module and then pass through a 1×1 convolutional layer to obtain the second-level features; the second-level features and the features of the input conditional denoising module are added to obtain the intermediate features; the intermediate features are further processed by layer normalization, 1×1 convolutional layer, simple gating unit and 1×1 convolutional layer, and then added to the intermediate features to generate the output features of the conditional denoising module; the processing of the input features by the simple gating unit includes: dividing the input data into two parts along the channel direction, and performing element-by-element multiplication operation on the two divided sub-data; the processing of the input features by the lightweight channel attention module includes: applying global average pooling to the input features of the dimension of H×W×C and then performing dimensional transformation to convert them into 1×C, then using a one-dimensional convolutional layer with a length of 3 to perform convolution in the dimension of C, and performing dimensional transformation on the weights obtained after the convolution and then multiplying them element-by-element with the input features of the lightweight channel attention module to obtain the output features of the lightweight channel attention module.

[0040] In this embodiment, before step S1, the process further includes training a hyperspectral image super-resolution network: S101, pre-training an autoencoder network consisting of an encoder network and a decoder network; S102, combining the encoder network and the decoder network in the pre-trained autoencoder network, using a multi-scale conditional encoder to obtain conditional information, using the conditional information in the low-dimensional residual latent space to guide the diffusion model training, and finally completing the training of the entire hyperspectral image super-resolution network.

[0041] When the autoencoder network composed of the encoder network and the decoder network is pre-trained in step S101 of this embodiment, the autoencoder network is determined when the loss function is minimized by optimizing the loss function when training with the training set; wherein the training set is composed of high-resolution hyperspectral reference images; the loss function refers to the error between the high-resolution hyperspectral reconstructed image obtained by inputting the high-resolution hyperspectral reference image into the autoencoder network and the corresponding high-resolution hyperspectral reference image. Specifically, when the autoencoder network composed of the encoder network and the decoder network is pre-trained in step S101 of this embodiment, it includes training the autoencoder network with the training set, and determining the optimal network parameters of the autoencoder network when the loss function is minimized by optimizing the preset first loss function, and the loss function uses L1 error, spectral angle loss and spectral gradient loss for weighted summation. Specifically, the functional expression of the first loss function is: , , , , in, is the preset first loss function, is the absolute value error loss, is the spectral angle loss, is the spectral gradient loss, and is a weight constant (in this embodiment, the values ​​are 0.5 and 0.3 respectively), are the total height, total width and total number of bands of high-resolution hyperspectral images, represents the high-resolution hyperspectral reference image in the training set, Indicates that Input the high-resolution hyperspectral reconstructed image obtained by the autoencoder network, For high-resolution hyperspectral reconstruction The pixel value at For high-resolution hyperspectral reference images The pixel value at are the height coordinate, width coordinate and channel coordinate of the high-resolution hyperspectral image, For high-resolution hyperspectral reconstruction The spectral gradient at For high-resolution hyperspectral reference images The spectral gradient at; the training set consists of pairs of low-resolution images and corresponding high-resolution hyperspectral reference images. The first stage of autoencoder training only uses high-resolution hyperspectral reference images (ground-truth). The process is "high-resolution hyperspectral reference image → autoencoder → high-resolution hyperspectral reconstructed image". Then the high-resolution hyperspectral reference image and the high-resolution hyperspectral reconstructed image are lost. Because this process is from the same resolution to the same resolution, I call the resulting image "high-resolution hyperspectral reconstructed image" rather than "high-resolution hyperspectral super-resolution image".

[0042] The diffusion model is determined by optimizing the loss function when the training set is used for training, so that the loss function is minimized; wherein the training set includes a low-resolution hyperspectral image and a corresponding high-resolution hyperspectral reference image; the loss function refers to the error between the noise predicted by the diffusion model and the corresponding added noise after the high-resolution hyperspectral reference image and the low-resolution hyperspectral image are sent into the entire model according to the multi-scale conditional encoder to obtain conditional information and the conditional information is used to guide the diffusion model training in the low-dimensional residual latent space. In step S102 of this embodiment, when the multi-scale conditional encoder is used to obtain conditional information and the conditional information is used to guide the diffusion model training in the low-dimensional residual latent space, the expression of the second loss function used is: , In the above formula, is the second loss function (L1 error), The high-resolution hyperspectral reference image predicted by the diffusion model at the i-th time step The noise at The high-resolution hyperspectral reference image added for the i-th time step The noise at the time step i is between 1 and 2000. In the second stage of the diffusion model training process, noise is added to the reference image. The network predicts the noise added to the high-resolution hyperspectral reference image, and the low-resolution hyperspectral image is only used as conditional information to guide the model to denoise. After the model is trained, the inference process only needs to use the low-resolution hyperspectral image to guide the diffusion model to convert the Gaussian noise image into a high-resolution hyperspectral super-resolution image.

[0043] In this embodiment, the Chikusei dataset, Houston dataset and Pavia Center dataset are used to verify the practicality and effectiveness of the hyperspectral image super-resolution method based on the latent space diffusion model in this embodiment. This embodiment gives an experimental process. The low-resolution hyperspectral image in the experiment is obtained by downsampling the actual hyperspectral image by double-triple interpolation, and the original actual hyperspectral image is used as the high-resolution hyperspectral reference image. The high-resolution hyperspectral reference image and the corresponding low-resolution hyperspectral image are composed of a training set and a test set. The method in this embodiment is mainly compared with a traditional super-resolution method of double-triple interpolation and five deep learning methods of 3DFCNN, VDSR, EDSR, MCNet, and EUNet. The results obtained are as follows. Figure 8 , Fig. 9 and Fig.10 shown. Figure 8Among them, (a) is a low-resolution four-fold double-triple interpolation hyperspectral image, (b) is a four-fold super-resolution image of the 3DFCNN method, (c) is a four-fold super-resolution image of the VDSR method, (d) is a four-fold super-resolution image of the EDSR method, (e) is a four-fold super-resolution image of the MCNet method, (f) is a four-fold super-resolution image of the EUNet method, (g) is a four-fold super-resolution image using the method of this embodiment, and (h) is a high-resolution hyperspectral reference image. Fig. 9 Among them, (a) is a low-resolution four-fold double-triple interpolation hyperspectral image, (b) is a four-fold super-resolution image of the 3DFCNN method, (c) is a four-fold super-resolution image of the VDSR method, (d) is a four-fold super-resolution image of the EDSR method, (e) is a four-fold super-resolution image of the MCNet method, (f) is a four-fold super-resolution image of the EUNet method, (g) is a four-fold super-resolution image using the method of this embodiment, and (h) is a high-resolution hyperspectral reference image. Fig.10 In the figure, (a) is a low-resolution four-fold bi-triple interpolation hyperspectral image, (b) is a four-fold super-resolution image using the 3DFCNN method, (c) is a four-fold super-resolution image using the VDSR method, (d) is a four-fold super-resolution image using the EDSR method, (e) is a four-fold super-resolution image using the MCNet method, (f) is a four-fold super-resolution image using the EUNet method, (g) is a four-fold super-resolution image using the method of this embodiment, and (h) is a high-resolution hyperspectral reference image. See Figure 8 , Fig. 9 and Fig.10 It can be seen that compared with the high-resolution hyperspectral reference image, Figure 8 (a)~(f) in the figure have some details lost. Figure 8 From (g) in FIG. 1 , it can be seen that the spectrum of the method described in the embodiment of the present invention is well maintained. Fig. 9 (a)~(f) and Fig.10 The edges of the super-resolution images in (a)~(f) are blurred and the details are seriously lost. Fig. 9 (g) and Fig.10 (g) in FIG. 3 shows that the method described in this embodiment has good detail preservation. It can be seen from the super-resolution image that the super-resolution image of the method described in this embodiment has little difference with the high-resolution hyperspectral reference image, and the spectrum preservation and detail injection are good.

[0044] By analyzing the super-resolution image, we can have an intuitive understanding of the super-resolution result. In this embodiment, objective evaluation indicators are used to further evaluate the image super-resolution result. The method described in this embodiment evaluates the super-resolution result through six indicators: MPSNR (average peak signal-to-noise ratio), MSSIM (average structural similarity), CC (correlation coefficient), RMSE (root mean square error), SAM (spectral angle mapping), and ERGAS (relative global dimensionless error). The optimal values ​​of RMSE, SAM and ERGAS are 0, the optimal values ​​of MSSIM and CC are 1, and the optimal value of MPSNR is positive infinity. The objective evaluation indicators of the super-resolution results of different methods in different data sets are shown in Tables 1, 2, and 3.

[0045] Table 1: Objective evaluation indicators of super-resolution results of Chikusei dataset test images

[0046] Table 2: Objective evaluation indicators of super-resolution results of test images in the Houston dataset

[0047] Table 3: Objective evaluation indicators of super-resolution results of Pavia Center dataset test images

[0048] It can be seen from Tables 1 to 3 that the method described in this embodiment outperforms other methods in all indicators on the Chikusei, Houston and Chikusei datasets. Table 4 shows the average time required for inferring an image with a resolution of 128×128 when the method performs a 4x super-resolution experiment on the Houston dataset.

[0049] Table 4: Analysis of efficiency indicators of super-resolution of test images in the Houston dataset

[0050] Figure 4 The efficiency of this embodiment is proved from the reasoning time.

[0051] The ablation experiment in this embodiment is a 4x super-resolution experiment performed on the Houston dataset. The experimental results are shown in Tables 5, 6 and 7.

[0052] Table 5: Effectiveness of low-dimensional residual latent space

[0053] It can be seen from Table 5 that the best effect is achieved by adopting the method of this embodiment, where “w / o Auto-encoder” means removing the auto-encoder, and “w / o residual space” means removing the residual space.

[0054] Table 6: Effectiveness of multi-scale conditional encoder and efficient diffusion model

[0055] Table 6 shows that the multi-scale encoder combined with the efficient diffusion model in this embodiment not only achieves the best super-resolution effect, but also significantly reduces the number of model parameters. f Encoder " represents a multi-scale conditional encoder, "SEDNet" represents a highly efficient diffusion model network, and "UNet" represents a commonly used network structure for diffusion models.

[0056] Table 7 Effectiveness of conditional denoising module

[0057] In Table 7, "NAFNet" is used as the benchmark for module design, "Simple CA→ECA" means replacing the simple channel attention module (Simple CA) in NAFNet with the efficient channel attention module (ECA), and "ECA→SECA" means replacing the efficient channel attention module with the lightweight channel attention module, which is an improvement of this embodiment. The baseline and Model-1 are two comparison models of this embodiment. Table 7 further verifies the effectiveness of the conditional denoising module of this embodiment. The diffusion model composed of the conditional denoising module not only has a small number of parameters, but also achieves excellent super-resolution effect.

[0058] In summary, the present embodiment includes grouping low-resolution hyperspectral images after upsampling, encoding each group of images in turn through the encoder network dimensionality reduction encoding to obtain low-dimensional prior latent space information, inputting the low-dimensional prior latent space information into the multi-scale conditional encoder encoding to obtain conditional information, splicing the low-dimensional prior latent space information with the Gaussian noise channel, and using the diffusion model to convert the Gaussian noise into high-frequency residual latent space information under the guidance of the conditional information, and adding it to the low-dimensional prior latent space information to obtain low-dimensional latent space information with high-frequency details, and then decoding and restoring it through the pre-trained decoder network to obtain the final high-resolution hyperspectral super-resolution image. The present embodiment adopts a grouping strategy, combines the autoencoder to transform the hyperspectral pixel space into a low-dimensional residual latent space, and uses a lightweight diffusion model to efficiently generate high-frequency residual detail information. The method of the present embodiment can not only speed up the image generation speed, but also greatly reduce the computing resource requirements and significantly improve the practicality of the diffusion model. The method of the present embodiment obtains excellent super-resolution effects and can be widely used in remote sensing image processing, target detection, target recognition and other fields.

[0059] In addition, this embodiment also provides a hyperspectral image super-resolution system based on a latent space diffusion model, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the hyperspectral image super-resolution method based on the latent space diffusion model.

[0060] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the hyperspectral image super-resolution method based on the latent space diffusion model through a processor.

[0061] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the hyperspectral image super-resolution method based on the latent space diffusion model through a processor.

[0062] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0063] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A hyperspectral image super-resolution method based on latent space diffusion model, characterized in that: The method comprises the following steps: using a hyperspectral image super-resolution network composed of an encoder network, a multi-scale conditional encoder, a diffusion model and a decoder network to obtain a final high-resolution hyperspectral super-resolution image using a low-resolution hyperspectral image: S1, after upsampling the low-resolution hyperspectral image, the image is grouped according to the number of bands; S2, each group of images is sequentially reduced in dimension in the spectral dimension through a pre-trained encoder network to obtain low-dimensional prior latent space information and encoded using a multi-scale conditional encoder to obtain conditional information; S3, after concatenating the low-dimensional prior latent space information with a Gaussian noise channel, a diffusion model is used to convert the Gaussian noise into high-frequency residual latent space information under the guidance of the conditional information; S4, the high-frequency residual latent space information is added to the low-dimensional prior latent space information to obtain low-dimensional latent space information with high-frequency details; S5, each group of low-dimensional latent space information is decoded and restored through a pre-trained decoder network to obtain the final high-resolution hyperspectral super-resolution image.

2. The hyperspectral image super-resolution method based on latent space diffusion model according to claim 1, characterized in that: The encoder network in step S2 includes a 5×5 convolutional layer, multiple residual modules and a 5×5 convolutional layer connected in sequence. The data input to the encoder network first passes through a 5×5 convolutional layer to obtain preliminary features. The preliminary features pass through multiple residual modules in sequence and are then added to the preliminary features. Finally, a 5×5 convolutional layer is passed to obtain the final output of the encoder network. The decoder network includes a 5×5 convolutional layer, a first group of multiple residual modules, a 5×5 convolutional layer, a 5×5 convolutional layer, a second group of multiple residual modules and a 5×5 convolutional layer connected in sequence. The data input to the decoder network first passes through a 5×5 convolutional layer to obtain preliminary features. The preliminary features pass through the first group of multiple residual modules in sequence and are then added to the preliminary features. Finally, a 5×5 convolutional layer is passed to obtain the final output of the encoder network. The convolution layer obtains deep feature information, channels of multiple deep feature information are spliced ​​to obtain deep image features, the deep image features are first passed through a 5×5 convolution layer to obtain reconstructed image features, the reconstructed image features are sequentially passed through a second group of multiple residual modules and then added to the reconstructed image features, and finally passed through a 5×5 convolution layer to obtain reconstructed image information, and the reconstructed image information is added to the deep image features to obtain the output of the decoder network; the residual module includes a 5×5 convolution layer, a LeakyReLU activation function and a 5×5 convolution layer, and the data input to the residual module is sequentially processed by a 5×5 convolution layer, a LeakyReLU activation function and a 5×5 convolution layer, and then added to the data input to the residual module to obtain the output data of the residual module.

3. The hyperspectral image super-resolution method based on latent space diffusion model according to claim 1 is characterized in that: The multi-scale conditional encoder in step S2 includes a 3×3 convolutional layer and a plurality of spatial spectral prior modules connected in sequence. The data input to the multi-scale conditional encoder first passes through the 3×3 convolutional layer to extract preliminary features, and then the preliminary features pass through the plurality of spatial spectral prior modules in sequence to obtain conditional information of a plurality of different scales; the spatial spectral prior module includes a 3×3 convolutional layer, a plurality of spatial modules, a plurality of spatial spectral modules and a 3×3 downsampling convolutional layer connected in sequence. The data input to the spatial spectral prior module first passes through the 3×3 convolutional layer to extract initial depth features; the initial depth features pass through the three spatial modules in sequence and are added to the initial depth features to generate intermediate depth features; The intermediate depth feature is then processed by multiple spatial-spectral modules and added to the intermediate depth feature and the initial depth feature to obtain an enhanced depth feature. Finally, the enhanced depth feature is output through a 3×3 downsampling convolution layer to obtain the output conditional information; the spatial module includes a 3×3 convolution layer, a LeakyReLU activation function, a 3×3 convolution layer, another 3×3 convolution layer, a LeakyReLU activation function and a final 3×3 convolution layer connected in sequence. The data input to the spatial module first passes through the first 3×3 convolution layer and the LeakyReLU activation function in sequence, and then passes through the second 3×3 convolution layer and is added to the initial input data to obtain preliminary processing features. Subsequently, these preliminary processing features are processed in sequence through the third 3×3 convolution layer, the LeakyReLU activation function and the final 3×3 convolution layer, and then added to the preliminary processing features to form the final output features of the spatial module; The spatial spectrum module includes a 3×3 convolution layer, a LeakyReLU activation function, a 3×3 convolution layer, a 1×1 convolution layer, a LeakyReLU activation function, a 1×1 convolution layer and a channel attention module connected in sequence. The data input to the spatial spectrum module first passes through the first 3×3 convolution layer and the LeakyReLU activation function in sequence, and then passes through the second 3×3 convolution layer. It is added to the initial input data to generate preliminary features. This preliminary feature is then processed by the first 1×1 convolution layer, the LeakyReLU activation function, the second 1×1 convolution layer and the channel attention module, and then added to the preliminary feature to form the final output feature of the spatial spectrum module.

4. The hyperspectral image super-resolution method based on latent space diffusion model according to claim 1, characterized in that: The diffusion model in step S3 includes a first 3×3 convolutional layer, a U-shaped network, a second 3×3 convolutional layer and a time encoder. The first 3×3 convolutional layer, the U-shaped network and the second 3×3 convolutional layer are connected in sequence. The time encoder is connected in parallel with the U-shaped network. The network structure of the U-shaped network includes three parts: an encoder, a bottom bottleneck layer and a decoder. The encoder part consists of four consecutive conditional denoising modules and a downsampling layer, which is used to extract features layer by layer and gradually reduce the spatial dimension of the feature map to complete feature compression. The bottom bottleneck layer consists of a conditional denoising module, which is used to connect the encoder and the decoder. The compressed features of the encoder are deeply processed. The decoder consists of 4 consecutive conditional denoising modules and upsampling layers, which are used to restore the spatial dimensions of the feature map layer by layer. The decoder is directly connected to the corresponding encoder layer through a jump connection. The time encoder includes a sinusoidal encoding operation, a multi-layer perceptron MLP composed of two linear layers and a SiLU activation function, and a reshaping operation. The time step t input to the time encoder is first sinusoidally encoded, and then the encoded result is input into the multi-layer perceptron for processing. Finally, the shape of the time embedding is adjusted through the reshaping operation to achieve the effective fusion of time information and image features in the subsequent implementation.

5. The hyperspectral image super-resolution method based on latent space diffusion model according to claim 4 is characterized in that: The processing of the input features by the conditional denoising module includes: the features of the input conditional denoising module are sequentially subjected to layer normalization, 1×1 convolution layer, 3×3 deep convolution layer, and simple gating unit to generate first-level features; the first-level features are subjected to a lightweight channel attention module and then to a 1×1 convolution layer to obtain second-level features; the second-level features and the features of the input conditional denoising module are added to obtain intermediate features; the intermediate features are further subjected to layer normalization, 1×1 convolution layer, simple gating unit, and 1×1 convolution layer, and then added to the intermediate features to generate output features of the conditional denoising module; The simple gating unit processes the input features including: dividing the input data into two parts along the channel direction, and performing element-by-element multiplication on the two divided sub-data; the lightweight channel attention module processes the input features including: applying global average pooling to the input features with a dimension of H×W×C and then transforming the dimensions to convert them into 1×C, then using a one-dimensional convolutional layer with a length of 3 to perform convolution in the dimension of C, transforming the weights obtained after the convolution and then multiplying them element-by-element with the input features of the lightweight channel attention module to obtain the output features of the lightweight channel attention module.

6. The hyperspectral image super-resolution method based on latent space diffusion model according to claim 1, characterized in that: Before step S1, it also includes training the hyperspectral image super-resolution network: S101, pre-training an autoencoder network consisting of an encoder network and a decoder network; S102, combining the encoder network and the decoder network in the pre-trained autoencoder network, using a multi-scale conditional encoder to obtain conditional information, using the conditional information in the low-dimensional residual latent space to guide the diffusion model training, and finally completing the training of the entire hyperspectral image super-resolution network.

7. The hyperspectral image super-resolution method based on latent space diffusion model according to claim 6 is characterized in that: When pre-training the autoencoder network composed of the encoder network and the decoder network in step S101, it includes training the autoencoder network using the training set, and determining the optimal network parameters of the autoencoder network when the loss function is minimized by optimizing a preset first loss function, and the function expression of the first loss function is: ， ， ， ， in, is the preset first loss function, is the absolute value error loss, is the spectral angle loss, is the spectral gradient loss, and is the weight constant, are the total height, total width and total number of bands of high-resolution hyperspectral images, represents the high-resolution hyperspectral reference image in the training set, Indicates that Input the high-resolution hyperspectral reconstructed image obtained by the autoencoder network, For high-resolution hyperspectral reconstruction The pixel value at For high-resolution hyperspectral reference images The pixel value at are the height coordinate, width coordinate and band coordinate of the high-resolution hyperspectral image, For high-resolution hyperspectral reconstruction The spectral gradient at For high-resolution hyperspectral reference images spectral gradient at; in step S102, when the multi-scale conditional encoder is used to obtain conditional information and the conditional information is used to guide the diffusion model training in the low-dimensional residual latent space, the expression of the second loss function used is: ， In the above formula, is the second loss function, The high-resolution hyperspectral reference image predicted by the diffusion model at the i-th time step The noise at The high-resolution hyperspectral reference image added for the i-th time step The noise at time step i is between 1 and 2000.

8. A hyperspectral image super-resolution system based on a latent space diffusion model, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the hyperspectral image super-resolution method based on latent space diffusion model as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the hyperspectral image super-resolution method based on latent space diffusion model described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the hyperspectral image super-resolution method based on latent space diffusion model described in any one of claims 1 to 7 through a processor.

Images