Hyperspectral image super-resolution method and device based on physical diffusion model

Abstract

The invention discloses a hyperspectral image super-resolution method and device based on a physical diffusion model. The method comprises the following steps: acquiring a low-resolution hyperspectral image and a corresponding high-resolution multispectral image; extracting edge information and semantic information from the high-resolution multispectral image; and inputting the low-resolution hyperspectral image, the edge information and the semantic information into a physical diffusion model, and reconstructing to obtain a high-resolution hyperspectral image. According to the invention, through a physical information guiding sub-network design, a physical information extraction and fusion method and a physical model guiding sub-network design, a physical mechanism of hyperspectral imaging and a scene physical attribute are successfully and deeply fused into a core iteration process of a diffusion model; the fusion significantly improves the spatial detail definition, spectral fidelity and overall physical consistency of the high-resolution hyperspectral image, and provides more effective physical guidance information and a more robust physical constraint optimization mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and image reconstruction, and particularly relates to a hyperspectral image super-resolution method and device based on a physical diffusion model. Background Art

[0002] Hyperspectral imaging captures the spectral information of an object using hundreds of continuous bands, providing unique spectral signatures for material identification and applications in precision agriculture, environmental monitoring, and other fields. However, due to the physical limitations of imaging sensors, hyperspectral imaging systems struggle to maintain both high spectral and spatial resolution. This low spatial resolution severely limits performance in scenarios such as small crop classification and mineral microstructure analysis. To address this physical limitation, hyperspectral super-resolution technology has become crucial.

[0003] Traditional hyperspectral image super-resolution technologies are primarily based on the following mathematical theoretical frameworks: In the field of statistical learning, the Bayesian probability model is based on the statistical correlation between the observed data and the target image, achieving image reconstruction through an iterative optimization algorithm; non-negative matrix factorization decouples spectral features from spatial information and adopts a hierarchical optimization strategy for dual-modal reconstruction; CP decomposition technology based on tensor algebra fully utilizes the three-dimensional characteristics of hyperspectral data to achieve joint spatial-spectral feature extraction; and a dual-resolution dictionary mapping system is constructed based on sparse representation methods to achieve resolution improvement through cross-scale feature matching. However, traditional methods rely too much on idealized assumptions and manually designed features, which leads to limited robustness, efficiency, and generalization capabilities in complex real-world scenarios.

[0004] The development of deep learning methods has driven progress in various fields, and hyperspectral image super-resolution has benefited significantly. The application of neural networks, particularly convolutional neural networks (CNNs), transformers, and generative adversarial networks (GANs), in hyperspectral image super-resolution has become relatively mature, but these models still face common and daunting challenges. CNNs exhibit excellent reconstruction performance, but their internal structures often lack transparency and theoretical support. Transformers, while adept at processing long sequences of data, suffer from high model complexity and require extensive training data. GAN models have demonstrated their ability to produce high-quality results, but they suffer from training instability, lack of interpretability, and susceptibility to mode collapse, significantly impacting the quality of super-resolution hyperspectral images.

[0005] In recent years, the denoising diffusion probability model (DDPM) has become renowned for its high-quality generation, strong interpretability, and high flexibility. Therefore, applying DDPM to spectral image super-resolution could be a solution to alleviate the aforementioned model dilemma. For example, Li et al. introduced the Dual Conditional Diffusion Model (DCDM), which comprises two noise prediction networks with different conditional inputs. It is specifically designed to extract image features from low-resolution spectral images and panchromatic images, adapt to the different characteristics of the input images, and reconstruct high-resolution spectral image features for fusion. However, these methods directly combine DDPM with spectral image super-resolution, ignoring the constraints and guidance of the physical properties of spectral images on the reconstruction process. This can lead to deficiencies in the reconstruction results and limit their application in scenarios requiring high-precision quantitative analysis. Summary of the Invention

[0006] In order to solve the problem that the existing hyperspectral image super-resolution methods based on diffusion models do not fully consider the physical characteristics of spectral images, this paper proposes a hyperspectral image super-resolution method and device based on a physical diffusion model, which improves the reconstruction quality by introducing adaptive physical information guidance in the iteration of the diffusion model.

[0007] To achieve the above objectives, the present invention adopts the following technical solutions.

[0008] In a first aspect, the present invention provides a hyperspectral image super-resolution method based on a physical diffusion model, the method comprising:

[0009] Acquiring low-resolution hyperspectral images and the corresponding high-resolution multispectral image ; From high-resolution multispectral images Extract edge information and semantic information ;

[0010] Low-resolution hyperspectral images , edge information , semantic information Input into the physical diffusion model to reconstruct a high-resolution hyperspectral image .

[0011] Preferably, the physical diffusion model includes a forward Markov chain process and a reverse Markov chain process;

[0012] The reverse Markov chain process includes multiple stages; The stage gradient term incorporates the observation constraint into the generated image through the noise prediction network at the current time step The noise prediction network includes a physical information guided spectrum denoising sub-network and a physical model guided spectrum denoising sub-network connected in series.

[0013] Preferably, the training loss function of the physical diffusion model is expressed as:

[0014] Formula (1)

[0015] in represents the high-resolution hyperspectral image label, Represents the reverse Markov chain process in The final stage output of the prediction under stage.

[0016] Preferably, the physical information guides the spectral denoising sub-network The implementation process is:

[0017] Formula (2)

[0018] in Represents the splicing of multiple information in the channel dimension, Represents the mapping function of the physical information guided spectral denoising sub-network, represents a low-resolution hyperspectral image, Represents edge information, Represents semantic information, Indicates the Noisy high-resolution hyperspectral image of the stage.

[0019] Preferably, the physical information guided spectral denoising sub-network uses a Unet network.

[0020] Preferably, the physical model guided spectral denoising subnetwork includes K stages; wherein the implementation process of each stage is:

[0021] Formula (3)

[0022] Formula (4)

[0023] in and is a learnable parameter, For neural networks, ∈[0,K-1], B represents the spatial downsampling operation, S represents the spectral downsampling operation, represents the output of the k+1th stage, Adopt the Noisy high-resolution hyperspectral images of the stage .

[0024] Preferably, the neural network in the physical model-guided spectral denoising sub-network consists of a first convolutional layer, an activation layer, and a second convolutional layer.

[0025] In a second aspect, the present invention provides a hyperspectral image super-resolution device, comprising:

[0026] Data acquisition module, responsible for acquiring low-resolution hyperspectral images and the corresponding high-resolution multispectral image ; From high-resolution multispectral images Extract edge information and semantic information ;

[0027] Image reconstruction module, responsible for converting low-resolution high-spectral images , edge information , semantic information Input into the physical diffusion model to reconstruct a high-resolution hyperspectral image .

[0028] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.

[0029] In a fourth aspect, the present invention provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described is implemented.

[0030] The beneficial effects of the present invention include at least:

[0031] (1) This paper proposes a hyperspectral image super-resolution method based on a physical diffusion model. This method constructs a diffusion model through a physical information guidance subnetwork and a physical model guidance subnetwork, thereby improving the generation quality of high-resolution hyperspectral images while making full use of physical information. Specifically, the physical information guidance subnetwork explicitly uses the edge information and semantic information extracted from the high-resolution multispectral image as conditional input, together with the low-resolution hyperspectral image and the current noisy state. This enables the denoising process to adaptively utilize the inherent spatial structure and semantic category information of the scene, guiding the model to generate high-quality spatial details that are more consistent with the spatial structure and semantic constraints of the real scene. The physical model guidance subnetwork solves the physical imaging model by unfolding the optimization and fusing the learnable parameters and neural network. In each iteration, the reconstruction result is forcibly constrained to a feasible solution space that conforms to the imaging physical process (defined by spatial downsampling B and spectral downsampling S). This dual guidance mechanism (physical property guidance + physical model constraints) is deeply coupled to the inverse denoising chain of the diffusion model, ensuring that the final reconstructed high-resolution hyperspectral image not only has high visual quality, but also is consistent with the edge structure of the input high-resolution multispectral image in spatial details, and satisfies the constraints of the degradation model in the spectral dimension, significantly improving the physical consistency and spectral fidelity of the reconstruction results.

[0032] (2) The present invention proposes a physical information extraction method that combines edge detection and semantic segmentation. This method can extract physical information from high-resolution multispectral images to improve the quality of high-resolution hyperspectral image generation of the denoising diffusion model. Specifically, edge information provides a strong prior for high-frequency spatial details such as object contours and texture boundaries; the extracted semantic information provides pixel-level labels for different category areas in the scene. By inputting these high-confidence physical information as conditions into the physical information guidance subnetwork, the diffusion model is provided with richer and more accurate scene spatial structure and semantic context guidance than a single low-resolution image Y in each step of the denoising process. This method effectively overcomes the problems of spatial structure ambiguity or unreasonable semantics that may arise from relying solely on data-driven learning, enabling the model to generate high-resolution spectral images with clearer spatial details and more reasonable semantics.

[0033] (3) This paper proposes an iterative optimization mechanism that integrates a physical imaging model to improve reconstruction accuracy and algorithm robustness. The physical model guides the spectral denoising subnetwork to explicitly construct the physical degradation model of hyperspectral imaging as an objective function and embeds it into each denoising step of the diffusion model through a K-stage iterative optimization process. This approach of embedding the physical model as an iterative optimization module into the diffusion model continuously imposes hard constraints during the generation process, significantly improving the accuracy of the reconstruction results to the imaging physical process and enhancing the adaptability to different degradation conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a general flow chart of the hyperspectral image super-resolution method based on the physical diffusion model disclosed in the present invention.

[0035] Figure 2 This is a schematic diagram of the principle of spectral denoising based on physical information in the present invention.

[0036] Figure 3 This is a schematic diagram of the principle of spectral denoising based on the physical model in the present invention.

[0037] Figure 4 This is a schematic diagram of the principle of iteratively reconstructing a high-resolution hyperspectral image using a diffusion model in the present invention.

[0038] Figure 5 These are diagrams showing the application effects of the present invention on different datasets, where (a) is the ICVL dataset, (b) is the Harvard dataset, and (c) is the Chikusei dataset. DETAILED DESCRIPTION

[0039] This paper discloses a hyperspectral image super-resolution method based on a physically guided diffusion model. It can uniformly process low-resolution hyperspectral images from different datasets and generate diverse high-resolution hyperspectral images, expanding the application scope of hyperspectral image super-resolution. The method has applications in various fields, including precision agriculture and environmental monitoring.

[0040] In order to better illustrate the advantages of the present invention, the invention is further described below with reference to the accompanying drawings and examples.

[0041] like Figure 1 As shown, this embodiment discloses a hyperspectral image super-resolution method based on a physical diffusion model, including:

[0042] Step S1: Generate a low-resolution hyperspectral image for testing using a real high-resolution hyperspectral image and the corresponding high-resolution multispectral image .

[0043] In one implementation, this example uses hyperspectral images from the publicly available ICVL, Harvard, and Chikusei datasets. Each hyperspectral image retains the number of channels, and is cropped and normalized in the spatial dimension to produce a 256×256 spatial resolution hyperspectral image. The low-resolution hyperspectral image is obtained by applying a 9×9 Lanczos blur to the cropped image and downsampling it by a factor of 8. The high-resolution multispectral image is obtained by multiplying the cropped image by the spectral response function.

[0044] Step S2: edge detection operator from high-resolution multispectral image Extract edge information ;

[0045] In one embodiment, the edge detection operator uses the Sobel operator. Although the high-resolution hyperspectral image and the high-resolution multispectral image of the same scene have different spectral dimensions, they share the same spatial edge structure (such as object contours and texture boundaries). Figure 2 Specifically, it can be expressed as:

[0046] Formula (1)

[0047] Formula (2)

[0048] Formula (3)

[0049] in represents the longitudinal gradient operator, represents the transverse gradient operator, Represents high-resolution multispectral images The gradient strength, that is, the edge strength.

[0050] Step S3: Semantic segmentation from high-resolution multispectral images Extracting semantic information ;

[0051] In one embodiment, the semantic segmentation operator uses SAM (Segment Anything Model) to obtain an extremely detailed semantic segmentation mask for a given image. The process is expressed as follows:

[0052]

[0053] in, represents the semantic segmentation operator, Represents the extracted segmentation semantic information;

[0054] Step S4: Convert low-resolution hyperspectral image , edge information , semantic information Input into the physical diffusion model to reconstruct a high-resolution hyperspectral image .

[0055] The physical diffusion model includes a forward Markov chain process and a reverse Markov chain process;

[0056] The forward Markov chain process The stages are represented as:

[0057] Formula (4)

[0058] in For a given hyperparameter, , Indicates the The noisy high-resolution hyperspectral image of the stage, represents the noise term that conforms to the standard normal distribution; represents the high-resolution hyperspectral image labels in the training dataset, represents the noise term at step t-1.

[0059] The reverse Markov chain process includes multiple stages;

[0060] No. The stage gradient term incorporates the observation constraint into the process of generating the image at the current time step as:

[0061] Formula (5)

[0062] in represents the pre-trained noise prediction network;

[0063] The reverse Markov chain process The final stage output of the prediction under stage Expressed as:

[0064] Formula (6)

[0065] The training loss function is expressed as:

[0066] Formula (7)

[0067] like Figure 4 The noise prediction network includes a physical information guided spectrum denoising sub-network and a physical model guided spectrum denoising sub-network connected in series;

[0068] like Figure 2 The physical information guides the spectral denoising subnetwork The implementation process is:

[0069] Formula (8)

[0070] in Represents the splicing of multiple information in the channel dimension, Represents the mapping function of the physical information guided spectral denoising sub-network, High-resolution multispectral images.

[0071] As an example, the physical information guided spectral denoising sub-network may use a Unet network.

[0072] The physical model guides the spectral denoising sub-network, which expands the HQS optimization algorithm to fully utilize the information of high-resolution multispectral images and low-resolution hyperspectral images. The objective function is expressed as:

[0073] Formula (9)

[0074] The objective function can be solved using HQS. and Two steps, The steps can be solved using the gradient descent method. The steps can be solved using neural networks.

[0075] like Figure 3 , the physical model guided spectral denoising sub-network includes K stages; the implementation process of each stage is:

[0076] Formula (10)

[0077] Formula (11)

[0078] in and is a learnable parameter, For neural networks, ∈[0,K-1], B represents the spatial downsampling operation, S represents the spectral downsampling operation, represents the output of the k+1th stage. = .in Output of the spectral denoising sub-network guided by the physical model.

[0079] As an example, The neural network consists of the first convolutional layer, an activation layer, and the second convolutional layer.

[0080] In the F update step, we directly use the observed data (Z, Y) and the current estimate ( , ) calculates gradients, forcibly adjusting the intermediate solution in a direction consistent with physical observations. During the update step V, a lightweight neural network D is used to impose image priors. The introduction of learnable parameters (ρ, α) enables the model to adaptively adjust the weights of physical constraints and priors during the optimization process, adapting to varying degradation strengths and image content.

[0081] The hyperspectral image super-resolution method based on the physical diffusion model of the present invention can process different low-resolution hyperspectral images and reconstruct high-resolution hyperspectral images. Figure 5 Middle (a) - Attachment Figure 5(c) shows the super-resolution effect of the present invention on the ICVL dataset, Harvard dataset, and Chikusei dataset. The hyperspectral image super-resolution method of the present invention can reconstruct high-quality high-resolution hyperspectral images.

[0082] To further illustrate the effects of the present invention, this example compares various methods under the same experimental conditions. The experimental results are shown in the following table:

[0083]

[0084] The results in the table show that the method disclosed in this invention achieves excellent super-resolution results, with PSNR, SSIM, and ERGAS metrics significantly exceeding those of the comparison methods across multiple datasets. PSNR and SSIM primarily measure the spatial quality of super-resolved hyperspectral images, while ERGAS primarily measures the spectral quality of super-resolved hyperspectral images. Therefore, the method disclosed in this invention achieves hyperspectral images with significantly smaller spatial and spectral errors, outperforming other methods in both spatial quality and spectral fidelity.

[0085] This embodiment also discloses a hyperspectral image super-resolution device, including:

[0086] Data acquisition module, responsible for acquiring low-resolution hyperspectral images and the corresponding high-resolution multispectral image ; From high-resolution multispectral images Extract edge information and semantic information ;

[0087] Image reconstruction module, responsible for converting low-resolution high-spectral images , edge information , semantic information Input into the physical diffusion model to reconstruct a high-resolution hyperspectral image .

[0088] This embodiment further discloses a computing device. Specifically, the computing device includes a memory and a processor. The memory stores executable code. When the processor executes the executable code, the method described in any one of the embodiments is implemented.

[0089] The storage component may be configured with high-speed random access memory (RAM) and may also be expanded to include non-volatile storage units (such as at least one disk storage device). Data channels between the device node and other nodes are established via at least one transmission interface (supporting wired and wireless communication modes), adapting to multi-level network architectures (including the Internet, wide area networks, local area networks, and metropolitan area networks).

[0090] The bus system can be of the Industrial Standard Architecture (ISA) bus, Peripheral Component Interconnect (PCI) bus or Extended Industrial Standard Architecture (EISA) bus type. Its internal architecture is divided into three functional modules: address transmission channel, data transmission channel and control signal channel.

[0091] The storage component is used to store executable code. When the operation unit receives an execution instruction, the method flow described in each embodiment of the present invention is implemented by parsing and executing the code set. The running carrier of the method flow is the operation unit.

[0092] The arithmetic unit can adopt a semiconductor integrated circuit architecture and have digital signal analysis and computing capabilities. Its implementation method includes two paths: directly executing hardware decoding through physical layer transistor logic circuits, or performing software and hardware collaborative operations based on programmable instruction sets. In specific implementations, the arithmetic unit can adapt to multiple types of computing architectures: general-purpose computing units (including CPUs / NPs) are suitable for scalar instruction processing; heterogeneous computing units (including DSPs / ASICs / FPGAs) support customized algorithm acceleration; and reconfigurable computing units (including discrete gate circuits / transistor arrays) provide hardware-level dynamic configuration capabilities. The instruction execution carrier is implemented through a storage hierarchy system: the software instruction set resides in the non-volatile storage matrix (including NOR Flash / EEPROM) or high-speed cache area (SRAM / register stack) of the storage component, and is accessed and scheduled by the arithmetic unit through the bus architecture. The hardware acceleration module can be directly mapped to the on-chip storage medium (Cache / BRAM) to achieve zero-latency response to instructions.

[0093] An embodiment of the present invention provides a readable storage medium in the form of a computer program product, which includes a computer-readable storage carrier with a program code storage function. The program code carried by the carrier includes executable instructions for implementing the technical solution in the aforementioned method embodiment. The specific implementation details of the relevant method steps have been fully recorded in the aforementioned method embodiment, and will not be repeated in this embodiment.

[0094] When the functional module is implemented as an independent software unit and commercialized as a commodity, its operating data can be stored in a digital information storage medium. Based on this technical implementation principle, the innovative core value of this patent or the characteristic module that is different from the existing technology can be presented in the form of an application package. Such a program suite usually resides in a data carrier and has multiple built-in operating commands to guide electronic devices (covering hardware facilities such as personal terminals, cloud servers, and networked devices) to execute all or core operating links of the process detailed in the patent embodiment. The information carrier includes: portable flash drives, external storage devices, firmware memory (ROM), dynamic access memory (RAM), magnetic storage disks or optical recording media and other physical media with program code storage capabilities.

[0095] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. Hyperspectral image super-resolution method based on physical diffusion model, characterized by: The method comprises: Acquiring low-resolution hyperspectral images and the corresponding high-resolution multispectral image ; From high-resolution multispectral images Extract edge information and semantic information ; Low-resolution hyperspectral images , edge information , semantic information Input into the physical diffusion model to reconstruct a high-resolution hyperspectral image .

2. The method according to claim 1, characterized in that The physical diffusion model includes a forward Markov chain process and a reverse Markov chain process; The reverse Markov chain process includes multiple stages; The stage gradient term incorporates the observation constraints into the generated image via a noise prediction network at the current time step; the noise prediction network includes a physical information guided spectral denoising subnetwork and a physical model guided spectral denoising subnetwork connected in series.

3. The method according to claim 2, characterized in that The training loss function of the physical diffusion model is expressed as: Formula (1) in represents the high-resolution hyperspectral image label, Represents the reverse Markov chain process in The final stage output of the prediction under stage.

4. The method according to claim 3, characterized in that The physical information guides the spectral denoising subnetwork The implementation process is: Formula (2) in Represents the splicing of multiple information in the channel dimension, Represents the mapping function of the physical information guided spectral denoising sub-network, represents a low-resolution hyperspectral image, Represents edge information, Represents semantic information, Indicates the Noisy high-resolution hyperspectral images of the phase.

5. The method according to claim 4, characterized in that: The physical information guided spectral denoising sub-network uses the Unet network.

6. The method according to claim 3, characterized in that: The physical model guided spectral denoising sub-network includes K stages; the implementation process of each stage is: Formula (3) Formula (4) in and is a learnable parameter, For neural networks, ∈[0,K-1], B represents the spatial downsampling operation, S represents the spectral downsampling operation, represents the output of the k+1th stage, Adopt the Noisy high-resolution hyperspectral images of the stage .

7. The method according to claim 6, characterized in that The neural network in the spectral denoising sub-network guided by the physical model consists of a first convolutional layer, an activation layer, and a second convolutional layer.

8. A hyperspectral image super-resolution device for implementing the method according to any one of claims 1 to 7, characterized in that: include: Data acquisition module, responsible for acquiring low-resolution hyperspectral images and the corresponding high-resolution multispectral image ; From high-resolution multispectral images Extract edge information and semantic information ; Image reconstruction module, responsible for converting low-resolution high-spectral images , edge information , semantic information Input into the physical diffusion model to reconstruct a high-resolution hyperspectral image .

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.

Images