Hyperspectral image super-resolution method and device based on edge perception multi-modal residual diffusion model and storage medium

Abstract

This invention discloses a hyperspectral image super-resolution method, device, and storage medium based on an edge-aware multimodal residual diffusion model. The invention introduces a multimodal residual mechanism to promote efficient information transfer between high-resolution multispectral images, low-resolution hyperspectral images, and high-resolution hyperspectral images in the diffusion model, significantly improving fusion efficiency. It utilizes edge information from high-resolution multispectral images to design an edge-aware noise strategy, applying stronger noise perturbations to edge regions to guide the model to prioritize the reconstruction of high-frequency details. A bilateral attention fusion denoising network is proposed, and a multi-scale supervision mechanism is designed to achieve progressive reconstruction and collaborative optimization of spectral and spatial features.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a hyperspectral image super-resolution method, device, and storage medium based on an edge-aware multimodal residual diffusion model. Background Technology

[0002] Hyperspectral images can capture the spatial and spectral information of objects across hundreds of consecutive bands, and have wide applications in agriculture, environmental monitoring, and geological exploration. However, due to the physical limitations of imaging sensors, hyperspectral images cannot simultaneously possess high spatial and spectral resolution. To overcome this limitation, improving the spatial resolution of hyperspectral images using signal processing techniques has become a new technological approach. Existing spectral super-resolution techniques are generally divided into two main categories: single-image super-resolution and spectral fusion super-resolution. Spectral fusion super-resolution incorporates spatial information from high-resolution multispectral images, thus achieving better imaging quality.

[0003] Traditional hyperspectral image super-resolution methods are mainly based on mathematical models and artificially designed prior knowledge, including sparse representation-based methods, nonnegative matrix factorization methods, and tensor factorization methods. Although these methods can achieve image reconstruction under certain conditions, their linear model assumptions are fundamentally contradictory to the inherent nonlinear characteristics of hyperspectral images. This leads to problems such as spectral distortion and insufficient spatial detail in complex real-world scenes, and they are also sensitive to noise and have limited generalization ability.

[0004] With the development of deep learning technology, significant progress has been made in hyperspectral image super-resolution methods based on neural networks. These methods can be broadly classified into supervised learning and unsupervised learning. Supervised methods minimize the difference between the reconstructed image and the real high-resolution image by constructing an end-to-end mapping function, but their performance is highly dependent on the scale and quality of the paired training data. While unsupervised methods reduce the dependence on paired data, the lack of explicit supervision signals often leads the network to get stuck in local optima, limiting the reconstruction quality.

[0005] In recent years, diffusion models have demonstrated strong potential in image generation, providing new approaches for hyperspectral image super-resolution. Diffusion models achieve image generation through a stepwise denoising process, offering advantages such as stable training, high generation quality, and good sample diversity. For example, Rui et al. used a low-rank diffusion model for hyperspectral image sharpening. Wu et al. proposed a hyperspectral image fusion method based on a conditional diffusion model. Qu et al. constructed a spatial-spectral bidirectional cyclic diffusion framework for hyperspectral image fusion.

[0006] Existing standard diffusion models face three main challenges when directly applied to hyperspectral image super-resolution tasks: First, diffusion models typically require hundreds or even thousands of sampling steps, resulting in extremely low inference efficiency; second, the Gaussian noise used in traditional diffusion models is uniformly distributed, making it difficult to effectively guide the model to focus on high-frequency detail regions crucial to visual quality; and finally, the conventional UNet network structure suffers from problems such as feature fusion interference, blind upsampling, and insufficient utilization of observation information in hyperspectral image reconstruction. To address these issues, this invention proposes a hyperspectral image super-resolution method, device, and storage medium based on an edge-aware multimodal residual diffusion model. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a hyperspectral image super-resolution method, device, and storage medium based on an edge-aware multimodal residual diffusion model.

[0008] In a first aspect, the present invention provides a hyperspectral image super-resolution method based on an edge-aware multimodal residual diffusion model, the method comprising:

[0009] Low-resolution hyperspectral images and high-resolution multispectral images are generated from real high-resolution hyperspectral images and divided into training and testing sets. Edge-aware noise is generated using edge information from the high-resolution multispectral images and used as Gaussian noise in the edge-aware multimodal residual diffusion model. Multimodal residuals are constructed between the low-resolution hyperspectral images, high-resolution multispectral images, and high-resolution hyperspectral images, and the multimodal residuals are passed through the Markov chain of the edge-aware multimodal residual diffusion model. A bilateral attention fusion denoising network is constructed as the denoising network for the reverse process of the edge-aware multimodal residual diffusion model, and is trained and tested on the training and testing sets respectively.

[0010] In application, a low-resolution hyperspectral image and its corresponding high-resolution multispectral image are acquired, and edge-aware noise is extracted. The low-resolution hyperspectral image, its corresponding high-resolution multispectral image, and the edge-aware noise are input into a trained and tested edge-aware multimodal residual diffusion model, and the high-resolution hyperspectral image is recovered using a reverse process.

[0011] Preferably, the process of generating edge-aware noise using edge information from high-resolution multispectral images is as follows:

[0012] Convert high-resolution multispectral images into grayscale images by channel averaging.

[0013] The grayscale image is convolved using the defined Sobel operator to obtain the horizontal and vertical gradients.

[0014] Edge strength is obtained by using horizontal and vertical gradients;

[0015] The edge-aware weight is obtained by normalizing the edge intensity, and then multiplied element-wise with ordinary Gaussian noise to obtain the edge-aware noise. This noise has a higher intensity in the edge region of the image and a lower intensity in the flat region.

[0016] Preferably, the multimodal residual is an image stitched together in the band dimension from a low-resolution hyperspectral image and a high-resolution multispectral image that have undergone bicubic interpolation upsampling, along with the high-resolution hyperspectral image and its preceding data. The multimodal residual is obtained by subtracting the images stitched together along the band dimension. This multimodal residual contains both spatial detail differences and spectral feature differences.

[0017] Preferably, the implementation process of introducing multimodal residuals into the edge-aware multimodal residual diffusion model is as follows:

[0018] For the forward process of the edge-aware multimodal residual diffusion model, the process is represented by propagating the multimodal residual to the Markov chain as follows:

[0019]

[0020] in This represents the image at stage t. The parameter representing the adjustment of edge-sensing noise intensity. This represents the amount of multimodal residual shift added in stage t. Indicates multimodal residuals, Indicates edge-aware noise; This represents the cumulative shift percentage of the multimodal residuals, and ;

[0021] The reverse process for the edge-aware multimodal residual diffusion model involves stitching together a low-resolution hyperspectral image (upsampled by bicubic interpolation) and a high-resolution multispectral image along the band dimension. and edge-aware noise Initially, noise is gradually reduced using a bilateral attention fusion denoising network, and the multimode residuals are then backpropagated. To restore the stage 0 image .

[0022] Preferably, the bilateral attention fusion denoising network includes two encoding paths, a bottleneck layer, and one decoding path; The first encoding path consists of K concatenated time-step embedding encoding layers, where K > 1; each time-step embedding encoding layer includes a multi-scale attention module, a time-step embedding encoded by sine and cosine positions, and a downsampling block; the input image of the first time-step embedding encoding layer is the t-th stage image of the Markov chain. The input image of other time-step embedding coding layers is the output of the previous time-step embedding coding layer; The second encoding path includes K cascaded attention coding layers. Each attention coding layer includes a multi-scale group attention module and a downsampling block. The input of the first attention coding layer is the image stitched together in the band dimension of the low-resolution hyperspectral image and the high-resolution multispectral image after bicubic interpolation upsampling. The input of other attention coding layers is the output of the previous attention coding layer. The bottleneck layer employs a multi-scale group attention module, whose input is the concatenation result of the output of the last attention coding layer and the output of the last time step embedding coding layer. The decoding path includes K cascaded decoding layers. Each decoding layer includes a multi-scale group attention module and an upsampling block. The multi-scale group attention module is connected to the output of the same attention coding layer and the output of the same time-step embedding coding layer in a skip connection. After processing by the upsampling block, the output of the current decoding layer is obtained. and will With images ,image Perform direct addition to enhance information reuse and use images As a supervisory measure, the upsampling process is clearly defined; including the image... This represents the image obtained by upsampling a low-resolution hyperspectral image using n-fold bicubic interpolation. Represents high-resolution multispectral images after The image obtained after downsampling by a factor of 10. Representing high-resolution hyperspectral images after The image obtained after downsampling by a factor of n, where n represents the super-resolution factor. , , , where k represents the current k-th layer.

[0023] This invention's bilateral attention fusion denoising network sets up a denoising path and a guiding path to process the noisy input image and the guiding image stitched from the observed images, respectively, avoiding information interference caused by early feature fusion. Furthermore, each layer integrates grouped convolution, a spatial-channel hybrid attention mechanism, and an adaptive feature fusion strategy to comprehensively capture local details, channel dependencies, and spatial context information. In addition, during the upsampling process, the upsampling results of low-resolution hyperspectral images at different scales and the downsampling results of high-resolution multispectral images are stitched together and injected into the corresponding scale's decoding layer. The downsampling version of the real high-resolution hyperspectral image at the corresponding scale is used as the supervision target to achieve progressive reconstruction.

[0024] Preferably, the multi-scale loss function of the bilateral attention fusion denoising network Represented as:

[0025]

[0026] in This indicates splicing along a band dimension.

[0027] Preferably, the implementation process of the multi-scale group attention module is as follows:

[0028] Input image First, the first feature map I is obtained through convolution, layer normalization, and LeakyReLU activation function. Then, the second feature map J is obtained through spatial attention enhancement.

[0029] The second feature map J extracts complementary features through two parallel branches, one of which uses a convolution with grouping of 1 to obtain local features. Another branch uses depthwise separable convolutions grouped into s to obtain global features. , s≠1;

[0030] For local features and global features The weights are obtained through global average pooling, convolution, and softmax activation functions, respectively. and The features of each branch are weighted to adaptively fuse global and local features to obtain the fused feature h;

[0031] The fused features h are then subjected to nonlinear transformation and enhancement via a multilayer perceptron, and finally connected to the input image via residual connections. The final output, Out, is obtained.

[0032] In a second aspect, the present invention provides a hyperspectral image super-resolution system, comprising:

[0033] The data acquisition module is responsible for acquiring low-resolution hyperspectral images and corresponding high-resolution multispectral images, and extracting edge-aware noise;

[0034] The image restoration module is responsible for inputting low-resolution hyperspectral images, corresponding high-resolution multispectral images, and edge-aware noise into the trained and tested edge-aware multimodal residual diffusion model, and using the reverse process to restore the high-resolution hyperspectral image.

[0035] Thirdly, the present invention provides an electronic device including a storage component and a computing unit, wherein the storage component stores machine-executable instructions that can be executed by the computing unit, and the computing unit executes the machine-executable instructions to implement the hyperspectral image super-resolution method.

[0036] Fourthly, the present invention provides a machine-readable storage medium comprising a computer-readable storage carrier having program code storage function, wherein the program code carried on the carrier contains executable instructions for the hyperspectral image super-resolution method.

[0037] The beneficial effects of this invention are at least as follows:

[0038] (1) By introducing a multimodal residual mechanism, this invention transforms the traditional blind denoising process that starts from pure noise into a targeted reconstruction process that starts from the observed image, which significantly shortens the diffusion path and reduces the number of sampling steps required from hundreds of steps to tens or even a few steps, greatly improving inference efficiency while ensuring reconstruction quality.

[0039] (2) This invention proposes an edge-aware noise, which innovatively uses the edge information of high-resolution multispectral images to modulate diffuse noise and generate edge-aware noise. This enables the model to prioritize high-frequency detail regions that are crucial to visual quality during the denoising process, effectively improving the spatial detail fidelity and edge sharpness of the reconstructed image.

[0040] (3) This invention proposes a bilateral attention fusion denoising network specifically for hyperspectral image reconstruction tasks. It avoids early fusion interference of different modal information through a dual-path structure, comprehensively captures spatial-spectral features through multi-scale group attention modules, and achieves progressive reconstruction from coarse to fine through a multi-scale supervision mechanism, which significantly improves the feature expression ability and reconstruction accuracy of the model. Attached Figure Description

[0041] Figure 1 This is the overall flowchart of the hyperspectral image super-resolution method based on the edge-aware multimodal residual diffusion model disclosed in this invention;

[0042] Figure 2 This is a schematic diagram of the edge-aware multimodal residual diffusion model in this invention;

[0043] Figure 3 This is a schematic diagram of the edge-sensing noise principle in this invention;

[0044] Figure 4 This is a schematic diagram of the bilateral attention fusion denoising network in this invention;

[0045] Figure 5 This is a schematic diagram of the structure of the multi-scale group attention module in this invention;

[0046] Figure 6These are application effect diagrams of the present invention on the ICVL dataset, Harvard dataset, and Chikusei dataset. Among them, (a), (d), and (g) are three low-resolution hyperspectral images, (b), (e), and (h) are the high-resolution hyperspectral images restored from (a), (d), and (g) respectively, and (c), (f), and (i) are the real high-resolution hyperspectral images from (a), (d), and (g) respectively. Detailed Implementation

[0047] To better illustrate the invention and advantages of this project, the invention will be further explained below with reference to the accompanying drawings and examples.

[0048] Example 1.

[0049] This embodiment discloses a hyperspectral image super-resolution method based on an edge-aware multimodal residual diffusion model. This invention can process low-resolution hyperspectral images and high-resolution multispectral images from different datasets to generate diverse high-resolution hyperspectral images. This invention can be applied to multiple fields such as precision agriculture, environmental monitoring, and geological exploration.

[0050] like Figure 1 As shown, this embodiment discloses a hyperspectral image super-resolution method based on an edge-aware multimodal residual diffusion model, including the following steps:

[0051] Step S1: Generate low-resolution hyperspectral images and high-resolution multispectral images from real high-resolution hyperspectral images, and divide them into training set and test set;

[0052] Step S2: Use the edge information from the high-resolution multispectral image to generate edge-aware noise to replace the ordinary Gaussian noise in the edge-aware multimodal residual diffusion model;

[0053] Step S3: Construct multimodal residuals between low-resolution hyperspectral images, high-resolution multispectral images, and high-resolution hyperspectral images, and pass the multimodal residuals through a Markov chain;

[0054] Step S4: Construct a bilateral attention fusion denoising network UNet as the denoising network for the reverse process of the edge-aware residual diffusion model, and train and test it on the training set and test set respectively;

[0055] Step S5: When applying, acquire the low-resolution hyperspectral image and the corresponding high-resolution multispectral image, and extract edge-aware noise; input the low-resolution hyperspectral image, the corresponding high-resolution multispectral image, and the edge-aware noise into the trained and tested edge-aware multimodal residual diffusion model, and use the reverse process to recover the high-resolution hyperspectral image.

[0056] Specifically, step S1 is as follows:

[0057] This embodiment uses hyperspectral images from the publicly available ICVL, Harvard, and Chikusei datasets. A 256×256 spatially sized image patch is cropped from the original hyperspectral image as a reference ground value for the high-resolution hyperspectral image. A blurring effect is simulated by applying a 3×3 Gaussian blur kernel with a standard deviation of 2, followed by 8x bicubic interpolation downsampling to generate a low-resolution hyperspectral image. Then, the corresponding high-resolution multispectral image is generated by multiplying the high-resolution hyperspectral image with the spectral response function of a Nikon D700 camera.

[0058] like Figure 3 The edge-aware noise is obtained from the edge information of the high-resolution multispectral image, specifically:

[0059] The edge-aware noise acquisition process described in step S2 first converts the high-resolution multispectral image into a grayscale image by channel averaging, as shown in the following process:

[0060]

[0061] in Represents high-resolution multispectral images, These represent the height, width, and number of bands of a high-resolution multispectral image, respectively.

[0062] The grayscale image is then convolved using the defined Sobel operator, as shown in the following process:

[0063]

[0064] in and Set as:

[0065]

[0066] For the obtained horizontal gradient and vertical gradient Edge strength can be obtained by calculating the gradient magnitude, the process of which is expressed as:

[0067]

[0068] For the edge intensity M, it is normalized to obtain the edge-aware weight W, which is then multiplied by ordinary Gaussian noise N to obtain the edge-aware noise. The process is expressed as follows:

[0069]

[0070] in This refers to edge-sensing noise;

[0071] like Figure 2 As shown, the edge-aware multimodal residual diffusion model introduces a multimodal residual mechanism, which significantly shortens the diffusion path and enhances information reuse. Specifically:

[0072] The multimodal residuals mentioned in step S3 are obtained from low-resolution hyperspectral images, high-resolution multispectral images, and high-resolution hyperspectral images. The process is represented as follows:

[0073]

[0074] in This represents the image obtained by stitching together a low-resolution hyperspectral image and a high-resolution multispectral image in the band dimension, after bicubic interpolation upsampling. This represents a high-resolution hyperspectral image and its first S bands stitched together along the band dimension. This refers to multimodal residuals;

[0075] For the forward process of the edge-aware multimodal residual diffusion model, by propagating the multimodal residuals to the Markov chain, the diffusion model is effectively guided to reconstruct the missing spatial structure and spectral properties, instead of relying solely on the general denoising process. The process is expressed as follows:

[0076]

[0077] in This represents the image at stage t. The parameter representing the adjustment of edge-sensing noise intensity. This represents the amount of multimodal residual shift added in stage t. Indicates multimodal residuals, Indicates edge-aware noise; The cumulative shift percentage of the multimodal residuals is a monotonically increasing sequence. .and When t=1, Approaching 0, when t=T, When it approaches 1, Similar to Add edge-aware noise In this way, multi-mode residuals are added to the Markov chain, which reduces the number of sampling steps and effectively improves inference efficiency.

[0078] The reverse process for the edge-aware multimodal residual diffusion model involves stitching together a low-resolution hyperspectral image (upsampled by bicubic interpolation) and a high-resolution multispectral image along the band dimension. and edge-aware noise Initially, noise is gradually reduced using a bilateral attention fusion denoising network, and the multimode residuals are then backpropagated. To restore the stage 0 image The process is represented as follows:

[0079]

[0080] in This indicates a two-sided attention fusion denoising network used for prediction. ;

[0081] like Figure 4 As shown, the bilateral attention fusion denoising network adopts the UNet framework, which includes two encoding paths, a bottleneck layer, and one decoding path;

[0082] The first encoding path consists of K concatenated temporal step embedding encoding layers. Each temporal step embedding encoding layer includes a multi-scale attention module, a temporal step embedding encoded by sine and cosine positions, and a downsampling block. The multi-scale attention module extracts features from the input image through multi-scale attention. The temporal step embedding converts discrete time steps into encoded tensors, transforms them using a multi-layer perceptron, and injects them into the current multi-scale attention module. The downsampling block is implemented using convolutions with a stride of 2. The input image of the first temporal step embedding encoding layer is the image at stage t. The input image of other time-step embedding coding layers is the output of the previous time-step embedding coding layer;

[0083] The second encoding path includes K cascaded attention coding layers. Each attention coding layer includes a multi-scale group attention module and a downsampling block. The input of the first attention coding layer is the image stitched together in the band dimension of the low-resolution hyperspectral image and the high-resolution multispectral image after bicubic interpolation upsampling. The input of other attention coding layers is the output of the previous attention coding layer.

[0084] The bottleneck layer employs a multi-scale group attention module, whose input is the concatenation result of the output of the last attention coding layer and the output of the last time step embedding coding layer.

[0085] The decoding path includes K cascaded decoding layers. Each decoding layer includes a multi-scale group attention module and an upsampling block. The multi-scale group attention module is skipped between the output of the same attention coding layer and the output of the same time-step embedding coding layer. After processing by the upsampling block, the output of the current decoding layer is obtained. and will With images ,image Perform direct addition to enhance information reuse and use images As a supervisory measure, the upsampling process is clearly defined; including the image... This represents the image obtained by upsampling a low-resolution hyperspectral image using n-fold bicubic interpolation. Represents high-resolution multispectral images after The image obtained after downsampling by a factor of 10. Representing high-resolution hyperspectral images after The image obtained after downsampling by a factor of n, where n represents the super-resolution factor. , , , where k represents the current k-th layer.

[0086] The first encoding path processes the noisy t-th stage image. It also incorporates time-step embedding via sine and cosine position coding. The second coding path uses an image stitched together in the band dimension from a low-resolution hyperspectral image upsampled by bicubic interpolation and a high-resolution multispectral image. As input, the focus is on spatial structure and spectral priors. These two paths integrate deep features rather than early concatenation, effectively preventing information interference and enabling independent learning of noise distribution and structural features.

[0087] The above-mentioned Multiscale Group Attention Module (MSGAB), attached Figure 5 This demonstrates the structural composition of a multi-scale attention module, starting with the input image. First, the first feature map I is obtained through convolution, layer normalization, and the LeakyReLU activation function. Then, the second feature map J is obtained through spatial attention enhancement. The process is represented as follows:

[0088]

[0089]

[0090] Among them, LReLU, LN and These represent the LeakyReLU activation function, layer normalization, and element-wise multiplication, respectively.

[0091] The second feature map J extracts complementary features through two parallel branches, one of which uses a convolution with grouping of 1 to obtain local features. To achieve dense cross-channel interaction, another branch uses depthwise separable convolutions grouped into s to obtain global features. To maintain sparse channel connectivity and preserve spectral characteristics, s≠1; the process is represented as:

[0092]

[0093]

[0094] in Contains two groups with different numbers of groups Convolution, two LeakyReLU activation functions, and one layer normalization;

[0095] For local features and global features The weights are obtained through global average pooling, convolution, and softmax activation functions, respectively. and The features of each branch are weighted to adaptively fuse global and local features to obtain the fused feature h; the process is represented as:

[0096]

[0097]

[0098] Where GAP is the average pooling function for the entire series, and Softmax is the Softmax activation function;

[0099] The fused feature h is then subjected to nonlinear transformation and enhancement through a multilayer perceptron to obtain feature V, which is then connected to the input image via residual connections. The process of obtaining the final output Out is represented as follows:

[0100]

[0101]

[0102] For the multi-scale supervision used in the sampling process of the bilateral attention fusion denoising network, the low-resolution hyperspectral image is obtained by upsampling through bicubic interpolation at different magnifications. High-resolution multispectral images are downsampled to the corresponding resolution to obtain... Then, these sums are injected into each upsampling stage of the bilateral attention fusion denoising network. Secondly, we employ band stitching of progressively downsampled high-resolution hyperspectral images and pseudo-high-resolution multispectral images. As supervision at each stage, the upsampling process is clearly limited, and a multi-scale loss function is used. It can be represented as:

[0103]

[0104] in This indicates splicing along the band dimension, and K can take the value 3.

[0105] This invention can process low-resolution hyperspectral images and high-resolution multispectral images and reconstruct high-resolution hyperspectral images. (See attached image.) Figure 6 (a) - Appendix Figure 6 Figure (i) shows the super-resolution performance of the proposed method on the ICVL, Harvard, and Chikusei datasets. For comparison, from left to right, there is a low-resolution hyperspectral image, a high-resolution hyperspectral image reconstructed by the present invention, and a real high-resolution hyperspectral image. It can be seen that the hyperspectral image super-resolution method of the present invention can reconstruct high-resolution hyperspectral images of good quality.

[0106] This invention also provides a hyperspectral image super-resolution system for implementing the above method, comprising:

[0107] The data acquisition module is responsible for acquiring low-resolution hyperspectral images and corresponding high-resolution multispectral images, and extracting edge-aware noise;

[0108] The image restoration module is responsible for inputting low-resolution hyperspectral images, corresponding high-resolution multispectral images, and edge-aware noise into the trained and tested edge-aware multimodal residual diffusion model, and using the reverse process to restore the high-resolution hyperspectral image.

[0109] This invention discloses a data processing device, namely an electronic device. Specifically, the device includes a storage component and a computing unit. The storage component contains an executable instruction set. When the computing unit runs the instruction set, it can execute various operation processes described in the embodiments.

[0110] The storage component may be configured with high-speed random access memory (RAM) and may also be expanded with non-volatile storage units (such as at least one disk storage device). Data channels between device nodes and other nodes are established through at least one transmission interface (supporting wired / wireless communication modes), adapting to multi-level network architectures (including the Internet, wide area networks, local area networks, and metropolitan area networks).

[0111] The bus system can be of various types, such as the Industry Standard Architecture (ISA) bus, the Peripheral Interconnect (PCI) bus, or the Extended Industry Standard Architecture (EISA) bus. Its internal architecture is divided into three functional modules: address transmission channel, data transmission channel, and control signal channel.

[0112] The storage component is used to store executable code. When the computing unit receives an execution instruction, it implements the method flow described in the various embodiments of the present invention by parsing and executing the code set. The computing unit is the carrier for running the method flow.

[0113] The computing unit can adopt a semiconductor integrated circuit architecture, possessing digital signal parsing and computation capabilities. Its implementation involves a dual path: direct hardware decoding via physical layer transistor logic circuits, or hardware-software collaborative computation based on a programmable instruction set. Specifically, the computing unit can adapt to multiple computing architectures: general-purpose computing units (including CPUs / NPs) are suitable for scalar instruction processing; heterogeneous computing units (including DSPs / ASICs / FPGAs) support customized algorithm acceleration; and reconfigurable computing units (including discrete gate circuits / transistor arrays) provide hardware-level dynamic configuration capabilities. The instruction execution carrier is implemented through a storage hierarchy: the software instruction set resides in a non-volatile memory matrix (including NOR Flash / EEPROM) or a high-speed temporary storage area (SRAM / register file) of the storage component, and is accessed and scheduled by the computing unit through a bus architecture. The hardware acceleration module can be directly mapped to on-chip storage media (Cache / BRAM) to achieve zero-latency instruction response.

[0114] This invention provides a readable storage medium in the form of a computer program product, which includes a computer-readable storage carrier with program code storage function. The program code carried on the carrier includes executable instructions for implementing the technical solutions in the foregoing method embodiments. The specific implementation details of the relevant method steps have been fully described in the foregoing method embodiments, and will not be repeated in this embodiment.

[0115] When the functional modules are implemented as independent software units and commercially operated as commodities, their operational data can be stored in digital information storage media. Based on this technical principle, the core innovative value of this patent, or the feature modules that distinguish it from existing technologies, can be presented in the form of application package. Such program suites typically reside in data carriers and contain multiple built-in operation commands to guide electronic devices (covering hardware facilities such as personal terminals, cloud servers, and networked devices) to execute all or core operation steps of the process detailed in the patent embodiments. The information carriers include physical media with program encoding and storage capabilities such as portable flash drives, external storage devices, firmware memory (ROM), dynamic access memory (RAM), magnetic storage disks, or optical recording media.

[0116] The embodiments described in this invention are merely for illustrating the implementation schemes of the patented technology and do not constitute a limitation on the scope of the rights. Although this patent has been analyzed in detail through specific embodiments, those skilled in the art should understand that within the technical framework disclosed in the patent, professionals can still adjust parameters, optimize structures, or make equivalent substitutions of technical elements in the disclosed technical solutions. Such technical adjustments or adaptive modifications, as long as they do not deviate from the core ideas and innovative boundaries of the patent design principle, fall within the scope of the rights of this patent.

Claims

1. A hyperspectral image super-resolution method based on an edge-aware multimodal residual diffusion model, characterized in that, The method includes: Low-resolution hyperspectral images and high-resolution multispectral images are generated from real high-resolution hyperspectral images and divided into training and testing sets. Edge-aware noise is generated using edge information from the high-resolution multispectral images and used as Gaussian noise in the edge-aware multimodal residual diffusion model. Multimodal residuals are constructed between the low-resolution hyperspectral images, high-resolution multispectral images, and high-resolution hyperspectral images, and the multimodal residuals are passed through the Markov chain of the edge-aware multimodal residual diffusion model. A bilateral attention fusion denoising network is constructed as the denoising network for the reverse process of the edge-aware multimodal residual diffusion model, and is trained and tested on the training and testing sets respectively. In application, a low-resolution hyperspectral image and its corresponding high-resolution multispectral image are acquired, and edge-aware noise is extracted. The low-resolution hyperspectral image, its corresponding high-resolution multispectral image, and the edge-aware noise are input into a trained and tested edge-aware multimodal residual diffusion model, and the high-resolution hyperspectral image is recovered using a reverse process.

2. The method according to claim 1, characterized in that, The process of generating edge-aware noise using edge information from high-resolution multispectral images is as follows: The high-resolution multispectral image is converted into a grayscale image by channel averaging, and the horizontal and vertical gradients are further calculated. Edge strength is obtained by using horizontal and vertical gradients; The edge-aware weight is obtained by normalizing the edge intensity, and then multiplied by ordinary Gaussian noise to obtain the edge-aware noise.

3. The method according to claim 1, characterized in that, The multimodal residual is obtained by subtracting the image obtained by stitching together the low-resolution hyperspectral image and the high-resolution multispectral image in the band dimension after bicubic interpolation upsampling, and the image obtained by stitching together the high-resolution hyperspectral image and its first s bands in the band dimension, where s > 0.

4. The method according to claim 1, characterized in that, The implementation process of introducing multimodal residuals into the edge-aware multimodal residual diffusion model is as follows: For the forward process, multimodal residuals are propagated to the Markov chain; The reverse process starts with the image stitched together in the band dimension from the low-resolution hyperspectral image and the high-resolution multispectral image after bicubic interpolation upsampling, along with edge-aware noise. The image in stage 0 is then restored by gradually denoising through a bilateral attention fusion denoising network and reverse-transmitting the multimode residual.

5. The method according to claim 1, characterized in that, The bilateral attention fusion denoising network includes two encoding paths, a bottleneck layer, and one decoding path; The first encoding path consists of K concatenated time-step embedding encoding layers, where K > 1; each time-step embedding encoding layer includes a multi-scale attention module, a time-step embedding encoded by sine and cosine positions, and a downsampling block; the input image of the first time-step embedding encoding layer is the t-th stage image of the Markov chain. The input image of other time-step embedding coding layers is the output of the previous time-step embedding coding layer; The second encoding path includes K cascaded attention coding layers. Each attention coding layer includes a multi-scale group attention module and a downsampling block. The input of the first attention coding layer is the image stitched together in the band dimension of the low-resolution hyperspectral image and the high-resolution multispectral image after bicubic interpolation upsampling. The input of other attention coding layers is the output of the previous attention coding layer. The bottleneck layer employs a multi-scale group attention module, whose input is the concatenation result of the output of the last attention coding layer and the output of the last time step embedding coding layer. The decoding path includes K cascaded decoding layers. Each decoding layer includes a multi-scale group attention module and an upsampling block. The multi-scale group attention module is connected to the output of the same attention coding layer and the output of the same time-step embedding coding layer in a skip connection. After processing by the upsampling block, the output of the current decoding layer is obtained. and will With images ,image Perform direct addition to enhance information reuse and use images As a supervisory measure, the upsampling process is clearly defined; including the image... This represents the image obtained by upsampling a low-resolution hyperspectral image using n-fold bicubic interpolation. Represents high-resolution multispectral images after The image obtained after downsampling by a factor of 10. Representing high-resolution hyperspectral images after The image obtained after downsampling by a factor of n, where n represents the super-resolution factor. , , , where k represents the current k-th layer.

6. The method according to claim 5, characterized in that, The implementation process of the multi-scale group attention module is as follows: The input image is first processed through convolution, layer normalization, and LeakyReLU activation function to obtain the first feature map, and then spatial attention enhancement is used to obtain the second feature map; The second feature map extracts complementary features through two parallel branches, yielding local and global features respectively. The local and global features are weighted by global average pooling, convolution, and softmax activation functions, respectively, and the features of each branch are weighted to obtain the fused features. The fused features are then subjected to nonlinear transformation and enhancement through a multilayer perceptron, and finally the input image is connected through residuals to obtain the final output.

7. The method according to claim 6, characterized in that, In the multi-scale group attention module, one of the two parallel branches uses a convolution with grouping 1 to obtain local features, and the other branch uses a depthwise separable convolution with grouping s to obtain global features, where s≠1.

8. A hyperspectral image super-resolution system implementing the method of any one of claims 1-7, characterized in that, include: The data acquisition module is responsible for acquiring low-resolution hyperspectral images and corresponding high-resolution multispectral images, and extracting edge-aware noise; The image restoration module is responsible for inputting low-resolution hyperspectral images, corresponding high-resolution multispectral images, and edge-aware noise into the trained and tested edge-aware multimodal residual diffusion model, and using the reverse process to restore the high-resolution hyperspectral image.

9. An electronic device, characterized in that, The method includes a storage component and a processing unit, wherein the storage component stores machine-executable instructions that can be executed by the processing unit, and the processing unit executes the machine-executable instructions to implement the method according to any one of claims 1-7.

10. A machine-readable storage medium, characterized in that, A computer-readable storage medium having program code storage capabilities, wherein the program code carried by the medium contains executable instructions for the method of any one of claims 1-7.

Images