A hyperspectral image reconstruction method based on wavelet attention mechanism

Abstract

The present application belongs to the field of RGB to hyperspectral reconstruction, aiming at the problem of insufficient multi-scale frequency feature mining and insufficient cross-channel information fusion of existing methods, a wavelet attention fusion network reconstruction method is proposed. The method relies on three core modules to realize efficient and accurate recovery of hyperspectral images: the wavelet transform attention module decomposes the frequency through multi-scale, and combines three types of sub-modules to strengthen the global structure and local detail features; the bottom feature enhancement attention module captures and enhances the edge, texture and other bottom details with the help of double-dimensional position attention; the HyperRGB Bridge module deeply fuses the RGB and HSI features, and avoids the image degradation problem of network training. Experiments based on the NTIRE 2020-clean dataset show that, compared with mainstream methods such as HRNet, the method performs better in PSNR, RMSE, SAM and other indicators, with PSNR reaching 42.6813dB and RMSE as low as 0.0136, verifying its advancement and effectiveness, and can be widely applied in remote sensing monitoring, medical image analysis and other fields.

Description

Technical Field

[0001] This invention belongs to the field of RGB to hyperspectral reconstruction, and relates to the research on a hyperspectral image reconstruction method based on a wavelet attention mechanism. Background Technology

[0002] RGB-to-spectral reconstruction refers to the task of recovering a hyperspectral image (HSI) containing dozens of continuous spectral bands from an RGB image containing only three bands. Hyperspectral images capture the spectral reflectance characteristics of objects at different wavelengths, enabling detailed characterization of the chemical composition and physical properties of substances. Therefore, they have significant application value in many fields such as remote sensing monitoring, medical image analysis, agricultural crop growth assessment, and environmental pollutant detection.

[0003] However, spectral reconstruction faces numerous technical challenges. On the one hand, RGB images only contain information in three broad bands: red, green, and blue. This creates a significant information gap compared to the dozens of narrow bands in hyperspectral images, making it difficult to directly recover fine spectral features through simple mapping. On the other hand, the reconstruction process must simultaneously ensure the consistency of spatial structure (such as edges and textures) and the accuracy of spectral curves (such as matching reflectance at different wavelengths). Balancing these two aspects places stringent demands on model design. Furthermore, variations in lighting in complex scenes, material mixing, and noise interference in RGB images further increase the difficulty of accurately recovering spectral details.

[0004] Currently, spectral reconstruction methods can be broadly categorized into two types: traditional prior modeling-based methods and deep learning-based methods. The former is a model-driven approach, relying on manually designed spectral priors (such as sparsity and smoothness) to establish the relationship between RGB and HSI through dictionary learning, manifold mapping, and other methods. These methods can achieve basic reconstruction in simple scenes, but due to the limitations of manual priors, they struggle to capture nonlinear spectral relationships in complex scenes, resulting in poor reconstruction accuracy and generalization ability. The performance of traditional methods heavily depends on manual parameter tuning, leading to insufficient robustness when dealing with images of different materials and lighting conditions.

[0005] Conversely, in a data-driven learning paradigm, deep learning methods can automatically learn feature maps from large-scale RGB-HSI paired data, significantly improving spectral reconstruction performance. Based on different network structures, deep learning methods can be divided into three categories: The first category is based on Convolutional Neural Networks (CNNs), such as HSCNN++ and HRNet, which extract local spatial-spectral features through deep convolutions, but struggle to model long-range dependencies, resulting in insufficient spectral continuity. The second category is based on Transformers, such as MST++ and CESST, which utilize self-attention mechanisms to capture global correlations, but their computational complexity increases quadratically with image resolution, leading to low efficiency. The third category is based on emerging state-space models (such as Mamba), represented by GMSR-Net. These methods leverage Mamba's linear computational complexity and global receptive field advantage, combined with the local feature extraction capabilities of CNNs (such as gradient attention mechanisms), achieving a balance between efficiency and global modeling. However, these methods still fall short in the targeted processing of multi-scale frequency features (such as high-frequency details and low-frequency structures), and fail to perform differentiated modeling of different frequency components like wavelet transform, resulting in limited accuracy in spectral detail recovery in complex scenarios.

[0006] Despite the significant progress made by deep learning in spectral reconstruction, existing methods still have obvious shortcomings: they fail to fully exploit multi-frequency features in RGB images (such as low-frequency global structure and high-frequency local details), lack targeted processing of different frequency components, and the cross-channel information fusion is not refined enough, making it difficult to fully utilize the complementarity of the three RGB channels. Therefore, how to efficiently extract multi-scale frequency information and enhance cross-channel interaction has become an urgent problem to be solved in the field of spectral reconstruction. Based on this, we propose a spectral reconstruction method based on a wavelet attention fusion network to achieve more accurate and efficient hyperspectral image restoration. Summary of the Invention

[0007] The purpose of this invention is to provide an efficient and accurate spectral reconstruction method for the field of hyperspectral image restoration. With the application of deep learning in spectral reconstruction, significant progress has been made in reconstruction accuracy and generalization ability. However, existing methods still struggle to fully extract multi-scale frequency features (such as low-frequency global structure and high-frequency local details) from RGB images, and their fusion of cross-channel information is not refined enough. Therefore, this invention proposes a spectral reconstruction method based on a wavelet attention fusion network. By selectively processing different frequency components and enhancing the information interaction between the spatial and spectral dimensions, it extracts more comprehensive features and fully utilizes information from each stage of the network, thereby achieving superior spectral reconstruction results. Its core consists of the following three parts:

[0008] 1. Wavelet Transform Attention Module (WAB)

[0009] For RGB to hyperspectral conversion tasks, RGB images lack the fine spectral information of hyperspectral images, and hyperspectral images exhibit strong coupling between their spectral and spatial dimensions, requiring the simultaneous capture of local spatial details and cross-band spectral correlations. Furthermore, directly extracting features through single-scale convolution can easily lead to redundancy of spectral information or loss of spatial details. Therefore, a wavelet transform attention module (WTB) is proposed to perform multi-scale frequency decomposition and feature interaction on RGB images, fully mining the spectral information in different frequency components.

[0010] First, the input RGB feature map is processed by a 3×3 convolution to obtain a coarse hyperspectral image. The lossless decomposition via Discrete Wavelet Transform (DWT) is as follows:

[0011]

[0012] in For 2×2 neighborhood low-pass filtering, For high-pass filtering, zero parameters are used to achieve "structure-detail" decoupling.

[0013] The Low-Frequency Global-Local Fusion Block (LGMB) is designed specifically for the low-frequency component LL. It collaboratively extracts local texture details and global structural context through parallel local multi-scale perception paths and global spatial relationship modeling paths.

[0014] Local multi-scale perception path: Employing three parallel branches—1×1 convolution (capturing spectral features), 3×3 convolution (capturing spatial features), and 3×3→1×1 convolution (capturing joint spatial-spectral features)—features are fused through element-wise summation to enhance shared activation regions and suppress noise.

[0015]

[0016]

[0017]

[0018]

[0019] Global spatial relationship modeling path: Through layer normalization, spatial multi-head self-attention mechanism and feedforward network, model the long-distance dependency of all spatial locations, and inject global spatial priors:

[0020]

[0021] Cross-domain feature coordination and compression: transforming local features With global features By concatenating along the channel dimension and using 1×1 convolution to achieve information interaction and dimensionality reduction, the output contains low-frequency enhanced features that simultaneously include local details and global structure.

[0022]

[0023] The Two-Dimensional Spatial Attention Module (DSAM) builds upon the LGM output by introducing the DSAM module. Through global statistical distribution capture and local extremum response detection, it enhances the model's ability to resolve complex spatial structures: It performs average pooling (capturing global statistics) and max pooling (capturing local extrema) on the input features respectively; and performs convolutional encoding in the horizontal and vertical orthogonal dimensions to generate a spatial attention map. and The input features are weighted after Sigmoid activation to obtain globally enhanced features. and local enhancement features After splicing, the parts are fused using a linear mapping:

[0024]

[0025]

[0026]

[0027] The Spatial-Spectral Attention Block (SSAB) targets the high-frequency (HF) component. The SSAB module jointly models the feature dependencies of the spatial and spectral dimensions, enhancing texture edge details while maintaining spectral consistency. The spatial attention branch uses two parallel 3×3 convolutional layers: one generates a spatial attention map using sgmiod (dynamically assigning weights to key information), and the other extracts local structural information. The two layers are multiplied element-wise to enhance effective information in salient regions. The spectral attention branch groups the spatially enhanced features by channel (each group contains adjacent spectral bands), generates channel weights through global average pooling and a multilayer perceptron, and then concatenates and reassembles the weighted features.

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] Finally, the processed low-frequency enhancement features High-frequency enhancement features Through inverse wavelet transform Reconstruction yields multi-frequency fusion features:

[0034] 2. Low-level Feature Enhancement Attention Module (LFEA)

[0035] Deep networks can acquire high-level semantic information, but in RGB-to-hyperspectral image reconstruction tasks, the accurate preservation of low-level details (such as edges, textures, and local spatial correlations) directly affects the spectral fidelity and spatial resolution of hyperspectral images. Since RGB images contain only three channels of visible light information, while hyperspectral images contain tens to hundreds of consecutive spectral channels, the loss or distortion of low-level features can lead to problems such as spectral aliasing and spatial blurring in the reconstruction results. To address this, we designed a low-level feature enhancement attention module, aiming to specifically capture and enhance low-level detail features in RGB images that are closely related to hyperspectral reconstruction, providing reliable spatial structure support for subsequent spectral extension.

[0036] The Low-Level Feature Enhancement Attention Module (LFEA) addresses the issue of multi-head spectral self-attention (MSA) modules in backbone networks, which focus on channel-dimensional modeling and are prone to losing spatial details. The LFEA module enhances low-level structural features such as edges and textures through a two-dimensional positional attention mechanism. Feature statistical extraction: [Further details about feature extraction would follow here.] In the width ( and height Calculate the mean along the direction to capture global dependencies in the horizontal and vertical dimensions:

[0037]

[0038] in

[0039] Feature fusion and dimensionality reduction: and Feature fusion and dimensionality compression are achieved through channel-level concatenation using 1×1 convolution, batch normalization (BN), and ReLU activation.

[0040] Attention weight generation: fusing features Split along the height and width dimensions, then perform a 1×1 convolution and sigmoid activation to generate attention weights in the horizontal and vertical directions:

[0041] Weighted augmentation: This method multiplies attention weights by input feature elements to enhance detailed features at key spatial locations.

[0042] 3. HyperRGB Bridge Module

[0043] In reconstruction tasks, the network training process is highly susceptible to image degradation. When deep networks process multimodal data, as the number of layers increases and complex calculations accumulate, the spatial details of RGB images and the spectral information of HSI images will be lost or distorted to varying degrees during transmission.

[0044] Although RGB images only have three visible light channels, they carry rich spatial structural information such as edges and textures, which are the important foundation for constructing HSI spatial features. However, during network training, due to the limited receptive field of convolutional kernels and downsampling operations, the low-level detailed features of RGB images are easily blurred or lost. For example, the sharp edges of objects in an image may become smooth after multiple convolutions, and the details of fine textures are gradually lost, resulting in a significant reduction in spatial resolution of the final fused image, with blurred object edges and unclear spatial structure.

[0045] The HyperRGB Bridge fusion module addresses the image degradation problem during network training by fusing RGB and HSI information through a multi-step process:

[0046] 1. Branch Feature Extraction: The RGB and HSI images are convolutionally (Conv) and ReLU activated respectively to obtain basic features.

[0047]

[0048]

[0049] 2. Spatial Guided Channel Attention (SGCA) Enhances HSI Features: First, a spatial mask is generated using 1×1 convolution.

[0050]

[0051] Then and flattened HSI features Multiplying together Then, channel attention weights are generated through a fully connected layer and a Sigmoid function. ,final

[0052]

[0053] 3. Cross-modal fusion and self-attention: The HSI features enhanced by RGB and SGCA are concatenated and normalized to form bonds (V), carrying the initial RGB feature information. After normalization and mapping to keys (K) and values ​​(V), attention scores are calculated and weighted (V) is added to obtain... ,after The HSI features enhanced by the lower path are concatenated to obtain Finally, it is output after a 1×1 convolution. This achieves deep fusion of RGB and HSI, avoiding image degradation by introducing original RGB information. Attached Figure Description

[0054] Figure 1 A schematic diagram of the overall model of the wavelet attention fusion network for the invention;

[0055] Figure 2 A schematic diagram of the wavelet attention module (WAB) in the model of the invention;

[0056] Figure 3 A schematic diagram of the underlying feature enhancement attention module (LFEA) in the model of the invention;

[0057] Figure 4 This is a schematic diagram of the HyperRGB Bridge fusion module; Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0059] Step 1: Preprocess the data used;

[0060] Step 2: Input the preprocessed data into the network for feature extraction;

[0061] Step 3: Enhance cross-modal feature association using a multi-scale attention module;

[0062] Step 4: Fuse the hierarchical features of the encoder and decoder through a multi-stage feature fusion module;

[0063] Step 5: Fuse data from different modalities to obtain the final hyperspectral reconstruction map;

[0064] Furthermore, step 1 specifically includes the following steps:

[0065] Step 1-1: To ensure the uniformity of model input specifications, improve computational and memory efficiency, simplify the data augmentation process, and facilitate model training, we uniformly processed the input images by cropping and scaling them to a size of 256×256.

[0066] Steps 1-2: Next, the images are first normalized, and then data augmentation operations are performed. Augmentation methods include randomly flipping the image.

[0067] Furthermore, step 2 specifically includes the following steps:

[0068] Step 2-1: Before inputting the RGB image into the network, the 3 channels of the RGB image are first mapped to the channels of the target hyperspectral image through a 3×3 convolution to achieve channel alignment between the RGB and HSI data; then, a 3×3 convolution is used to embed the features to obtain preliminary coarse HSI features, which are used as input to the subsequent encoder-decoder structure and are gradually refined as the network propagates forward.

[0069] Step 2-2: Use MST++ as the backbone architecture of the entire network for feature extraction. Its overall structure consists of an encoder, intermediate layers and a decoder.

[0070] Furthermore, step 3 specifically includes the following steps:

[0071] Step 3-1: Losslessly decompose the initial HSI feature map X into low-frequency component LL and high-frequency component HF = [LH, HL, HH] using Discrete Wavelet Transform (DWT):

[0072]

[0073] Where DWT_{LL} is a 2×2 low-pass filter (preserving global structure) and DWT_{HF} is a high-pass filter (preserving edges / details).

[0074] Step 3-2: The LL components are sequentially processed through the LGMB and DSAM modules to enhance the collaborative representation of global structure and local texture, resulting in... ;

[0075] Step 3-3: The HF component is processed using the SSAB module to enhance edge and texture details while maintaining spectral consistency, resulting in... ;

[0076] Steps 3-4: Use IWT to... and Reconstructed as multi-frequency enhancement features,

[0077] Furthermore, step 4 specifically includes the following steps:

[0078] Step 4-1: Connect the LFEA module at the bottom stage of the encoder and input the deep features output by the encoder;

[0079] Step 4-2: Through the two-dimensional attention mechanism of the LFEA module, horizontal and vertical attention weights are generated to enhance low-level detail features such as edges and textures.

[0080] Step 4-3: Input the enhanced low-level features into the decoder and perform hierarchical fusion with the multi-frequency enhanced features.

[0081] Furthermore, step 5 specifically includes the following steps:

[0082] Step 5-1: Extract the deep spectral features output by the decoder, and input them together with the spatial features of the original RGB image after feature embedding into the HyperRGB Bridge module;

[0083] Step 5-2: Enhance the spatial-spectral consistency of HSI spectral features through the SGCA module, and then achieve deep fusion of RGB spatial details and HSI spectral features through a cross-modal self-attention mechanism;

[0084] Step 5-3: Adjust the channel dimensions of the fused features through a 3×3 convolution and superimpose long-range residual connections to obtain the final hyperspectral reconstructed image. Finally, pass the features obtained in the above process through a 3×3 convolution and a residual connection to obtain the final refined HSI image.

[0085] We used the NTIRE 2020-clean public dataset to verify the advancement and effectiveness of the method in this invention. The NTIRE 2020-clean dataset contains 510 pairs of RGB-HSI samples, covering 31 bands from 400 to 700 nm, including clean and real-noise scenes; the selected dataset covers different resolutions, spectral ranges and scene complexities, ensuring a comprehensive evaluation of the model's generalization ability in diverse scenarios.

[0086] In the CNN method, RepCPSI and HRNet were selected; in the Transformer method, MST++ was selected; and in the Mamba method, GMSR-Net was selected for comparison with the method of this invention. This invention uses PSNR (Peak Signal-to-Noise Ratio), RMSE (Root Mean Square Error), and SAM (Spectral Angle) for quantitative evaluation: higher PSNR and lower RMSE / SAM indicate better reconstruction performance. The results are shown in Table 1.

[0087] Experimental results show that the method of this invention significantly outperforms the comparative methods on all datasets—compared to MST++, PSNR is improved by 4.5891 dB and RMSE is reduced by 0.0026; compared to GMSR-Net, PSNR is increased by 1.2399 dB. Furthermore, ASSIM and SAM also have certain advantages over other methods. These results verify the superiority of the method of this invention.

[0088] Table 1. Comparison of various indicators from RGB to hyperspectral reconstruction

[0089] The results are shown in Table 1. The experimental results demonstrate that the method of this invention is significantly superior to the comparative algorithm, thus verifying the excellence of the method of this invention.

Claims

1. A hyperspectral image reconstruction method based on wavelet attention mechanism, characterized in that, Includes the following steps: S1. Data preprocessing: Perform size unification, normalization, and data augmentation operations on the input RGB images of the three channels; S2. Feature Extraction and Preliminary Mapping: The RGB image channels are mapped to the target HSI channel number through convolution operations to obtain preliminary coarse HSI features, and multi-layer features are extracted using the backbone network of the encoder-decoder structure; S3. Multi-frequency feature enhancement (with wavelet attention block WAB intervention): The initial HSI features are decomposed into low-frequency components (LL) and high-frequency components (HF={LH, HL, HH}) using Discrete Wavelet Transform (DWT). The low-frequency components are then enhanced sequentially using a Low-Frequency Global-Local Fusion Block (LGMB) and a Two-Dimensional Spatial Attention Module (DSAM) to collaboratively capture global structure and local details. The high-frequency components are enhanced using a Spatial-Spectral Attention Block (SSAB) to maintain spectral consistency while enhancing texture edges. The processed low-frequency and high-frequency components are then recombined using Inverse Wavelet Transform (IWT) to obtain multi-frequency enhanced features. S4. Low-level detail enhancement (LFEA module intervention): The Low-level Feature Enhancement Attention Module (LFEA) is used to process the deep RGB features output by the encoder, and the spatial awareness of low-level details such as edges and textures is enhanced through a two-dimensional positional attention mechanism. S5. Cross-modal fusion and reconstruction (HyperRGB Bridge module intervention): The enhanced RGB low-level features and deep HSI features are fused through the HyperRGB Bridge module. Feature adaptive fusion is achieved with the help of Spatial Guided Channel Attention (SGCA) and cross-modal self-attention mechanism. The final hyperspectral reconstructed image is output through convolution operation and residual connection.

2. The hyperspectral image reconstruction method based on wavelet attention fusion network according to claim 1, characterized in that, Step S1 includes the following steps: S1.1 The input image is cropped and scaled to a uniform size of 256×256 to improve computational efficiency and simplify the data processing flow; S1.2 Normalization and data augmentation operations are performed sequentially on the images, including random image flipping to increase the diversity of training samples; S1.3 The training and validation data used the NTIRE2020-Clean public dataset.

3. The hyperspectral image reconstruction method based on wavelet attention fusion network according to claim 1, characterized in that, Step S2 includes the following steps: S2.1 The 3 channels of the RGB image are mapped to the target HSI channel number through 3×3 convolution to achieve channel alignment; then, feature embedding is performed through 3×3 convolution to obtain preliminary coarse HSI features; S2.2 uses MST++ as the backbone network, which consists of an encoder, intermediate layers and a decoder, and includes long-range residual connections. The preliminary coarse HSI features are input into the multi-head spectral self-attention (MSA) module of MST++ to extract global context features in the spatial and spectral domains.

4. The hyperspectral image reconstruction method based on wavelet attention fusion network according to claim 1, characterized in that, Step S3 includes the following steps: S3.1 The initial HSI feature X is losslessly decomposed into low-frequency component LL and high-frequency component HF using discrete wavelet transform (DWT); ; in, It is a 2×2 low-pass filter. For high-pass filtering; S3.2 Low-frequency component LL processing: S3.2.1 Extracting local-global collaborative features via the LGMB module: Local features are obtained by summing three parallel branches: 1×1 convolution (spectral features), 3×3 convolution (spatial features), and 3×3→1×1 convolution (spatial-spectral joint features). Global features are obtained through spatial multi-head self-attention. The two are then concatenated and fused using a 1×1 convolution. S3.2.2 Enhance spatial structure resolution capability through DSAM module: Perform average pooling and max pooling on the LGMB output features respectively to generate spatial attention maps in the horizontal and vertical dimensions, and then fuse and superimpose the residuals after weighting through linear mapping; S3.2.3 Output Low-Frequency Enhancement Characteristics ; S3.3 High-Frequency Component (HF) Processing: Spatial-spectral dependence is jointly modeled using the SSAB module. First, texture edges are enhanced using spatial attention weights. Then, features are grouped by channel, and spectral channel weights are generated through global average pooling and a multilayer perceptron. These weighted features are then concatenated and recombined to output the high-frequency enhanced features. ; S3.4 Multi-frequency feature recombination: via inverse wavelet transform Will and Reconstruction yields multi-frequency enhanced features: .

5. The hyperspectral image reconstruction method based on wavelet attention fusion network according to claim 1, characterized in that, Step S4 includes the following steps: S4.1 Deep RGB Features of Encoder Output The average value is calculated in both the width (dim=3) and height (dim=2) directions to obtain the following results. and After dimensional transformation and splicing, it becomes ; S4.2 Dimensionality reduction and fusion are achieved through 1×1 convolution, batch normalization (BN), and ReLU followed by 1×1 convolution, resulting in... ; S4.3 will Split by height / width, and generate spatial attention weights via 1×1 convolution and Sigmoid. and Weighted output of enhanced low-level features (element-wise multiplication).

6. The hyperspectral image reconstruction method based on wavelet attention fusion network according to claim 1, characterized in that, Step S5 includes the following steps: S5.1 Performs 3×3 convolution and ReLU activation on the enhanced RGB features and deep HSI features respectively to extract information carrying the initial RGB features. and deep HSI features ; S5.2 onwards employs a Spatial Guided Channel Attention (SGCA) module to enhance HSI features: a spatial mask is generated through 1×1 convolution. The spatially weighted features are obtained by multiplying the flattened HSI features; channel attention weights are then generated by passing the fully connected layer and the Sigmoid function. Output Enhancement Features ; S5.3 will enhance HSI features using the SGCA module. After normalization, the query (Q) will be generated, carrying the initial RGB feature information. After normalization and mapping to keys (K) and values ​​(V), attention scores are calculated and weighted (V), then positional encoding (PE) is superimposed to obtain... ; S5.4 will and After concatenating along the channel dimension in the lower branch, the information from both is further extracted and fused using a 3×3 convolution and SGCA module; then it is combined with the previously obtained hybrid features. The image is stitched together again along the channel dimension, then the number of channels is adjusted using 1×1 convolution for further fusion, and ReLU activation is applied; finally, a 3×3 convolution is combined with residual connections to output the final spectrophotometric reconstructed image.

Images