Hyperspectral and multispectral image fusion method and system
By employing an adaptive wavelet transform and spatial-spectral bi-branch feature extraction method, the problems of spectral distortion and texture blurring in hyperspectral and multispectral image fusion were solved, achieving high-precision image data fusion and improving the accuracy of ground feature identification and ecological assessment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAO TONG UNIVERSITY INNER MONGOLIA RESEARCH INSTITUTE
- Filing Date
- 2026-01-26
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies struggle to effectively fuse hyperspectral and multispectral images, leading to spectral distortion or blurred spatial textures. Furthermore, traditional methods cannot flexibly adapt to the spectral morphology of different land cover types.
An adaptive wavelet transform and spatial-spectral bi-branch feature extraction method is adopted. Spectral and spatial features are extracted through one-dimensional and two-dimensional wavelet transforms. Feature fusion is performed by combining a pixel-wise Softmax gating mechanism and a cross-scale bidirectional attention module to generate a high-resolution hyperspectral image.
It achieves the fusion of high spectral precision and high spatial detail image data, significantly improving the accuracy of land cover classification and identification, and is applicable to fields such as grassland ecological monitoring, agricultural growth assessment, mineral exploration and environmental change analysis.
Smart Images
Figure CN122048683A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing technology, specifically to a method and system for hyperspectral and multispectral image fusion. More particularly, it relates to a hyperspectral and multispectral image fusion method based on adaptive wavelet transform and spatial-spectral bi-branch feature extraction. Background Technology
[0002] Hyperspectral (HSI) and multispectral (MSI) image fusion, as an important research direction in remote sensing information processing, aims to combine the rich spectral information of hyperspectral images with the high spatial resolution of multispectral images to obtain super-resolution hyperspectral images (HR-HSI) that possess both hyperspectral and high spatial characteristics. In fields such as grassland ecosystem monitoring, crop growth assessment, mineral exploration, and environmental change detection, the acquisition of high-resolution hyperspectral data is crucial for refined ground cover identification and quantitative inversion. However, limited by the hardware and energy constraints of optical sensors, a single sensor cannot simultaneously achieve both high spectral and high spatial resolution. Therefore, algorithmic fusion of multi-source data has become an important approach to achieving high-quality imagery.
[0003] Traditional hyperspectral and multispectral fusion methods include multi-resolution analysis (MRA), principal component analysis (PCA), nonnegative matrix factorization (NMF), and tensor decomposition. These methods can improve the spatial clarity of fused images to some extent, but they often assume a linear relationship between spectral and spatial information, neglecting the complex nonlinear coupling characteristics in hyperspectral data. Furthermore, fixed wavelet bases or transform functions cannot flexibly adapt to the spectral morphology of different land cover types, leading to spectral distortion or blurred spatial texture in the fusion results.
[0004] In recent years, deep learning methods have demonstrated powerful feature representation and self-learning capabilities in the field of remote sensing image fusion. Convolutional Neural Networks (CNNs) and Transformer architectures, in particular, can automatically learn nonlinear spectral-spatial mappings from large-scale samples. However, single deep network models are often limited by local receptive fields or computational complexity, making it difficult to simultaneously capture global semantic information and local texture details. Furthermore, traditional convolutional structures are weak in frequency domain analysis, failing to effectively capture smooth changes and abrupt changes in the spectral dimension.
[0005] Patent application CN115565045A discloses a method for fusion of hyperspectral and multispectral images based on multi-scale spatial-spectral transformation. This method includes: using a dual-branch network structure to perform cross-branch fusion before fusion to enhance the correlation between features extracted by the dual-branch network; employing multi-head self-attention based on spatial domain partitioning to extract spectral features, effectively acquiring global spectral features; using multi-scale patching to extract spatial features to better mine the rich spatial features in the multispectral image; and then fusing the spectral and spatial features for image reconstruction to obtain a high-resolution hyperspectral image. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for hyperspectral and multispectral image fusion.
[0007] The hyperspectral and multispectral image fusion method provided by the present invention includes:
[0008] Step 1: Acquire hyperspectral and multispectral images of the same spatial region, and perform radiometric calibration, atmospheric correction, and spatial alignment on the hyperspectral and multispectral images; Step 2: Upsample the low-resolution hyperspectral image using bilinear interpolation to align it with the spatial resolution of the multispectral image; crop the processed image data and divide it into training and test sets; Step 3: Input the preprocessed image data into the shallow feature extraction module to extract shallow features; input the shallow features into the one-dimensional wavelet transform module and the two-dimensional wavelet transform module respectively; Step 4: Perform wavelet decomposition on the input features in the spectral dimension using the one-dimensional wavelet transform module to extract low-frequency and high-frequency features in the spectral dimension; perform wavelet decomposition on the input features in the spatial dimension using the two-dimensional wavelet transform module to extract low-frequency and high-frequency features in the spatial dimension. Step 5: Select the wavelet basis with the highest weight from the candidate wavelet basis set for each pixel through a pixel-by-pixel Softmax gating mechanism; perform preliminary fusion of the outputs of the one-dimensional wavelet transform module and the two-dimensional wavelet transform module; Step 6: Input the preliminary fusion results into the spectral branch and the spatial branch respectively for feature extraction; the spectral branch is grouped in the channel dimension, and multi-scale convolution operation is performed on different groups to obtain multi-granularity spectral features; the spatial branch sequentially performs global average pooling, convolution and sigmoid activation operations to generate channel weight coefficients to adaptively weight the input features to obtain enhanced spatial features; Step 7: Input the features output by the spectral branch and the features output by the spatial branch into the cross-scale bidirectional attention module, and realize bidirectional information interaction and fusion between spectral features and spatial features by calculating cross attention; Step 8: Aggregate the interactively fused features and reconstruct them to high resolution through deconvolution and upsampling operations to generate a high-resolution hyperspectral fused image.
[0009] Preferably, the operation process of the shallow feature extraction module is expressed as follows:
[0010] in, As input features, and For batch normalization operations, For learnable scaling factors, The output is the shallow feature. For a 3×3 convolution operation, This is the activation function.
[0011] Preferably, the operation process of the one-dimensional wavelet transform module is expressed as follows:
[0012] The operation process of the two-dimensional wavelet transform module is expressed as follows:
[0013] in, It is the feature map output after one-dimensional wavelet transform; It is a one-dimensional wavelet transform function used to perform frequency domain decomposition of the input signal in the spectral dimension; It is a low-resolution hyperspectral image after upsampling; It is the feature map output after two-dimensional wavelet transform; It is a two-dimensional wavelet transform function used to perform multi-scale decomposition of images in the spatial dimension; It is a high-resolution multispectral image; The operation process of the pixel-by-pixel Softmax gating mechanism and the preliminary fusion process are represented as follows:
[0014]
[0015] in, It is an intermediate fusion feature map. It's a splicing operation. It is a 1×1 convolution operation. This is the final feature map after initial fusion. It is a shallow feature map obtained after the multispectral image Y has passed through the shallow feature extraction module. It is a low-resolution hyperspectral image after upsampling. The shallow feature map is obtained after passing through the shallow feature extraction module.
[0016] Preferably, the computation process of the cross-scale bidirectional attention module is expressed as follows:
[0017]
[0018] in, It is the output feature of the spectral branch after being processed and enhanced by the cross-scale bidirectional attention module. It is the result of the attention mechanism calculation, query ,key ,value It is an attention mapping tensor generated by the interaction between the features of the current branch and the features of another branch; It is a layer normalization operation; It is a multilayer perceptron, used to perform nonlinear transformation and information fusion on normalized features; It is the output feature of the spatial branch after being processed and enhanced by the cross-scale bidirectional attention module; The expression for aggregating features is:
[0019] in, These are the features after aggregation. It is the output feature of the cross-scale bidirectional attention mechanism. These are the multi-granularity spectral features output by the spectral branch extraction module. It is the enhanced spatial feature output by the spatial branch extraction module. It is a cross-scale bidirectional attention computation process.
[0020] Preferably, the grouping and multi-scale convolution operation of the spectral branches specifically includes: dividing the channels of the input features into 2 groups, 4 groups, or 8 groups; performing convolution operations in parallel using convolution kernels of sizes 3×3, 5×5, and 7×7 within each group to extract spectral features at different scales; and concatenating the convolution outputs of each group and each scale along the channel dimension.
[0021] The hyperspectral and multispectral image fusion system provided by the present invention includes: Module M1: Acquires hyperspectral and multispectral images of the same spatial region, and performs radiometric calibration, atmospheric correction, and spatial alignment on the hyperspectral and multispectral images; Module M2: Upsamples the low-resolution hyperspectral image using bilinear interpolation to align it with the spatial resolution of the multispectral image; crops the processed image data and divides it into training and testing sets. Module M3: Inputs the preprocessed image data into the shallow feature extraction module to extract shallow features; inputs the shallow features into the one-dimensional wavelet transform module and the two-dimensional wavelet transform module respectively; Module M4: Performs wavelet decomposition on the input features in the spectral dimension using a one-dimensional wavelet transform module to extract low-frequency and high-frequency features in the spectral dimension; performs wavelet decomposition on the input features in the spatial dimension using a two-dimensional wavelet transform module to extract low-frequency and high-frequency features in the spatial dimension. Module M5: Selects the wavelet basis with the highest weight from the candidate wavelet basis set for each pixel through a pixel-by-pixel Softmax gating mechanism; performs preliminary fusion of the outputs of the one-dimensional wavelet transform module and the two-dimensional wavelet transform module; Module M6: The preliminary fusion results are input into the spectral branch and the spatial branch respectively for feature extraction; the spectral branch is grouped in the channel dimension, and multi-scale convolution operation is performed on different groups to obtain multi-granularity spectral features; the spatial branch sequentially performs global average pooling, convolution and sigmoid activation operations to generate channel weight coefficients to adaptively weight the input features to obtain enhanced spatial features; Module M7: Inputs the features output by the spectral branch and the features output by the spatial branch into the cross-scale bidirectional attention module, and realizes bidirectional information interaction and fusion between spectral features and spatial features by calculating cross attention; Module M8 aggregates the interactively fused features and reconstructs them to high resolution through deconvolution and upsampling operations, generating a high-resolution hyperspectral fused image.
[0022] Preferably, the operation process of the shallow feature extraction module is expressed as follows:
[0023] in, As input features, and For batch normalization operations, For learnable scaling factors, The output is the shallow feature. For a 3×3 convolution operation, This is the activation function.
[0024] Preferably, the operation process of the one-dimensional wavelet transform module is expressed as follows:
[0025] The operation process of the two-dimensional wavelet transform module is expressed as follows:
[0026] in, It is the feature map output after one-dimensional wavelet transform; It is a one-dimensional wavelet transform function used to perform frequency domain decomposition of the input signal in the spectral dimension; It is a low-resolution hyperspectral image after upsampling; It is the feature map output after two-dimensional wavelet transform; It is a two-dimensional wavelet transform function used to perform multi-scale decomposition of images in the spatial dimension; It is a high-resolution multispectral image; The operation process of the pixel-by-pixel Softmax gating mechanism and the preliminary fusion process are represented as follows:
[0027]
[0028] in, It is an intermediate fusion feature map. It's a splicing operation. It is a 1×1 convolution operation. This is the final feature map after initial fusion. It is a shallow feature map obtained after the multispectral image Y has passed through the shallow feature extraction module. It is a low-resolution hyperspectral image after upsampling. The shallow feature map is obtained after passing through the shallow feature extraction module.
[0029] Preferably, the computation process of the cross-scale bidirectional attention module is expressed as follows:
[0030]
[0031] in, It is the output feature of the spectral branch after being processed and enhanced by the cross-scale bidirectional attention module. It is the result of the attention mechanism calculation, query ,key ,value It is an attention mapping tensor generated by the interaction between the features of the current branch and the features of another branch; It is a layer normalization operation; It is a multilayer perceptron, used to perform nonlinear transformation and information fusion on normalized features; It is the output feature of the spatial branch after being processed and enhanced by the cross-scale bidirectional attention module; The expression for aggregating features is:
[0032] in, These are the features after aggregation. It is the output feature of the cross-scale bidirectional attention mechanism. These are the multi-granularity spectral features output by the spectral branch extraction module. It is the enhanced spatial feature output by the spatial branch extraction module. It is a cross-scale bidirectional attention computation process.
[0033] Preferably, the grouping and multi-scale convolution operation of the spectral branches specifically includes: dividing the channels of the input features into 2 groups, 4 groups, or 8 groups; performing convolution operations in parallel using convolution kernels of sizes 3×3, 5×5, and 7×7 within each group to extract spectral features at different scales; and concatenating the convolution outputs of each group and each scale along the channel dimension.
[0034] Compared with the prior art, the present invention has the following beneficial effects: The hyperspectral and multispectral image fusion method of this invention is applicable to a wide range of fields, including remote sensing monitoring, land cover identification, and ecological assessment. By fusing the rich spectral information of hyperspectral images with the high spatial resolution of multispectral images, it can simultaneously obtain image data with high spectral accuracy and high spatial detail. This method not only significantly improves the accuracy of land cover classification and identification, but also plays an important role in grassland ecological monitoring, agricultural growth assessment, mineral exploration, and environmental change analysis. The fused high-resolution hyperspectral images provide more refined, stable, and interpretable data support for ecosystem structure monitoring, resource management, and climate response analysis. Attached Figure Description
[0035] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a hyperspectral and multispectral image fusion method. Figure 2 This is a schematic diagram of an embodiment; Figure 3a , Figure 3b and Figure 3c These are low-resolution hyperspectral images, high-resolution multispectral images, and fused images, respectively. Detailed Implementation
[0036] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0037] Example 1 like Figure 1 and Figure 2 This invention provides a hyperspectral and multispectral image fusion method based on adaptive wavelet transform and spatial-spectral bi-branch feature extraction, comprising the following steps: Step 1: Dataset Collection; Collect hyperspectral (HSI) and multispectral (MSI) data to ensure that both cover the same spatial area, and obtain corresponding ground feature samples or scene information according to actual needs.
[0038] Step 2: Data preprocessing; Radiometric calibration, atmospheric correction, and spatial alignment were performed on the original HSI and MSI data. The data was then cropped and up-cropped to create training and test sets for subsequent validation. Bilinear interpolation upsampling was applied to the LR-HSI data to maintain its spatial resolution consistent with HR-HSI for subsequent feature fusion.
[0039] Step 3: Pixel-by-pixel adaptive wavelet transform; The pixel-wise wavelet adaptive wavelet transform mainly includes a shallow feature extraction module, a 1D wavelet transform module, and a 2D wavelet transform module, which perform feature extraction in the spatial and frequency domains, and finally perform preliminary fusion in the spatial domain.
[0040] The shallow feature extraction module (SEM) mainly consists of two convolutional layers, two normalization layers, and an activation function. It uses residual connections and introduces learnable scaling parameters. Initially set to 0.2, gradient updates will be performed during subsequent backpropagation. This effectively suppresses excessive amplification of residual terms, balances the intensity of gradient propagation and feature perturbation, and allows the network to focus more on enhancing spatial details.
[0041] The expression for SEM is:
[0042] in, As input features, and For batch normalization operations, For learnable scaling factors, The output is the shallow feature. For a 3×3 convolution operation, This is the activation function.
[0043] The 1D wavelet transform module is mainly used for extracting hyperspectral data in the spectral dimension. Hyperspectral images have characteristics such as local absorption peaks and abrupt changes in reflection in the spectral dimension. Therefore, the "local basis" in 1D wavelet transform can naturally adapt to these characteristics. By using a filter, the frequency domain is decomposed into low-frequency and high-frequency parts. The low-frequency part can capture the smoothness of the overall spectrum, while the high-frequency part can capture the abrupt changes in spectral lines and the details of absorption peaks. Extraction is performed in the frequency domain, and finally, dimensionality is restored in the spatial domain.
[0044] The expression for 1D wavelet transform is:
[0045] The 2D wavelet transform module can perform multi-scale decomposition of images in the spatial dimension, dividing the image into low-frequency and high-frequency sub-bands. The low-frequency part mainly preserves the overall structure and brightness information, while the high-frequency part mainly captures information such as texture and edges. Compared with traditional convolution and Fourier convolution, 2D wavelet transform has localization features in both the spatial and frequency domains, which can better and more accurately describe the texture boundary information of ground objects.
[0046] The expression for 2D wavelet transform is:
[0047] in, It is the feature map output after one-dimensional wavelet transform; It is a one-dimensional wavelet transform function used to perform frequency domain decomposition of the input signal in the spectral dimension; It is a low-resolution hyperspectral image after upsampling; It is the feature map output after two-dimensional wavelet transform; It is a two-dimensional wavelet transform function used to perform multi-scale decomposition of images in the spatial dimension; It is a high-resolution multispectral image; The initial fusion, for each pixel, involves the autonomous selection of a wavelet basis for each pixel. This is equivalent to implementing "learnable wavelet dictionary selection" in both the spectral and spatial dimensions, constructing different sets of wavelet basis functions using arrays. Adaptation to different features is achieved in a soft manner through a pixel-wise gating system. For each pixel, a softmax function is used to analyze the weights of candidate wavelet basis functions, and the system autonomously selects the one with the highest weight as the wavelet basis function for that pixel, creating a convex structure for the candidates. This is analogous to a scoring mechanism, where the wavelet basis function with the highest score for each pixel is extracted in the frequency domain. Finally, the data is concatenated and convolved in the spatial dimension to restore the original dimension.
[0048] The initial fusion expression:
[0049]
[0050] in, It is an intermediate fusion feature map. It's a splicing operation. It is a 1×1 convolution operation. This is the final feature map after initial fusion. It is a shallow feature map obtained after the multispectral image Y has passed through the shallow feature extraction module. It is a low-resolution hyperspectral image after upsampling. The shallow feature map is obtained after passing through the shallow feature extraction module.
[0051] Step 4: Cross-scale bidirectional attention mechanism; By dividing the data into windows, we can avoid computational complexity and improve efficiency. For two inputs, we generate Q, K, and V values respectively, and then perform cross-products for better interaction. Finally, we use LN and MLP for nonlinear transformation to better learn the similarities and differences between the two inputs, enabling feature fusion. Our other input is the original low-resolution hyperspectral image, which aims to compensate for feature loss in convolution and wavelet transform across scales.
[0052] The expression for bidirectional attention can be represented as:
[0053]
[0054] in, It is the output feature of the spectral branch after being processed and enhanced by the cross-scale bidirectional attention module. It is the result of the attention mechanism calculation, query ,key ,value It is an attention mapping tensor generated by the interaction between the features of the current branch and the features of another branch; It is a layer normalization operation; It is a multilayer perceptron, used to perform nonlinear transformation and information fusion on normalized features; It is the output feature of the spatial branch after being processed and enhanced by the cross-scale bidirectional attention module; Step 5: Spectral branch extraction module; We use the initial fusion result of adaptive wavelet transform as input, grouping it into 2, 4, and 8 channels. Then, we perform multi-scale convolution on different groups. This grouping physically locks related adjacent bands in the same subspace, reducing ineffective coupling between groups. After convolution on different groups, we use a cross-scale bidirectional attention mechanism to compensate for the loss of detailed features caused by convolution, and use the attention mechanism to better facilitate interactive learning between features.
[0055] Step Six: Spatial Branch Extraction Module; In the spatial feature extraction module, we use global average pooling, convolution, and sigmoid to form a weighting mechanism that allows the network to automatically learn the weight ratio of each channel, thus achieving adaptive weighting of features. The two inputs are the preliminary fusion result of wavelet transform and the original multispectral data, respectively, which compensates for the spatial loss in convolution and wavelet transform.
[0056] Step 7: Integration Module; The features output by the spatial and spectral modules are further learned in the input cross-scale bidirectional attention, respectively, to learn the regions of interest between different modalities, enhance effective information, and enable the final output to be improved in both spectral fidelity and spatial clarity.
[0057] The expression for the fusion module is:
[0058] in, These are the features after aggregation. It is the output feature of the cross-scale bidirectional attention mechanism. These are the multi-granularity spectral features output by the spectral branch extraction module. It is the enhanced spatial feature output by the spatial branch extraction module. It is a cross-scale bidirectional attention computation process.
[0059] Step 8: Experimental Environment; The system was tested using MATLAB R 2014a, equipped with an Intel Core i9 processor and 32GB of RAM, running Windows 11. Deep learning methods were primarily implemented using Python 3.8 and PyTorch 1.8, accelerated by an NVIDIA RTX 4060Ti graphics card. Data preprocessing and analysis were performed using MATLAB R 2014a, along with Python's NumPy and Pandas libraries. The learning rate was set to 1e-4.
[0060] like Figure 3a , Figure 3b and Figure 3cThese are low-resolution hyperspectral images, high-resolution multispectral images, and fused images, respectively.
[0061] A hyperspectral and multispectral image fusion method based on adaptive wavelet transform and spatial-spectral bi-branch feature extraction has broad application prospects. This method, through collaborative modeling in the frequency and spatial domains, can simultaneously maintain spectral consistency and spatial detail clarity, providing strong technical support for high-precision remote sensing image analysis. In grassland ecological monitoring, it can be used to achieve high-precision aboveground biomass inversion, vegetation health assessment, and degraded grassland identification; in agricultural remote sensing, it can be used for crop growth monitoring, yield estimation, and pest and disease identification; furthermore, this method also has significant application value in mineral exploration, forestry resource surveys, ecological environment assessment, and disaster early warning. By further combining deep learning and temporal analysis, dynamic tracking and prediction of the spectral-spatial characteristics of ground objects can be achieved, providing scientific data support and technical assurance for ecosystem management and sustainable development.
[0062] Example 2 This invention relates to a hyperspectral and multispectral image fusion method based on adaptive wavelet transform and spatial-spectral bi-branch feature extraction, belonging to the field of remote sensing image processing and image super-resolution reconstruction technology. This method introduces a pixel-wise adaptive wavelet transform and spatial-spectral bi-branch fusion framework, interspersed with a cross-scale bidirectional attention mechanism, to fully utilize the spectral information of the hyperspectral image and the spatial details of the multispectral image. Finally, through evaluation criteria, it ensures high-quality high-resolution hyperspectral image (HR-HSI) reconstruction.
[0063] The method of the present invention includes the following main steps: 1. Data acquisition and preprocessing; This research method is adapted to both public datasets and UAV data equipped with hyperspectral imagers. It performs preliminary simple processing on HSI and MSI data of the same region, including radiometric calibration, atmospheric correction, noise removal, and spatial registration. The data is also cropped, upsampled, and dimensionally aligned for subsequent data processing and experiments.
[0064] 2. Construct the network framework; 2.1 Adaptive wavelet transform; Shallow features are extracted from the preprocessed HSI and MSI, respectively, and an adaptive wavelet transform module is introduced. Specifically, in the spectral dimension, a one-dimensional wavelet transform (1D Wavelet) is used to extract spectral features such as smooth transitions and absorption peaks, while in the spatial dimension, a two-dimensional wavelet transform (2D Wavelet) is used to extract texture and edge information of ground features. A pixel-wise Softmax gating mechanism is used to select the optimal wavelet basis, achieving adaptive modeling of spectral and spatial features.
[0065] 2.2 Spatial-Spectral Bi-branch Fusion Framework; By designing a dual-path structure with spectral and spatial branches, spectral feature compression and spatial texture enhancement are achieved respectively. The spectral branch extracts multi-scale band correlation features, while the spatial branch captures high-frequency details and structural features, thus realizing the complementary fusion of spectral and spatial information.
[0066] 2.3. Cross-scale bidirectional attention mechanism; The features output from the two branches are fused at multiple scales, and a cross-scale bidirectional attention mechanism is introduced to establish a long-range dependency between spatial and spectral data, enhancing information interaction. Finally, a high-resolution hyperspectral image (HR-HSI) is generated through reconstruction, achieving a balance between spectral consistency and spatial detail enhancement.
[0067] 3. Evaluation indicators; To comprehensively evaluate the performance of image fusion algorithms, several quantitative metrics are typically used for comparison, namely RMSE, PSNR, SAM, and ERGAS. These metrics are applied to verify the effectiveness of this framework in low-resolution hyperspectral and high-resolution multispectral image fusion tasks.
[0068] Root-Mean-Squared Error (RMSE): This measures the energy of the error between the fused result and the true value (the square root of the mean square error), directly reflecting pixel-level intensity deviation. The formula for calculating RMSE is:
[0069] Where H, W, and C represent the height, width, and number of bands of the image, respectively, and the lower the value, the better the quality of the reconstructed image; c, i, and j are the indices of H, W, and C, respectively. For real reference image, To fuse and reconstruct images; Peak Signal-to-Noise Ratio (PSNR): PSNR is currently the most widely used and commonly used objective method for measuring image quality. Essentially, it is an error suppression capability metric in the form of signal-to-noise ratio. The formula for calculating PSNR is:
[0070] Here, H and W represent the height and width of the image, respectively. The higher the value of the index, the higher the signal quality and the more fully the details are restored; L is the maximum possible value of the image pixel value (peak signal strength).
[0071] Spectral Angle Mapper (SAM): This measures the angle between spectral vectors and is used to calculate the similarity between two spectral curves, effectively measuring spectral differences between remotely sensed images. The formula for calculating SAM is:
[0072] Here, H and W represent the height and width of the image. This represents the inner product operation. The more consistent the spectral shape and the smaller the value, the better the spectral fidelity. For the real reference image at the position The L2 norm of the spectral vector at that point. To fuse and reconstruct images at location The L2 norm of the spectral vector at that location.
[0073] ERGAS (Earning Relative Globalization of Synthetic Errors): Essentially, it represents the mean square deviation of normalized spectral errors, quantifying the relative error changes between bands. A lower ERGAS value indicates better performance. The formula for calculating ERGAS is:
[0074] Where h represents the resolution of HR-MSI, l is the resolution of LR-HSI, C is the number of spectra, and B is the number of spectra. i For the emissivity of HR-MSI, M i This represents the average emissivity of the HR-MSI.
[0075] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.
[0076] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A method for fusion of hyperspectral and multispectral images, characterized in that, include: Step 1: Acquire hyperspectral and multispectral images of the same spatial region, and perform radiometric calibration, atmospheric correction, and spatial alignment on the hyperspectral and multispectral images; Step 2: Upsample the low-resolution hyperspectral image using bilinear interpolation to align it with the spatial resolution of the multispectral image; crop the processed image data and divide it into training and test sets; Step 3: Input the preprocessed image data into the shallow feature extraction module to extract shallow features; input the shallow features into the one-dimensional wavelet transform module and the two-dimensional wavelet transform module respectively; Step 4: Perform wavelet decomposition on the input features in the spectral dimension using the one-dimensional wavelet transform module to extract low-frequency and high-frequency features in the spectral dimension; perform wavelet decomposition on the input features in the spatial dimension using the two-dimensional wavelet transform module to extract low-frequency and high-frequency features in the spatial dimension. Step 5: Select the wavelet basis with the highest weight from the candidate wavelet basis set for each pixel through a pixel-by-pixel Softmax gating mechanism; perform preliminary fusion of the outputs of the one-dimensional wavelet transform module and the two-dimensional wavelet transform module; Step 6: Input the preliminary fusion results into the spectral branch and spatial branch respectively for feature extraction; The spectral branch groups the data along the channel dimension and performs multi-scale convolution operations on different groups to obtain multi-granularity spectral features. The spatial branch sequentially performs global average pooling, convolution, and sigmoid activation operations to generate channel weight coefficients to adaptively weight the input features, resulting in enhanced spatial features. Step 7: Input the features output by the spectral branch and the features output by the spatial branch into the cross-scale bidirectional attention module, and realize bidirectional information interaction and fusion between spectral features and spatial features by calculating cross attention; Step 8: Aggregate the interactively fused features and reconstruct them to high resolution through deconvolution and upsampling operations to generate a high-resolution hyperspectral fused image.
2. The hyperspectral and multispectral image fusion method according to claim 1, characterized in that, The operation process of the shallow feature extraction module is represented as follows: in, As input features, and For batch normalization operations, For learnable scaling factors, The output is the shallow feature. For a 3×3 convolution operation, This is the activation function.
3. The hyperspectral and multispectral image fusion method according to claim 2, characterized in that, The operation process of the one-dimensional wavelet transform module is expressed as follows: The operation process of the two-dimensional wavelet transform module is expressed as follows: in, It is the feature map output after one-dimensional wavelet transform; It is a one-dimensional wavelet transform function used to perform frequency domain decomposition of the input signal in the spectral dimension; It is a low-resolution hyperspectral image after upsampling; It is the feature map output after two-dimensional wavelet transform; It is a two-dimensional wavelet transform function used to perform multi-scale decomposition of images in the spatial dimension; It is a high-resolution multispectral image; The operation process of the pixel-by-pixel Softmax gating mechanism and the preliminary fusion process are represented as follows: in, It is an intermediate fusion feature map. It's a splicing operation. It is a 1×1 convolution operation. This is the final feature map after initial fusion. It is a shallow feature map obtained after the multispectral image Y has passed through the shallow feature extraction module. It is a low-resolution hyperspectral image after upsampling. The shallow feature map is obtained after passing through the shallow feature extraction module.
4. The hyperspectral and multispectral image fusion method according to claim 3, characterized in that, The computation process of the cross-scale bidirectional attention module is represented as follows: in, It is the output feature of the spectral branch after being processed and enhanced by the cross-scale bidirectional attention module. It is the result of the attention mechanism calculation, query ,key ,value It is an attention mapping tensor generated by the interaction between the features of the current branch and the features of another branch; It is a layer normalization operation; It is a multilayer perceptron, used to perform nonlinear transformation and information fusion on normalized features; It is the output feature of the spatial branch after being processed and enhanced by the cross-scale bidirectional attention module; The expression for aggregating features is: in, These are the features after aggregation. It is the output feature of the cross-scale bidirectional attention mechanism. These are the multi-granularity spectral features output by the spectral branch extraction module. It is the enhanced spatial feature output by the spatial branch extraction module. It is a cross-scale bidirectional attention computation process.
5. The hyperspectral and multispectral image fusion method according to claim 1, characterized in that, The grouping and multi-scale convolution operations of the spectral branches specifically include: dividing the channels of the input features into 2, 4, or 8 groups; performing convolution operations in parallel using convolution kernels of sizes 3×3, 5×5, and 7×7 within each group to extract spectral features at different scales; and concatenating the convolution outputs of each group and each scale along the channel dimension.
6. A hyperspectral and multispectral image fusion system, characterized in that, include: Module M1: Acquires hyperspectral and multispectral images of the same spatial region, and performs radiometric calibration, atmospheric correction, and spatial alignment on the hyperspectral and multispectral images; Module M2: Upsamples the low-resolution hyperspectral image using bilinear interpolation to align it with the spatial resolution of the multispectral image; crops the processed image data and divides it into training and testing sets. Module M3: Inputs the preprocessed image data into the shallow feature extraction module to extract shallow features; inputs the shallow features into the one-dimensional wavelet transform module and the two-dimensional wavelet transform module respectively; Module M4: Performs wavelet decomposition on the input features in the spectral dimension using a one-dimensional wavelet transform module to extract low-frequency and high-frequency features in the spectral dimension; performs wavelet decomposition on the input features in the spatial dimension using a two-dimensional wavelet transform module to extract low-frequency and high-frequency features in the spatial dimension. Module M5: Selects the wavelet basis with the highest weight from the candidate wavelet basis set for each pixel through a pixel-by-pixel Softmax gating mechanism; performs preliminary fusion of the outputs of the one-dimensional wavelet transform module and the two-dimensional wavelet transform module; Module M6: Input the preliminary fusion results into the spectral branch and spatial branch respectively for feature extraction; The spectral branch groups the data along the channel dimension and performs multi-scale convolution operations on different groups to obtain multi-granularity spectral features. The spatial branch sequentially performs global average pooling, convolution, and sigmoid activation operations to generate channel weight coefficients to adaptively weight the input features, resulting in enhanced spatial features. Module M7: Inputs the features output by the spectral branch and the features output by the spatial branch into the cross-scale bidirectional attention module, and realizes bidirectional information interaction and fusion between spectral features and spatial features by calculating cross attention; Module M8 aggregates the interactively fused features and reconstructs them to high resolution through deconvolution and upsampling operations, generating a high-resolution hyperspectral fused image.
7. The hyperspectral and multispectral image fusion system according to claim 6, characterized in that, The operation process of the shallow feature extraction module is represented as follows: in, As input features, and For batch normalization operations, For learnable scaling factors, The output is the shallow feature. For a 3×3 convolution operation, This is the activation function.
8. The hyperspectral and multispectral image fusion system according to claim 7, characterized in that, The operation process of the one-dimensional wavelet transform module is expressed as follows: The operation process of the two-dimensional wavelet transform module is expressed as follows: in, It is the feature map output after one-dimensional wavelet transform; It is a one-dimensional wavelet transform function used to perform frequency domain decomposition of the input signal in the spectral dimension; It is a low-resolution hyperspectral image after upsampling; It is the feature map output after two-dimensional wavelet transform; It is a two-dimensional wavelet transform function used to perform multi-scale decomposition of images in the spatial dimension; It is a high-resolution multispectral image; The operation process of the pixel-by-pixel Softmax gating mechanism and the preliminary fusion process are represented as follows: in, It is an intermediate fusion feature map. It's a splicing operation. It is a 1×1 convolution operation. This is the final feature map after initial fusion. It is a shallow feature map obtained after the multispectral image Y has passed through the shallow feature extraction module. It is a low-resolution hyperspectral image after upsampling. The shallow feature map is obtained after passing through the shallow feature extraction module.
9. The hyperspectral and multispectral image fusion system according to claim 8, characterized in that, The computation process of the cross-scale bidirectional attention module is represented as follows: in, It is the output feature of the spectral branch after being processed and enhanced by the cross-scale bidirectional attention module. It is the result of the attention mechanism calculation, query ,key ,value It is an attention mapping tensor generated by the interaction between the features of the current branch and the features of another branch; It is a layer normalization operation; It is a multilayer perceptron, used to perform nonlinear transformation and information fusion on normalized features; It is the output feature of the spatial branch after being processed and enhanced by the cross-scale bidirectional attention module; The expression for aggregating features is: in, These are the features after aggregation. It is the output feature of the cross-scale bidirectional attention mechanism. These are the multi-granularity spectral features output by the spectral branch extraction module. It is the enhanced spatial feature output by the spatial branch extraction module. It is a cross-scale bidirectional attention computation process.
10. The hyperspectral and multispectral image fusion system according to claim 6, characterized in that, The grouping and multi-scale convolution operations of the spectral branches specifically include: dividing the channels of the input features into 2, 4, or 8 groups; performing convolution operations in parallel using convolution kernels of sizes 3×3, 5×5, and 7×7 within each group to extract spectral features at different scales; and concatenating the convolution outputs of each group and each scale along the channel dimension.