Hyperspectral image super-resolution optimization method based on fused three-branch network
By integrating a three-branch network and a residual difference decomposition coupling strategy, the challenges of spectral fidelity and spatial detail enhancement in hyperspectral image super-resolution technology are solved, achieving efficient feature fusion and complementarity, and improving the quality and stability of image reconstruction.
Patent Information
- Application Number
- CN202610254880.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-12-26
- Filing Date
- 2026-03-04
- Publication Date
- 2026-04-03
AI Technical Summary
Existing hyperspectral image super-resolution techniques struggle to maintain spectral fidelity while enhancing spatial detail, and fail to fully exploit the feature complementarity between different modalities, resulting in unstable training processes and low reproducibility.
A method based on a fusion three-branch network is adopted, which uses a residual difference decomposition coupling strategy to perform targeted repair and enhancement of missing spatial information in hyperspectral images and insufficient spectral information in multispectral images. Combined with adaptive noise suppression and learnable weights, multi-level feature fusion and complementarity between hyperspectral and multispectral images are realized.
It significantly improves the spectral fidelity and spatial accuracy of hyperspectral images, reduces error indicators, enhances training stability and robustness of results, and achieves deep fusion and complementarity between hyperspectral and multispectral images.
Smart Images

Figure CN121788355A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of hyperspectral image processing, and in particular relates to a hyperspectral image super-resolution optimization method based on a fused three-branch network. Background Technology
[0002] Hyperspectral imaging technology can acquire continuous and detailed spectral information from the visible to the infrared bands, providing strong identification and classification support for many key fields such as remote sensing monitoring, medical diagnosis, and food safety. However, due to the limitations of the current physical characteristics and hardware conditions of imaging sensors, directly acquiring high spatial resolution hyperspectral images still faces some challenges. Hyperspectral image super-resolution technology aims to reconstruct corresponding high-resolution images from low spatial resolution hyperspectral images using computer vision methods. In this task, traditional methods often struggle to maintain spectral fidelity while improving spatial details, and generally fail to fully exploit the feature complementarity between different modalities. Existing methods are prone to introducing high-frequency noise during feature encoding, and fixed weights cannot be dynamically adjusted according to the spectral or spatial requirements of different regions, potentially leading to insufficient adaptability. Furthermore, uniform convolutional kernels and initialization strategies cannot specifically match the differentiated needs of spectral detail preservation and spatial texture capture, and the training process often exhibits instability and low reproducibility. Summary of the Invention
[0003] Objective of the Invention: The objective of this invention is to provide a hyperspectral image super-resolution optimization method based on a fusion three-branch network. By introducing a residual differential decoupling fusion strategy, it is possible to perform targeted repair and enhancement of missing spatial information in hyperspectral images and insufficient spectral information in multispectral images, respectively, achieving deep fusion and complementarity of multi-level features between hyperspectral and multispectral images.
[0004] Technical solution: The present invention provides a hyperspectral image super-resolution optimization method based on a fused three-branch network, comprising the following steps:
[0005] Step 1: Acquire and preprocess training data, collect low-resolution hyperspectral image LR-HSI, high-resolution multispectral image HR-MSI and corresponding high-resolution hyperspectral ground truth, and complete normalization, data augmentation and cross-modal feature dimension matching and interaction.
[0006] Step 2: Construct a three-branch network model. Input the low-resolution hyperspectral image LR-HSI into the hyperspectral imaging branch HSI to extract spectral features, and input the high-resolution multispectral image HR-MSI into the multispectral imaging branch MSI to extract spatial features.
[0007] Step 3: Input the spectral and spatial features into the fusion branch after residual calculation and noise suppression, and generate compensation features through convolution and skip connections;
[0008] Step 4: The compensated features are fused through a three-branch process using learnable weights to generate the fused image;
[0009] Step 5: Train the three-branch network model using an optimizer, and optimize the super-resolution of hyperspectral images based on the trained model.
[0010] Further, step 1 specifically involves: selecting a hyperspectral dataset, uniformly adjusting the size of the dataset to 256×256, independently normalizing it by band, applying standard data augmentation operations to the input image patches, specifically including random horizontal and vertical flipping and random rotation, and completing the dimensional alignment and preliminary fusion of HSI and MSI cross-modal features.
[0011] Furthermore, in step 2, specifically: the HSI branch uses 3×3 convolutional layers as the basic operation unit in its architecture, first processing the input features... The initial encoding is performed, which involves 3×3 convolution, batch normalization (BN), and GELU activation to obtain shallow features. The formula is: ; Then adaptive average pooling The features are downsampled to half their original value to generate the desired feature. : ; Represents the proportion; after double convolutional encoding and feature noise suppression module Processing yields spectrally accurate mid-layer core features. : ; Features are extracted through 1 / 4 downsampling and cubic convolutional encoding, and scale generation is restored through bilinear upsampling. , Skip connection features are obtained by 1×1 convolution dimensionality reduction. The two are spliced and fused together, and finally the HSI features with high output spectral fidelity are activated by 1×1 convolution and Sigmoid. The MSI branch uses a 7×7 convolutional kernel to match spatial features. The extraction process involves maintaining symmetry between the MSI and HSI branches, and using a series of convolution operations to extract and enhance spatial texture information from the image.
[0012] Furthermore, step 3 specifically involves: fusing the branches based on the core features of the HSI and MSI branches. As input, the residuals of the two are first calculated to capture the spectral-spatial complementarity error as follows: ; The output characteristics after noise adaptive filtering are: ; in, It is a spectral low-pass filter feature. It is a spatially adaptive gating system, generated from feature variance, in the noisy region. The effective region tends towards 1. It tends towards 0; After noise suppression, the residual features are encoded into 120 dimensions. Then, the process of 1 / 4 scale downsampling, feature enhancement, and bilinear upsampling is repeated. Combined with skip connection feature concatenation, the residual features are reconstructed, and the final output is a residual compensation feature with the same dimension as the input. Throughout the process, the fusion branch uses a 5×5 convolution kernel to balance the expression of spectral and spatial features. At the same time, the residual compensation intensity is dynamically adjusted by learning parameters activated by Sigmoid to avoid feature imbalance.
[0013] Furthermore, step 4 specifically involves: weighted summing of the spectral reference features output by the HSI branch and the residual compensation features generated by the fusion branch to obtain preliminary fusion features. Then, a numerical cropping operation is used to restrict the feature values to the range of 0 to 1. The above process can be expressed by the following formula: ; ; in, The output characteristics of the HSI branch, These are learnable parameters in the network. It is the Sigmoid activation function. Represents the learnable residual weights. To integrate the residual compensation features generated by the branch, This is a numerical clipping function, representing the process of clipping data. All elements are restricted to the interval [0,1]; the final output result By reusing and fusing features, the fusion generation of hyperspectral and multispectral features is completed.
[0014] Further, step 5 specifically involves the AdamW optimizer, with an initial learning rate of 4e-3. It performs linear warm-up in the first 1400 iterations and then gradually cosine annealing to 5e-6 in the subsequent 12600 iterations. This phased learning rate strategy balances the convergence speed of different modules, taking into account both rapid feature extraction in the early stages and fine-tuning of parameters in the later stages. To alleviate overfitting and improve training stability, an exponential moving average (EMA) optimization strategy is integrated, with an EMA decay coefficient set to 0.99. By updating the model weights using an exponential moving average, weight oscillations during training are suppressed.
[0015] Furthermore, after the model training is completed, six metrics are selected to quantitatively evaluate different methods, covering spatial, spectral and global characteristics, including correlation coefficient CC, RMSE, peak signal-to-noise ratio PSNR, universal image quality index UIQI, relative dimensionless global error in synthesis ERGAS and spectral angle mapper SAM; for PSNR, UIQI and CC, higher values indicate better fusion performance, while the ideal value for RMSE, ERGAS and SAM is zero.
[0016] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.
[0017] The present invention also discloses a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method of the present invention.
[0018] The present invention also discloses a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method of the present invention.
[0019] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:
[0020] This invention introduces a residual differential decoupling and fusion strategy, which can target and enhance the missing spatial information in hyperspectral images and the insufficient spectral information in multispectral images, respectively. On the Tiangong-1 dataset, this results in a SAM (spectral angle difference) as low as 1.0118, a 7.3% reduction compared to the suboptimal CNMF (1.0916), and a PSNR (spatial precision) of 49.1587 dB, an 8% improvement compared to STEREO (45.5809 dB), demonstrating that the dual-branch differential extraction and fusion branch collaboration can simultaneously optimize spectral and spatial performance. Through adaptive noise suppression, only network-generated noise is filtered without destroying the effective features of the original data, adapting to complex remote sensing scenarios. By calculating the complementary error of the mid-layer feature residuals for targeted capture, the ERGAS (relative error) is reduced to 0.1744, a 32.6% reduction compared to SCOTT (0.2590), and the RMSE (root mean square error) is 0.0029, a 31% reduction compared to CSTF (0.0042), demonstrating the mechanism's accurate repair capability for cross-modal errors. Learnable residual weights enhance the robustness of the results, achieving optimal results with a CC (correlation coefficient) of 0.9963 and a UIQI (structural similarity index) of 0.9998, representing an improvement of 0.1% to 0.5% compared to the comparison methods. This indicates that the design avoids spectral distortion or spatial artifacts during the fusion process while ensuring feature consistency. Furthermore, EMA (exponential moving average) and a complete log system are employed to improve training stability and traceability. A collaborative learning mechanism with three parallel network branches is designed to achieve deep fusion and complementarity of multi-level features between hyperspectral and multispectral images. Attached Figure Description
[0021] Figure 1 This is a flowchart of the present invention.
[0022] Figure 2 Diagram of the three-branch network structure for super-resolution hyperspectral imaging with fusion branches. Detailed Implementation
[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0024] like Figure 1 As shown, the present invention provides a hyperspectral image super-resolution optimization method based on a fused three-branch network, comprising the following steps:
[0025] Step 1: Acquire and preprocess training data, collect low-resolution hyperspectral image LR-HSI, high-resolution multispectral image HR-MSI and corresponding high-resolution hyperspectral ground truth, and complete normalization, data augmentation and cross-modal feature dimension matching and interaction.
[0026] Step 2: Construct a three-branch network model. Input the low-resolution hyperspectral image LR-HSI into the HSI branch to extract spectral features, and input the high-resolution multispectral image HR-MSI into the MSI branch to extract spatial features.
[0027] Step 3: Input the spectral and spatial features into the fusion branch after residual calculation and noise suppression, and generate compensation features through 5×5 convolution and skip connections;
[0028] Step 4: The compensated features are fused through a three-branch process using learnable weights to generate the fused image;
[0029] Step 5: Train the three-branch network model using the AdamW optimizer, and optimize the super-resolution of hyperspectral images based on the trained model.
[0030] Example
[0031] The network structure and specific process of this invention are as follows: Figure 2 As shown in the diagram, in the overall processing flow, low-resolution hyperspectral images (LR-HSI) and high-resolution multispectral images (HR-MSI) are first preprocessed, sequentially performing spatial scale unification, specific band selection, and data normalization. The preprocessed data is then synchronously input into a pre-processed SPE (Spectral Enhancement) feature interaction network for initial feature enhancement. Subsequently, the enhanced features are distributed in parallel to three paths, inputting into the core three-branch network architecture. The first path inputs to the HSI branch, the second path inputs to the MSI branch, and the third path receives the mid-level features extracted from the first two branches, calculates their residual differences, and uses this difference as input to the fusion branch. The fusion branch follows a sequential processing logic of "residual difference decoupling - noise suppression - upsampling stitching," specifically designed to capture and model the complementary errors between spectral and spatial modes, ultimately generating a refined residual compensation feature. In the final output stage of the network, the residual compensation feature generated by the fusion branch is superimposed on the spectral baseline features provided by the HSI branch as the reconstruction basis. This overlay process is dynamically adjusted using a set of learnable weight parameters to adaptively balance the contributions of different features, thereby generating the final high-resolution hyperspectral image reconstruction result. To verify the model performance, the entire process incorporates an exponential moving average inference mode to stabilize the output, and performs a systematic quantitative evaluation based on a series of objective evaluation metrics, thus completing the comprehensive verification and result output of the super-resolution performance.
[0032] The innovation of this invention lies in abandoning the traditional simple stitching fusion mode, adopting residual differential decoupling for the fusion branches, anchoring the HSI spectral accuracy while using MSI residuals to fill in spatial details; and employing adaptive noise suppression, filtering only the noise generated by the network without destroying the effective features of the original data, adapting to complex remote sensing scenarios; and using EMA equipment to simultaneously solve the training stability problem, taking into account the engineering implementation capability, and can be directly applied to the scenario of super-resolution reconstruction of remote sensing hyperspectral images.
[0033] To evaluate the actual contribution of this network architecture, a comparative experiment was conducted on the Tiangong-1 dataset. The results are shown in Table 1.
[0034] Table 1
[0035]
[0036] As shown in Table 1, compared with existing methods such as G-SOMP+ and CSU, the hyperspectral image super-resolution optimization method based on the fusion three-branch network has significant advantages in all indicators. Specifically, the spectral angle difference (SAM) is as low as 1.0118, which is further reduced compared with the best comparison method CNMF (1.0916), indicating better spectral fidelity; the peak signal-to-noise ratio (PSNR) reaches 49.1587dB, far exceeding the level of about 45dB of methods such as STEREO, and the spatial reconstruction quality is greatly improved; error indicators such as ERGAS (0.1744) and RMSE (0.0029) are also much lower than those of the comparison methods, reflecting that the deviation between the reconstruction results and the true values is smaller; and CC (0.9963) and UIQI (0.9998) are close to 1, proving that the method performs well in both spectral correlation and structural similarity, fully verifying the effectiveness of its residual difference decoupling and three-branch collaborative design in balancing the spectral accuracy and spatial details of hyperspectral images.
[0037] The above embodiments are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and equivalent substitutions without departing from the principle of the present invention. All such improvements and equivalent substitutions to the claims of the present invention fall within the protection scope of the present invention.
Claims
1. A hyperspectral image super-resolution optimization method based on a fused three-branch network, characterized in that, Includes the following steps: Step 1: Acquire and preprocess training data, collect low-resolution hyperspectral image LR-HSI, high-resolution multispectral image HR-MSI and corresponding high-resolution hyperspectral ground truth, and complete normalization, data augmentation and cross-modal feature dimension matching and interaction. Step 2: Construct a three-branch network model. Input the low-resolution hyperspectral image LR-HSI into the hyperspectral imaging branch HSI to extract spectral features, and input the high-resolution multispectral image HR-MSI into the multispectral imaging branch MSI to extract spatial features. Step 3: Input the spectral and spatial features into the fusion branch after residual calculation and noise suppression, and generate compensation features through convolution and skip connections; Step 4: The compensated features are fused through a three-branch process using learnable weights to generate the fused image; Step 5: Train the three-branch network model using an optimizer, and optimize the super-resolution of hyperspectral images based on the trained model.
2. The hyperspectral image super-resolution optimization method based on a fused three-branch network according to claim 1, characterized in that, Step 1 specifically involves: selecting a hyperspectral dataset, uniformly adjusting the dataset size to 256×256, independently normalizing it by band, applying standard data augmentation operations to the input image patches, specifically including random horizontal and vertical flipping and random rotation, and completing the dimensional alignment and preliminary fusion of HSI and MSI cross-modal features.
3. The hyperspectral image super-resolution optimization method based on a fused three-branch network according to claim 1, characterized in that, In step 2, specifically: the HSI branch uses 3×3 convolutional layers as the basic operation unit in its architecture, first processing the input features... The initial encoding is performed, which involves 3×3 convolution, batch normalization (BN), and GELU activation to obtain shallow features. The formula is: ; Then adaptive average pooling The features are downsampled to half their original value to generate the desired feature. : ; Represents the proportion; after double convolutional encoding and feature noise suppression module Processing yields spectrally accurate mid-layer core features. : ; Features are extracted through 1 / 4 downsampling and cubic convolutional encoding, and scale generation is restored through bilinear upsampling. , Skip connection features are obtained by 1×1 convolution dimensionality reduction. The two are spliced and fused together, and finally the HSI features with high output spectral fidelity are activated by 1×1 convolution and Sigmoid. The MSI branch uses a 7×7 convolutional kernel to match spatial features. The extraction process involves maintaining symmetry between the MSI and HSI branches, and using a series of convolution operations to extract and enhance spatial texture information from the image.
4. The hyperspectral image super-resolution optimization method based on a fused three-branch network according to claim 3, characterized in that, Step 3 specifically involves: fusing the branches based on the core features of the HSI and MSI branches. As input, the residuals of the two are first calculated to capture the spectral-spatial complementarity error as follows: ; The output characteristics after noise adaptive filtering are: ; in, It is a spectral low-pass filter feature. It is a spatially adaptive gating system, generated from feature variance, in the noisy region. The effective region tends towards 1. It tends towards 0; After noise suppression, the residual features are encoded into 120 dimensions. Then, the process of 1 / 4 scale downsampling, feature enhancement, and bilinear upsampling is repeated. Combined with skip connection feature concatenation, the residual features are reconstructed, and the final output is a residual compensation feature with the same dimension as the input. Throughout the process, the fusion branch uses a 5×5 convolution kernel to balance the expression of spectral and spatial features. At the same time, the residual compensation intensity is dynamically adjusted by learning parameters activated by Sigmoid to avoid feature imbalance.
5. The hyperspectral image super-resolution optimization method based on a fused three-branch network according to claim 1, characterized in that, Step 4 specifically involves: weighting and summing the spectral reference features output from the HSI branch with the residual compensation features generated by the fusion branch to obtain the preliminary fusion features. Then, a numerical cropping operation is used to restrict the feature values to the range of 0 to 1. The above process can be expressed by the following formula: ; ; in, The output characteristics of the HSI branch, These are learnable parameters in the network. It is the Sigmoid activation function. Represents the learnable residual weights. To integrate the residual compensation features generated by the branches, This is a numerical clipping function, representing the process of clipping data. All elements are restricted to the interval [0,1]; the final output result By reusing and fusing features, the fusion generation of hyperspectral and multispectral features is completed.
6. The hyperspectral image super-resolution optimization method based on a fused three-branch network according to claim 1, characterized in that, Step 5 specifically involves the AdamW optimizer, with an initial learning rate of 4e-3. It performs linear warm-up in the first 1400 iterations, followed by gradual cosine annealing to 5e-6 in the subsequent 12600 iterations. This phased learning rate strategy balances the convergence speed of different modules, taking into account both rapid feature extraction in the early stages and fine-tuning of parameters in the later stages. To alleviate overfitting and improve training stability, an exponential moving average (EMA) optimization strategy is integrated, with an EMA decay coefficient set to 0.
99. By updating the model weights using an exponential moving average, weight oscillations during training are suppressed.
7. The hyperspectral image super-resolution optimization method based on a fused three-branch network according to claim 1, characterized in that, After the model training was completed, six metrics were selected to quantitatively evaluate different methods, covering spatial, spectral and global characteristics, including correlation coefficient CC, RMSE, peak signal-to-noise ratio PSNR, universal image quality index UIQI, relative dimensionless global error in synthesis ERGAS and spectral angle mapper SAM; for PSNR, UIQI and CC, higher values indicate better fusion performance, while the ideal value for RMSE, ERGAS and SAM is zero.
8. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.
9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.
10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.
