A hyperspectral fusion imaging method and system based on hierarchical frequency integrated network

Abstract

This invention provides a hyperspectral fusion imaging method and system based on a hierarchical frequency integration network, solving the problem of achieving both high spatial and hyperspectral resolution with a single sensor, and fully utilizing the spectral and spatial complementarity of hyperspectral and multispectral images. First, shallow features are extracted from low-resolution hyperspectral and high-resolution multispectral images. Then, a spatial spectral information extraction module uses a spectral window self-attention and spatial alignment mechanism to extract features and correct degradation. Next, a frequency domain dual-branch interactive learning module fuses phase and amplitude spectra in local and global frequency spaces respectively, and combines difference information guidance and high-pass filtering to enhance high-frequency details. Finally, image reconstruction and residual concatenation generate a high-resolution hyperspectral image. A visualization system based on Python Qt is also provided to realize algorithm integration and multi-dimensional quality evaluation. This invention overcomes the limitations of a single sensor and improves the spatial clarity and spectral fidelity of fused images.

Description

Technical Field

[0001] This invention belongs to the field of computational imaging and remote sensing image processing, and specifically relates to a hyperspectral fusion imaging method and system based on a hierarchical frequency integration network. Background Technology

[0002] Hyperspectral imaging technology aims to construct high-dimensional spectral-spatial joint representations of ground objects. However, limited by the physical characteristics and hardware bottlenecks of imaging sensors, a single imaging modality cannot simultaneously achieve high-precision spatial resolution and high-dimensional spectral resolution. Therefore, multi-source information fusion has become a mainstream reconstruction paradigm, which utilizes multispectral images as auxiliary modalities to compensate for the spatial details missing in hyperspectral images, thereby achieving cross-modal feature enhancement and super-resolution reconstruction.

[0003] As a typical multi-source computational imaging task, the core of hyperspectral and multispectral image fusion lies in fully mining and aligning complementary features between heterogeneous data. Although existing deep learning-based fusion methods have made some progress, they still have significant shortcomings in high-dimensional feature representation and deep interaction mechanisms. First, most existing methods are limited to feature learning in the spatial domain, seriously ignoring the key structural priors contained in the frequency domain. During the extraction of spatial and spectral features, due to the lack of effective geometric constraints and cross-modal alignment mechanisms, the model is prone to feature degradation when performing high-dimensional mapping, resulting in significant spectral distortion and spatial artifacts in the reconstruction results, thereby reducing the spatial structural consistency and spectral fidelity of the fused image.

[0004] Secondly, traditional models based on deep networks or priors often struggle to maintain fine-grained structural integrity in complex terrain scenes. Feature extraction and downsampling processes are highly susceptible to irreversible loss of high-frequency information. In fact, hyperspectral and multispectral images exhibit complex and highly complementary nonlinear mapping relationships in the frequency domain. Global and local spectral analysis reveals that the phase spectrum of multispectral images contains extremely rich texture and structural priors, and the frequency domain energy distribution of the two differs significantly in different local perceptual regions. Existing methods struggle to dynamically adapt to the complementary differences in the frequency domain across diverse scenes, lacking a unified fusion framework that integrates spatial deformation alignment modeling with hierarchical frequency domain multi-branch interactive learning.

[0005] Therefore, a general fusion method capable of integrating joint spatial spectral modeling and hierarchical frequency integration is urgently needed. This method must effectively overcome degradation defects such as spectral distortion and spatial artifacts, achieve deep fusion of multimodal information in both local and global frequency spaces, and explicitly compensate for high-frequency information and enhance differential information to maximize the integrity of spatial structure while maintaining spectral fidelity. Furthermore, to meet the needs of practical engineering applications and quantitative evaluation, a supporting hyperspectral fusion imaging visualization system with multi-algorithm integration and multi-dimensional evaluation index calculation capabilities is also urgently required. Summary of the Invention

[0006] The technical problem this invention aims to solve is the inherent flaws of existing hyperspectral fusion imaging methods, which suffer from spectral distortion and spatial artifacts during feature extraction, and the loss of high-frequency texture details due to neglecting frequency domain information. This invention provides a hyperspectral fusion imaging method and system based on a hierarchical frequency integration network. By integrating spatial spectral joint modeling with hierarchical frequency domain multi-branch interactive learning, this invention effectively overcomes the degradation phenomenon in high-dimensional feature mapping, significantly improving the spectral fidelity of the fused image while maximizing the preservation of spatial structural consistency and high-frequency details in complex scenes.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A hyperspectral fusion imaging method based on a hierarchical frequency integration network includes:

[0009] 1) Perform spatial and spectral dimension alignment preprocessing on the input low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI), and extract shallow features through convolution to obtain initial hyperspectral features and initial multispectral features.

[0010] 2) Construct a hierarchical frequency integration network, which includes a spatial spectral information extraction module, a frequency domain dual-branch interactive learning module, and an image reconstruction module.

[0011] 3) Input the initial hyperspectral features and initial multispectral features into the spatial spectral information extraction module, and extract the spatial spectral features through a spectral-based window self-attention mechanism.

[0012] 4) Input the extracted spatial spectral features and the initial multispectral features into the frequency domain dual-branch interactive learning module, perform local and global information fusion in the frequency domain respectively, and combine difference information guidance and high-frequency information enhancement to output local frequency domain fusion features and global frequency domain fusion features.

[0013] 5) The frequency domain fusion features are reconstructed by the image reconstruction module to obtain residual features, and the reconstruction result is added to the low-resolution hyperspectral image after upsampling to generate the final high-resolution hyperspectral image.

[0014] Optionally, the detailed steps in step 2) include:

[0015] 2.1) The topology of the hierarchical frequency integration network is configured such that the extracted initial hyperspectral features and initial multispectral features are first input to the spatial spectral information extraction module to extract spatial spectral features. Then, the spatial spectral features and shallow features are input together to the frequency domain dual-branch interactive learning module for local and global frequency domain feature enhancement. The above modules can be cascaded in multiple stages to form a deep network. Finally, the image reconstruction module outputs the result.

[0016] Optionally, the detailed steps in step 3) include:

[0017] 3.1) Window Self-Attention Mechanism: The input features are mapped along the channel dimension and divided into multiple heads, and each head is divided into multiple windows along the spatial dimension; after rearranging the features within the window, the attention value is calculated using the following formula:

[0018]

[0019] In the above formula, , , These represent the query matrix, key matrix, and value matrix calculated within the window, respectively. Indicates the scaling factor; Indicates the activation function; This represents the matrix transpose operation. The resulting attention values ​​are rearranged and merged, and then used to obtain intermediate features through convolution mapping.

[0020] 3.2) Feedforward Neural Network: It uses multi-layer convolution and activation functions to introduce non-linear mapping, which enhances the model's expressive power and reduces overfitting.

[0021] 3.3) Window Interaction Submodule: This module utilizes large-scale sliding convolution for window interaction. The calculation formula is as follows:

[0022]

[0023]

[0024] In the above formula, This indicates the input characteristics of the submodule; Indicates the kernel size as Depth-separable convolution operations; These represent the weight matrices of the corresponding convolutional layers; This represents the calculated attention map;

[0025] This represents element-wise multiplication. This indicates the output characteristics after window interaction.

[0026] 3.4) Spatial Alignment Submodule: Constructs a bi-branch structure to correct degradation; the calculation formula is as follows:

[0027]

[0028]

[0029]

[0030] In the above formula, and These represent different nonlinear activation functions; Represents the weight matrix of a linear layer or convolution operation; Indicates the kernel size as Depth-separable convolution operations; This represents the spatial texture details learned in the first branch; This represents the spectral weight matrix learned by the second branch; This indicates that the two branches are multiplied element-wise. The final spatial alignment feature after aggregation.

[0031] Optionally, the detailed steps in step 4) include:

[0032] 4.1) Local frequency domain fusion: The hyperspectral and multispectral features are divided into multiple blocks in the spatial dimension, and a discrete Fourier transform is performed on each block. The phase spectrum and amplitude spectrum are obtained and fused using the following formula:

[0033]

[0034]

[0035] In the above formula, and These represent the amplitude spectra of the feature blocks corresponding to the hyperspectral and multispectral branches, respectively. and These represent the corresponding phase spectra; This indicates a stacking and splicing operation along the channel dimension; Indicates the convolution fusion operation; and These represent the amplitude spectrum and phase spectrum after fusion, respectively. The fused spectrum is then analyzed using the inverse discrete Fourier transform. The local frequency domain fusion features are obtained by converting back to the spatial domain and splicing them together in the original spatial order.

[0036] 4.2) Global frequency domain fusion: The complete hyperspectral and multispectral features are directly subjected to global discrete Fourier transform. In the frequency domain, the global phase spectrum and amplitude spectrum are superimposed and convolved and fused using the same logic as the local frequency domain. The global frequency domain features are obtained through inverse discrete Fourier transform.

[0037] 4.3) Difference Information Guidance: A weighted graph is generated using difference features for cross-fusion enhancement. The calculation formula is as follows:

[0038]

[0039]

[0040] In the above formula, and These represent local frequency domain fusion features and global frequency domain fusion features, respectively. This represents a weight map generation block consisting of convolutional layers and activation functions; Represents element-wise multiplication; and These represent the local frequency domain features and global frequency domain features after enhancement guided by difference information, respectively.

[0041] 4.4) High-frequency information enhancement: Extract high-frequency information from the multispectral image. The calculation formula is as follows:

[0042]

[0043] In the above formula, This represents the features obtained after shallow feature extraction from a multispectral image. and These represent the Discrete Fourier Transform and the Inverse Discrete Fourier Transform, respectively. This indicates the operation of moving the zero-frequency component to the center of the spectrum; This indicates a high-pass filter operation that sets the center position of the spectrum to zero; This indicates the operation of moving the zero-frequency component back to the top left corner; This represents the high-frequency spatial features of the extracted multispectral image. Finally, It is superimposed on the frequency domain fusion features to enhance information.

[0044] Optionally, the detailed steps in step 5) include:

[0045] 5.1) The image reconstruction module receives the features after preliminary fusion of the frequency domain dual branches. The image is then weighted, reconstructed, and compressed. The reconstructed residual features are then concatenated with the residuals of the original low-resolution hyperspectral image after spatial upsampling to generate the final high-resolution hyperspectral image. The calculation formula is as follows:

[0046]

[0047] In the above formula, This represents the characteristics after the initial fusion of the frequency domain bi-branch combination; This represents an image reconstruction module that includes convolution, average pooling, and sigmoid activation. This indicates a spatial upsampling operation; The input is the original low-resolution hyperspectral image; This is the final output high-resolution hyperspectral image.

[0048] 5.2) During the training phase, the model uses the L1 loss function to calculate the difference between the fused image and the reference image to update the parameters in the network. The calculation formula is as follows:

[0049]

[0050] In the above formula, These represent the image's height, width, and number of channels, respectively. These represent the row, column, and channel indices that determine the spatial location of a pixel, respectively. and These represent the fused image and the reference image, respectively.

[0051] Furthermore, the present invention also provides a hyperspectral fusion imaging system based on a hierarchical frequency integration network, including a computer device, the computer device including at least a microprocessor and a memory interconnected thereto, the microprocessor being programmed or configured to perform the steps of the hyperspectral fusion imaging method based on the hierarchical frequency integration network, or the memory storing a computer program programmed or configured to perform the image fusion method.

[0052] Furthermore, the present invention provides a computer-readable storage medium storing a computer program programmed or configured to perform the hyperspectral fusion imaging method based on a hierarchical frequency integration network.

[0053] Compared with existing technologies, this invention has the following advantages: Considering that in practical remote sensing observation and computational imaging applications, single-modality sensors struggle to simultaneously achieve high spatial resolution and high spectral resolution, and that existing fusion methods are mostly limited to spatial domain learning while neglecting frequency domain information, they are prone to spectral distortion, spatial artifacts, and loss of high-frequency texture details during feature extraction and mapping, this invention innovatively transforms images into a frequency space for hierarchical fusion by constructing a hierarchical frequency integration network. This invention utilizes the differences and complementarities between different modalities in the local and global frequency spaces, combining high-frequency information supplementation with difference information guidance, effectively overcoming the texture blurring and structural distortion phenomena caused by traditional fusion algorithms. This invention integrates spatial spectral and frequency domain features, introducing a spectral-based window self-attention mechanism and a large-scale sliding convolution interaction mechanism. This allows for the establishment of global and local dependencies between features in both spatial and spectral dimensions, and the calibration of degradation phenomena. Through a multi-branch structure, different modal features are fused in the frequency domain, fully utilizing the correlation between multispectral and hyperspectral images in phase and amplitude spectra. At the output end, a reconstruction module weights and reconstructs residual features, improving the model's learning ability for nonlinear mappings of complex terrain scenes. Furthermore, this invention is accompanied by a Python Qt-based visualization software system, significantly enhancing the convenience of quantitative analysis and qualitative assessment of image quality. This method is applicable to hyperspectral super-resolution reconstruction in various complex terrain scenes, exhibiting extremely high spectral fidelity and spatial structure consistency, providing possibilities for remote sensing Earth observation and refined interpretation in real-world environments. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram illustrating the working principle of the hierarchical frequency integration module in an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of the spatial spectral information extraction module in an embodiment of the present invention.

[0057] Figure 4 This is a schematic diagram of the frequency domain dual-branch interactive learning module structure in an embodiment of the present invention.

[0058] Figure 5 This is a schematic diagram of the image reconstruction module structure in an embodiment of the present invention.

[0059] Figure 6 This is a schematic diagram of the framework design of the hyperspectral fusion imaging system in an embodiment of the present invention. Detailed Implementation

[0060] Figure 1 and Figure 2 As shown, a hyperspectral fusion imaging method based on a hierarchical frequency integration network includes:

[0061] 1) Input low-resolution hyperspectral images and high-resolution multispectral images, perform spatial and spectral dimension alignment preprocessing, and extract shallow features to obtain initial hyperspectral features and initial multispectral features respectively.

[0062] 2) Construct a hierarchical frequency integration network, which includes a spatial spectral information extraction module, a frequency domain dual-branch interactive learning module, and an image reconstruction module.

[0063] 3) Input the initial hyperspectral features and initial multispectral features into the spatial spectral information extraction module, and extract the spatial spectral features through a spectral-based window self-attention mechanism.

[0064] 4) The extracted spatial spectral features are input into the frequency domain dual-branch interactive learning module along with the initial features. Local and global information fusion is performed in the frequency domain, and the results are combined with difference information guidance and high-frequency information enhancement to output local frequency domain fusion features and global frequency domain fusion features.

[0065] 5) The image reconstruction module combines the frequency domain fusion features and the reconstructed residual features. The reconstruction results are then joined with the upsampled low-resolution hyperspectral image to generate the final high-resolution hyperspectral image.

[0066] See Figure 2 As can be seen, step 1) is the process of data preprocessing and shallow feature extraction; step 2) is the construction of the overall fusion network architecture; step 3) is the extraction of spatial spectral information, capturing spatial spectral features and suppressing feature degradation through mechanisms such as window self-attention; step 4) is the frequency domain interactive learning, fusing local and global frequency spaces and introducing high-frequency and difference information enhancement; step 5) is the image reconstruction and output, generating the final high-resolution hyperspectral image.

[0067] like Figure 2 As shown, spatial and spectral dimension alignment preprocessing is performed on the input low-resolution hyperspectral image LR-HSI and high-resolution multispectral image HR-MSI, and shallow features are extracted to obtain initial hyperspectral features and initial multispectral features, respectively; in this embodiment, the detailed steps in step 1) include:

[0068] 1.1) Spatially upsample the low-resolution hyperspectral image using a bicubic interpolation algorithm to make its spatial dimensions consistent with those of the high-resolution multispectral image.

[0069] 1.2) Use convolution operations to perform shallow feature extraction on the upsampled hyperspectral image and high-resolution multispectral image respectively to obtain the initial hyperspectral features and initial multispectral features.

[0070] like Figure 2 As shown, step 2) constructs a hierarchical frequency integration network, which includes a spatial spectral information extraction module, a frequency domain dual-branch interactive learning module, and an image reconstruction module; in this embodiment, the detailed steps in step 2) include:

[0071] 2.1) The hierarchical frequency integration network is configured as follows: the initial hyperspectral features and initial multispectral features are input into the spatial spectral information extraction module, and the obtained spatial spectral features are input again into the frequency domain dual-branch interactive learning module.

[0072] 2.2) The spatial spectral information extraction module and the frequency domain dual-branch interactive learning module can be cascaded N times to extract deep features, and finally the image reconstruction module outputs the results.

[0073] like Figure 3 As shown, step 3) inputs the initial hyperspectral features and initial multispectral features into the spatial spectral information extraction module, and extracts spatial spectral features through a spectral-based window self-attention mechanism; in this embodiment, the detailed steps in step 3) include:

[0074] 3.1) Window Self-Attention Mechanism: To capture global spectral similarity and preserve spatial continuity between neighboring feature values, the input features are mapped along the channel dimension and divided into multiple heads. Each head is further divided into multiple windows along the spatial dimension. The query is computed within each window. ,key Sum The expression for calculating its attention value is:

[0075]

[0076] In the above formula, This is the scale scaling factor. For activation function, This is a transpose operation.

[0077] 3.2) Window Interaction and Spatial Alignment: To compensate for the lack of interaction in single-window attention and correct spatial artifacts, a convolution kernel size of... For depthwise separable convolutions used for window interaction, the mathematical expression is:

[0078]

[0079]

[0080] In the above formula, For input features, This is the weight matrix. This is an element-wise multiplication operation. Subsequently, features are aggregated through a spatial alignment submodule, and the first branch learns spatial texture details. The second branch learns the spectral weight matrix. and output fused features .

[0081] like Figure 4 As shown, step 4) inputs the extracted spatial spectral features and the initial multispectral features into the frequency domain dual-branch interactive learning module, performs local and global information fusion in the frequency domain, and combines difference information guidance and high-frequency information enhancement to output local frequency domain fusion features and global frequency domain fusion features; in this embodiment, the detailed steps in step 4) include:

[0082] 4.1) Frequency Domain and Spatial Fusion: For local frequency domain branches, the features are divided into multiple blocks, and a Discrete Fourier Transform is used.

[0083] Transforming to frequency space, the amplitude spectra of corresponding blocks in the hyperspectral and multispectral branches are superimposed and convolved for fusion. The mathematical expression is as follows:

[0084]

[0085] After the corresponding phase spectra are also fused, the inverse discrete Fourier transform is applied. Transform back to the spatial domain and concatenate sequentially. For the global frequency domain branch, directly perform a discrete Fourier transform on the complete feature map and fuse it in the frequency domain.

[0086] 4.2) High-frequency information enhancement and difference guidance: Extracting high-frequency information from multispectral images The mathematical expression for superposition enhancement is:

[0087]

[0088] In the above formula, It has multispectral characteristics. and For Fourier transform and inverse transform, This is a zero-frequency component centering operation. This is a high-pass filtering operation. Finally, the difference between the local and global fused features is calculated to generate a weight map, and the difference information is supplemented into the corresponding features through cross-guidance.

[0089] like Figure 5 As shown, step 5) reconstructs the frequency domain fusion features using the image reconstruction module to obtain residual features, and then performs residual concatenation between the reconstructed result and the upsampled low-resolution hyperspectral image to generate the final high-resolution hyperspectral image; in this embodiment, the detailed steps in step 5) include:

[0090] 5.1) Preliminary Feature Fusion and Compression: The frequency domain dual-branch output features are initially fused through stacking and convolution, and then input into the image reconstruction module containing convolutional layers, average pooling layers, and a sigmoid activation function. Feature-weighted reconstruction is then performed.

[0091] 5.2) Residual Connection and Output: The reconstructed result is combined with the upsampled low-resolution hyperspectral image. The mathematical expression for addition is:

[0092]

[0093] In the above formula, The frequency domain characteristics after initial merging. This is a spatial upsampling operation. This is the final fusion output result.

[0094] To verify the effectiveness of this invention, experiments were conducted on the CAVE and Harvard public datasets. The CAVE dataset contains high-resolution multispectral images covering a variety of natural and synthetic materials, encompassing 31 spectral bands within the 400-700 nm wavelength range, with a spatial resolution of 512 × 512 pixels for each scene. The Harvard dataset contains hyperspectral images under indoor and outdoor daylight conditions, as well as under artificial or mixed lighting conditions, covering 31 spectral bands within the 420-720 nm wavelength range, with a spatial resolution of 1392 × 1040 pixels for each scene. Three quantitative evaluation metrics were used to assess the fusion performance: Peak Signal-to-Noise Ratio (PSNR), Spectral Angle Mapper (SAM), and Dimensionless Relative Global Error (ERGAS).As a comparison with the method in this embodiment: The comparison method is SSRNet (see Zhang, X., Huang, W., Wang, Q., Li, X.: Ssr-net: Spatial–spectral reconstruction network for hyperspectral and multispectral image fusion. IEEE Transactions on Geoscience and Remote Sensing 59(7), 5953–5965 (2021)); the comparison method is PSRT (see Deng, S.-Q., Deng, L.-J., Wu, X., Ran, R., Hong, D., Vivone, G.: Psrt: Pyramid shuffle-and-reshuffle transformer for multispectral and hyperspectral image fusion. IEEE Transactions on Geoscience and Remote Sensing 61, 1–15 (2023)); the comparison method is DSPNet (see Sun, Y., Xu, H., Ma, Y., Wu, M, Mei, X., Huang, J., Ma, J.: Dual spatial-spectral pyramid network with transformer for hyperspectral image fusion. IEEE Transactions on Geoscience and Remote Sensing 61, 1-16 (2023). Tables 1 and 2 show the quantitative comparison results between the method of this invention and three advanced algorithms.

[0095] Table 1 compares the experimental results of the method in this embodiment with those of three advanced algorithms on the CAVE dataset.

[0096]

[0097] Table 2 compares the experimental results of the method in this embodiment with those of three advanced algorithms on the Harvard dataset.

[0098]

[0099] As shown in Tables 1 and 2, the method of the present invention outperforms the comparative method in all three evaluation indicators: PSNR, SAM, and ERGAS, proving that it can effectively recover spatial details while maintaining spectral fidelity.

[0100] In summary, in practical hyperspectral remote sensing and computational imaging applications, due to the physical characteristics of imaging spectrometers, it is difficult to simultaneously achieve high spatial and hyperspectral resolution with a single mode. Furthermore, most existing methods are limited to spatial domain learning while neglecting frequency domain information, easily leading to spectral distortion, spatial artifacts, and loss of high-frequency texture details during feature extraction and mapping. Therefore, the hyperspectral fusion imaging method in this embodiment constructs a hierarchical frequency integration network to fuse information from multispectral and hyperspectral images in local and global frequency spaces. This method utilizes the complementarity of frequency domain information from different modes, combined with difference information guidance and high-frequency information enhancement, effectively overcoming the texture blurring and structural distortion phenomena caused by traditional fusion algorithms, providing an effective and high-quality solution for hyperspectral super-resolution reconstruction in complex terrain scenes. Simultaneously, this embodiment provides a computer-readable storage medium storing a computer program or instructions. When the computer program or instructions are executed by a processor, the hyperspectral fusion imaging method based on topological sensing and differential homeomorphism mechanisms, as described above, is implemented.

[0101] Furthermore, this embodiment also provides a hyperspectral fusion imaging visualization system based on a hierarchical frequency integration network, including:

[0102] The image input and output program unit is used to process and display image information. For hyperspectral or multispectral images selected and loaded by the user, it analyzes and displays basic information such as image size, spatial dimensions and number of channels on the system interface.

[0103] The image preprocessing unit is used to perform various image data processing. For the loaded raw image data, its data distribution is adjusted to the range of 0 to 255, and specific channels are selected for subsequent visualization display.

[0104] The image fusion program unit is used to implement fusion algorithm selection and multi-source image fusion, respond to the user's selection operation of the built-in fusion algorithm, perform fusion processing on the input low-resolution hyperspectral image and high-resolution multispectral image, and generate and output a high-resolution hyperspectral image.

[0105] The image quality assessment program unit is used to select and calculate image quality assessment indicators, respond to the user's selection of various image quality evaluation indicators, calculate specific values ​​such as peak signal-to-noise ratio (PSNR) and spectral angle mapping (SAM) of the fused image, and display them in the corresponding positions on the interface.

[0106] The image display program unit is used to select, load, display, and reset images. It loads and displays selected multispectral images, hyperspectral images, and fused high-resolution images in various specified areas of the system interface, and supports clearing and resetting the image display area.

[0107] Furthermore, this embodiment also provides a hyperspectral fusion imaging system, including a computer device, which includes at least a microprocessor and a memory interconnected with each other. The microprocessor is programmed or configured to perform the steps of the aforementioned hyperspectral fusion imaging method based on a hierarchical frequency integration network, or the memory stores a computer program programmed or configured to perform the aforementioned hyperspectral fusion imaging method.

[0108] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program programmed or configured to execute the aforementioned hyperspectral fusion imaging method based on a hierarchical frequency integration network.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] Furthermore, these computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A hyperspectral fusion imaging method based on hierarchical frequency integrated network, characterized in that, include: 1) Aligning preprocessing of spatial and spectral dimensions is performed on input low-resolution hyperspectral image and high-resolution multispectral image , and initial hyperspectral features and initial multispectral features are respectively extracted by shallow feature extraction and shallow feature extraction ; 2) constructing a hierarchical frequency integrated network, the network comprising a spatial spectrum information extraction module , a frequency domain double-branch interactive learning module and an image reconstruction module ; 3) input the initial hyperspectral features and initial multispectral features into the spatial-spectral information extraction module to extract spatial-spectral features based on a spectral-based window self-attention mechanism ;​​ 4) extract the spatial spectral features with the initial features and input the frequency domain double-branch interactive learning module, respectively perform local and global information fusion in the frequency domain, combine difference information guidance and high-frequency information enhancement, and output local frequency domain fusion features and global frequency domain fusion features ; 5) the frequency domain fusion features are reconstructed by an image reconstruction module and to obtain residual features , the reconstruction result is connected with the up-sampled low-resolution hyperspectral image to generate the final high-resolution hyperspectral image .

2. The hyperspectral fusion imaging method according to claim 1, characterized in that, The preprocessing in step 1) comprises: performing bicubic interpolation algorithm upsampling to obtain and extracting hyperspectral and multispectral shallow features respectively by using convolution layers. and .

3. The hyperspectral fusion imaging method according to claim 1, characterized in that, Step 3) includes the following detailed steps: 3.1) To capture global spectral similarity and preserve the spatial continuity between neighboring feature values, a window self-attention mechanism is used to process the input features. Map along the channel dimension and divide into multiple heads. Each head is divided into multiple windows along the spatial dimension, and the query is computed within each window. ,key Sum Through formula Calculate the attention value, where The scale factor is used to rearrange and merge the final attention values. Convolutional mapping yields intermediate features ; 3.2) Targeting intermediate features The result is obtained by nonlinear mapping through a feedforward neural network. ; 3.3) Under this mapping, the window interaction submodule is used to enable interaction between neighboring windows and improve the ability to learn spatial information. First, a convolution kernel size of... The depthwise separable convolution computes the attention map, and then multiplies it with the transformed input features through element-wise multiplication to capture global spatial dependencies. 3.4) Construct a dual-branch structure. The first branch learns spatial texture details through convolution and the GELU activation function, while the second branch learns the spectral weight matrix through depthwise separable convolution and the Sigmoid activation function. The results of the two branches are aggregated by element-wise multiplication to correct spectral distortion and spatial artifacts.

4. The hyperspectral fusion imaging method according to claim 1, characterized in that, Step 4) includes the following detailed steps: 4.1) Local frequency domain fusion: The hyperspectral and multispectral features are divided into multiple blocks in the spatial dimension. The discrete Fourier transform is performed on each block to obtain the phase spectrum and amplitude spectrum. The corresponding phase spectrum and amplitude spectrum are superimposed along the channel and then fused by convolution. Finally, the inverse discrete Fourier transform is used to convert the fused spectrum back to the spatial domain and splice them to obtain the local frequency domain fused features. 4.2) Global frequency domain fusion: The complete hyperspectral and multispectral features are directly subjected to discrete Fourier transform, and the global phase spectrum and amplitude spectrum are superimposed and convolved in the frequency domain. The global frequency domain fused features are obtained through inverse discrete Fourier transform.

5. The hyperspectral fusion imaging method according to claim 1, characterized in that, In step 4), the specific steps of combining difference information guidance and high-frequency information enhancement include: Difference information guidance: Calculate local frequency domain fusion features separately Features fused with global frequency domain The difference is used to generate a weight map, and through cross-guidance, global difference information is supplemented to local features, and local difference information is supplemented to global features. High-frequency information enhancement: Perform a Fourier transform on the initial multispectral features, shift the zero-frequency component to the center of the spectrum and set it to zero to filter out low frequencies, then shift the zero-frequency component back and extract high-frequency information through an inverse Fourier transform. Finally, the extracted high-frequency information is superimposed onto the local and global frequency domain fusion features respectively.

6. The hyperspectral fusion imaging method according to claim 1, characterized in that, In step 5), the image reconstruction module performs weighted reconstruction of features through a channel attention mechanism that includes convolutional layers, average pooling layers, and a sigmoid activation function; during the model training phase, the L1 loss function is used to calculate the difference between the fused high-resolution hyperspectral image and the reference image in order to update the network parameters.

7. A hyperspectral fusion imaging system based on the method of any one of claims 1 to 6, characterized in that, This system is developed based on the Python Qt framework. Its architecture includes a user interface layer, a business logic layer, and a data access layer. The system's functional modules include: Image Input and Preprocessing Module: Loads low-resolution hyperspectral images and high-resolution multispectral images in .mat or .png format, analyzes the spatial dimensions and number of channels of the image, and adjusts the image data distribution to the range of 0-255 for display; Fusion Algorithm Integration Module: Internally integrates the hierarchical frequency fusion network model, responds to the user's "fusion image" operation through multi-threaded processing, performs image fusion, and visualizes the fused high-resolution hyperspectral image; Image Quality Evaluation Module: Provides a list of evaluation indicators, responds to user operations to calculate the peak signal-to-noise ratio, spectral angle mapping, root mean square error, relative global dimension synthesis error, structural similarity, and correlation coefficient of the fused image, and displays the calculated values ​​on the interface.

8. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by the processor, the hyperspectral fusion imaging method as described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, the hyperspectral fusion imaging method as described in any one of claims 1 to 6 is implemented.

Images