High-signal-noise and high-spatial-resolution hyperspectral fusion imaging method and high-signal-noise and high-spatial-resolution hyperspectral fusion imaging system

Abstract

The invention discloses a high-signal-noise and high-spatial-resolution hyperspectral fusion imaging method and system, and the method comprises the steps: carrying out the up-sampling of an input low-resolution hyperspectral image through a multistage heterogeneous compensation sampling module, so as to obtain multi-scale spectrum sampling information; performing channel adjustment on the input multispectral image by using a convolutional layer to obtain initial spatial information; and the multi-scale spectrum sampling information and the initial space information are sent into the combined residual fusion network for feature fusion to obtain a fused high-resolution hyperspectral image, and the heterogeneous compensation sampling module comprises a structure information selection branch, a basic information keeping branch, a connection module and a convolution layer. The method aims at solving the problems that an existing hyperspectral fusion imaging method is serious in distortion and low in signal-to-noise ratio, the three-dimensional characteristic of a hyperspectral image is effectively captured, and the imaging signal-to-noise ratio is improved.

Description

Technical Field

[0001] The present invention relates to a hyperspectral fusion imaging technology in the field of image processing, and in particular to a hyperspectral fusion imaging method and system with high signal-to-noise and high spatial resolution. Background Art

[0002] Hyperspectral images are high-dimensional images consisting of dozens to hundreds of continuous spectral bands, effectively capturing the spectral characteristics of objects. Therefore, hyperspectral imaging has been widely used in a wide range of fields, including agricultural surveys, environmental monitoring, mineral exploration, and medical diagnosis. However, due to the limitations of the spatial-bandwidth product imaging principle, existing hyperspectral imaging methods cannot simultaneously achieve high-resolution imaging in both spatial and spectral dimensions. This results in limited spatial resolution of hyperspectral images, severely restricting their application value. Hyperspectral fusion imaging offers an effective solution to this problem. By fusing multispectral and low-resolution hyperspectral images of the same scene, hyperspectral fusion imaging can significantly improve the spatial resolution of hyperspectral imaging and produce high-resolution hyperspectral images. However, the significant difference in the number of spatial pixels and spectral bands between multispectral and low-resolution hyperspectral images makes the matching and fusion of spatial and spectral information extremely difficult. Most existing fusion methods directly extract spatial and spectral information from multispectral and low-resolution hyperspectral images for fusion, ignoring the differences in modality and data dimensionality between the two. This results in significant spatial and spectral distortion in the fused hyperspectral images, resulting in low signal-to-noise ratios, which severely restricts the application value of hyperspectral fusion imaging technology. Summary of the Invention

[0003] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems of the prior art, a hyperspectral fusion imaging method and system with high signal-to-noise and high spatial resolution are provided. The present invention aims to solve the problems of severe distortion and low signal-to-noise ratio of the existing hyperspectral fusion imaging methods, effectively capture the three-dimensional characteristics of hyperspectral images, and improve the imaging signal-to-noise ratio.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A high-signal-to-noise and high-spatial-resolution hyperspectral fusion imaging method comprises the following steps: using a multi-level heterogeneous compensation sampling module to input a low-resolution hyperspectral image Upsampling to obtain multi-scale spectral sampling information ,in is the upsampling multiple Spectral sampling information of corresponding scale; use convolution layer to input multispectral image Perform channel adjustment to obtain initial spatial information ; Multi-scale spectral sampling information With initial spatial information The feature fusion is carried out in the combined residual fusion network to obtain the fused high-resolution hyperspectral image. The heterogeneous compensation sampling module includes a structure information selection branch , basic information maintenance branch , connection module and convolution layer, the feature map of the input heterogeneous compensation sampling module is respectively passed through the structure information selection branch Select structural information and maintain basic information branches Maintain basic information and select branches based on the structural information , basic information maintenance branch The feature maps output by the two are connected through the connection module and sent to the convolution layer for processing to obtain the feature map output by the heterogeneous compensation sampling module.

[0005] Optionally, the structural information selection branch Structural information selection includes: S101, input the feature map of the heterogeneous compensation sampling module through the convolution layer The channel dimension changes from C Expand to ,in H and W Indicates the size of the space, C represents the number of spectral channels, The upsampling multiple of the heterogeneous compensation sampling module Relevant hyperparameters; S102: Flatten all pixels in the feature map obtained by expanding the channel dimension into a single-channel spatial matrix ; S103, the size of the convolution kernel is The convolutional layer is a spatial matrix of a single channel Perform feature extraction to obtain feature maps ; S104, feature map Perform spatial downsampling to keep the number of spatial pixels consistent with the input feature map I to obtain structural information selection branch Output feature map.

[0006] Optionally, the basic information holds branches Maintaining basic information includes: inputting the feature map of the heterogeneous compensation sampling module through the convolution layer Extract features, then upsample the extracted features through bilinear interpolation to obtain the basic information keeping branch The output feature map, where H and W Indicates the size of the space, C Indicates the number of spectral channels.

[0007] Optionally, the multi-level heterogeneous compensation sampling module is used to upsample the input low-resolution hyperspectral image X to obtain multi-scale spectral sampling information When , it includes using at least four levels of heterogeneous compensation sampling modules to upsample the input low-resolution hyperspectral image X, and the obtained multi-scale spectral sampling information The upsampling multiple of the spectral sampling information at each scale are 1, 2, 4 and 8 respectively, and the spectral sampling information of the corresponding scales is 、 、 and .

[0008] Optionally, the multi-scale spectral sampling information With initial spatial information The feature fusion is carried out in the combined residual fusion network to obtain the fused high-resolution hyperspectral image. When , the combined residual fusion network includes fusion branches, of which Multi-scale spectral sampling information The number of scales, all fusion branches use multi-directional hybrid modeling modules to optimise the input feature maps Extract the fusion features of three directions from CW-H, CH-W and HW-C respectively and perform the fusion on the input feature map. Extract depth features, and splice the depth features and the fusion features of the three directions into splicing features in a way that maintains the original position relationship and three-dimensional spatial spectrum characteristics , and from the splicing features Extract features to obtain the output features of the multi-directional hybrid modeling module ,in H and W Indicates the size of the space, C Indicates the number of spectral channels.

[0009] Optionally, extracting fusion features in three directions from the CW-H, CH-W, and HW-C directions includes: S201, according to the following formula HW-C , CH-W and CW-H Feature maps of the input in three directions Extract modeling information in three directions: CW-H, CH-W, and HW-C: ; ; ; in, and Respectively represent the CW-H direction rotation and reverse rotation operation, and Respectively represent the rotation and reverse rotation operations in the CH-W direction, is the convolution operation, 、 and The modeling information extracted from the three directions of CW-H, CH-W and HW-C respectively; the feature map of the input Ordinary convolution is used to reduce the number of channels from Expand to 3 C , and then extract the feature sets of three directions through group convolution with a group number of 3 , for the feature set in three directions According to the following formula, the features of the three directions of CW-H, CH-W and HW-C are separated along the channel dimension: ; ; ; in, 、 and They are the characteristics in the three directions of CW-H, CH-W and HW-C. and The feature sets in three directions are Located in the middle 、 and Sub-blocks within the three channel range, is the input feature map The number of channels; S202: The modeling information extracted in the three directions of CW-H, CH-W, and HW-C and the separated features in the three directions of CW-H, CH-W, and HW-C are used to calculate the fusion features of the three directions according to the following formula: ; ; ; in, 、 and They are the fusion features of the three directions of CW-H, CH-W and HW-C, Represents element-by-element multiplication; the feature map of the input The function expression for extracting deep features is: ; in, is the deep feature, is a convolutional layer; the concatenated features Extract features to obtain the output features of the multi-directional hybrid modeling module The function expression is: .

[0010] Optionally, the Among the fusion branches, the input of the first N-1 fusion branches is multi-scale spectral sampling information Spectral sampling information of the corresponding scale , the input of the Nth fusion branch is multi-scale spectral sampling information Spectral sampling information of the corresponding scale and initial spatial information , any The fusion branch includes the splicing module, convolution layer and Cascaded multi-directional hybrid modeling module, The splicing module of the fusion branch is used to convert the spectral sampling information of the corresponding scale and initial spatial information Splicing, the splicing modules of the other fusion branches are used to combine the spectral sampling information of the corresponding scale and the output features of the first-level multi-directional hybrid modeling module in the next fusion branch ,forward Any of the fused branches Output features of the multi-directional hybrid modeling module After upsampling, it is combined with the next fusion branch Output features of the multi-directional hybrid modeling module After concatenation, it is used as the output feature of the backward output, and finally the The first fusion branch Output features of the multi-directional hybrid modeling module After a convolution layer, the output is the fused high-resolution hyperspectral image .

[0011] In addition, the present invention also provides a high-signal-to-noise and high-spatial-resolution hyperspectral fusion imaging system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the high-signal-to-noise and high-spatial-resolution hyperspectral fusion imaging method.

[0012] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method through a processor.

[0013] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method through a processor.

[0014] Compared with the prior art, the present invention can achieve the following beneficial effects: the present invention includes a multi-level heterogeneous compensation sampling module for inputting low-resolution hyperspectral images. Upsampling to obtain multi-scale spectral sampling information , using convolutional layers to transform the input multispectral image Perform channel adjustment to obtain initial spatial information ; Multi-scale spectral sampling information With initial spatial information The feature fusion is carried out in the combined residual fusion network to obtain the fused high-resolution hyperspectral image. , the heterogeneous compensation sampling module includes a structural information selection branch , basic information maintenance branch , connection modules and convolutional layers. The present invention can fully model the three-dimensional spatial-spectral characteristics of hyperspectral images to effectively capture the three-dimensional characteristics of hyperspectral images, utilize the spatial structural information between different spectral bands for complementary fusion, significantly reduce the spatial and spectral distortion in the fusion results, and solve the problems of severe distortion and low signal-to-noise ratio in existing hyperspectral fusion imaging methods, effectively capture the three-dimensional characteristics of hyperspectral images, and improve the imaging signal-to-noise ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the overall network structure of the method according to the embodiment of the present invention.

[0016] Figure 2 Schematic diagram of the network structure of the heterogeneous compensation sampling module in an embodiment of the present invention.

[0017] Figure 3 Schematic diagram of the network structure of the combined residual fusion network in an embodiment of the present invention.

[0018] Figure 4 Schematic diagram of the three directions of CW-H, CH-W and HW-C in an embodiment of the present invention.

[0019] Figure 5 To obtain the output features of the multi-directional hybrid modeling module in the embodiment of the present invention Schematic diagram. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0021] like Figure 1 and Figure 2 As shown, the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method of this embodiment includes the following steps: using a multi-level heterogeneous compensation sampling module to input a low-resolution hyperspectral image Upsampling to obtain multi-scale spectral sampling information ,in is the upsampling multiple Spectral sampling information of corresponding scale; use convolution layer to input multispectral image Perform channel adjustment to obtain initial spatial information ; Multi-scale spectral sampling information With initial spatial information The feature fusion is carried out in the combined residual fusion network to obtain the fused high-resolution hyperspectral image. The heterogeneous compensation sampling module includes a structure information selection branch , basic information maintenance branch , connection module and convolution layer, the feature map of the input heterogeneous compensation sampling module is respectively passed through the structure information selection branch Select structural information and maintain basic information branches Maintain basic information and select branches based on the structural information , basic information maintenance branch The feature maps output by the two are connected through the connection module and then sent to the convolution layer for processing to obtain the feature maps output by the heterogeneous compensation sampling module. This embodiment innovatively designs a heterogeneous compensation sampling module with heterogeneous compensation and multi-directional hybrid modeling. The heterogeneous compensation sampling module can make full use of the structural information of different spectral bands of the hyperspectral image to perform spatial compensation on itself, thereby reducing the spatial structure distortion in the fusion result. The multi-directional hybrid modeling module can effectively capture the three-dimensional characteristics of the hyperspectral image and improve the imaging signal-to-noise ratio. The multi-level heterogeneous compensation sampling module is used to input the low-resolution hyperspectral image. Upsampling to obtain multi-scale spectral sampling information When , the upsampling of the first-level heterogeneous compensation sampling module can be expressed as: ; The upsampling of the remaining heterogeneous compensation sampling modules can be expressed as: ; in, Indicates the upsampling multiple The corresponding heterogeneous compensation sampling module, superscript Indicates the upsampling multiple. As an optional implementation, in this embodiment The values ​​are set to 1, 2, 4, and 8 respectively; It is the output feature of the previous level heterogeneous compensation sampling module. Upsampling multiple Corresponding heterogeneous compensation sampling module It mainly consists of a structural information selection branch and a basic information preservation branch. Assuming the input feature is ,in and Indicates the size of the space, Indicates the number of spectral channels. The processing can be expressed as: ) represents the convolution operation, represents the splicing operation of the channel dimension, Indicates the structural information selection branch, Indicates that the basic information is maintained in the branch.

[0022] In this embodiment, the structure information selection branch Structural information selection includes: S101, input the feature map of the heterogeneous compensation sampling module through the convolution layer The channel dimension changes from C Expand to ,in H and W Indicates the size of the space, C represents the number of spectral channels, The upsampling multiple of the heterogeneous compensation sampling module Relevant hyperparameters; S102: Flatten all pixels in the feature map obtained by expanding the channel dimension into a single-channel spatial matrix ; In order to capture the spatial structure information between different spectral channels in the input features, the convolution layer is used to transform The channel dimension is expanded (from C Expand to , is a hyperparameter, and the upsampling multiple Related), and all the expanded pixels are tiled into a single-channel spatial matrix, which can be expressed as: ; in represents pixel tiling operation, is the obtained spatial matrix; S103, the size of the convolution kernel is The convolutional layer is a spatial matrix of a single channel Perform feature extraction to obtain feature maps , which can be expressed as: ; Among them, the size of the convolution kernel is , is the extracted feature; S104, feature map Perform spatial downsampling to make its spatial pixel count equal to the input feature map Keep consistent to get structural information to choose branches The output feature map. The spatial downsampling of the input feature map Keep consistent and get the structural selection branch This branch can fully capture the spatial correlation between different bands of the hyperspectral image while improving the size of the hyperspectral space.

[0023] In this embodiment, the basic information maintains the branch Maintaining basic information includes: inputting the feature map I of the heterogeneous compensation sampling module through the convolution layer Extract features, then upsample the extracted features through bilinear interpolation to obtain the basic information keeping branch The output feature map can be expressed as: ; in, Keep the output of the branch for basic information, Represents bilinear interpolation upsampling.

[0024] In this embodiment, a multi-level heterogeneous compensation sampling module is used to upsample the input low-resolution hyperspectral image X to obtain multi-scale spectral sampling information When , it includes using at least four levels of heterogeneous compensation sampling modules to upsample the input low-resolution hyperspectral image X, and the obtained multi-scale spectral sampling information The upsampling multiple of the spectral sampling information at each scale are 1, 2, 4 and 8 respectively, and the spectral sampling information of the corresponding scales is 、 、 and .

[0025] In this embodiment, the multi-scale spectral sampling information With initial spatial information The feature fusion is carried out in the combined residual fusion network to obtain the fused high-resolution hyperspectral image. When, such as Figure 3 As shown, the merged residual fusion network includes fusion branches, of which Multi-scale spectral sampling information The number of scales, all fusion branches use multi-directional hybrid modeling modules to optimise the input feature maps From three directions (such as CW-H, CH-W and HW-C) Figure 4 As shown) extract the fusion features of the three directions and input the feature map Extract the depth features, and splice the depth features and the fusion features of the three directions in a way that maintains the original position relationship and three-dimensional spatial spectrum characteristics to obtain the splicing features , and from the splicing features Extract features to obtain the output features of the multi-directional hybrid modeling module ,like Figure 5 As shown, H and W Indicates the size of the space, C Indicates the number of spectral channels.

[0026] In this embodiment, the fusion features of the three directions extracted from the CW-H, CH-W and HW-C directions include: S201, assuming the input feature is , in order to capture the three-dimensional characteristics of hyperspectral images, HW-C , CH-W and CW-H Three directions Analyze, where CH-W and CW-H The analysis of the direction involves the image rotation and reverse rotation operations. Specifically, according to the following formula HW-C , CH-W and CW-H Feature maps of the input in three directions Extract modeling information in three directions: CW-H, CH-W, and HW-C: ; ; ; in, and Respectively represent the CW-H direction rotation and reverse rotation operation, and Respectively represent the rotation and reverse rotation operations in the CH-W direction, is the convolution operation, 、 and The modeling information extracted from the three directions of CW-H, CH-W and HW-C respectively; the feature map of the input Ordinary convolution is used to reduce the number of channels from Expand to 3 C , and then extract the feature sets of three directions through group convolution with a group number of 3 , for the feature set in three directions According to the following formula, the features of the three directions of CW-H, CH-W and HW-C are separated along the channel dimension: ; ; ; in, 、 and They are the characteristics in the three directions of CW-H, CH-W and HW-C. and The feature sets in three directions are Located in the middle 、 and Sub-blocks within the three channel range, is the input feature map The number of channels; S202: The modeling information extracted in the three directions of CW-H, CH-W, and HW-C and the separated features in the three directions of CW-H, CH-W, and HW-C are used to calculate the fusion features of the three directions according to the following formula: ; ; ; in, 、 and They are the fusion features of the three directions of CW-H, CH-W and HW-C, Represents element-by-element multiplication; the feature map of the input The function expression for extracting deep features is: ; in, is the deep feature, is a convolutional layer; the concatenated features Extract features to obtain the output features of the multi-directional hybrid modeling module The function expression is: .

[0027] The splicing feature is obtained by splicing the fusion features and deep features in three directions , and then from the splicing features Extract features to obtain the output features of the multi-directional hybrid modeling module , which can effectively model the three-dimensional characteristics of hyperspectral images, thereby reducing the spectral and spatial distortion in the fused image and improving the imaging signal-to-noise ratio.

[0028] The combined residual fusion network can adopt the required network structure as needed, for example, as an optional implementation, such as Figure 3 As shown, in this embodiment Among the fusion branches, the input of the first N-1 fusion branches is multi-scale spectral sampling information Spectral sampling information of the corresponding scale , the input of the Nth fusion branch is multi-scale spectral sampling information Spectral sampling information of the corresponding scale and initial spatial information , any The fusion branch includes the splicing module, convolution layer and Cascaded multi-directional hybrid modeling module, The splicing module of the fusion branch is used to convert the spectral sampling information of the corresponding scale and initial spatial information Splicing, the splicing modules of the other fusion branches are used to combine the spectral sampling information of the corresponding scale and the output features of the first-level multi-directional hybrid modeling module in the next fusion branch ,forward Any of the fused branches Output features of the multi-directional hybrid modeling module After upsampling, it is combined with the next fusion branch Output features of the multi-directional hybrid modeling module After concatenation, it is used as the output feature of the backward output, and finally the The first fusion branch Output features of the multi-directional hybrid modeling module After a convolution layer, the output is the fused high-resolution hyperspectral image See also Figure 3 , merge the residual fusion network to and As input, the hyperspectral image is reconstructed step by step at multiple scales. The combined residual fusion network in this embodiment is specifically composed of four branches of different scales. Composition, from high to low are represented as , , , .remove In addition to the branches, each branch consists of two parts: feature encoding and multi-level fusion ( The branch is the lowest-scale branch and only contains the encoding extraction part). The input of the branch is the initial spatial information and multi-scale spectral sampling information , and the other three branches The input is multi-scale spectral sampling information With downsampling Branch encoding results. For the input information, each branch first performs initial fusion through concatenation and convolution operations: ; in express The initial fusion information of the branch, express Branch encoding results, represents the downsampling operation. Subsequently, a single multi-directional hybrid modeling module is used to Perform feature encoding: ; in is the encoding result, It is a multi-directional hybrid modeling module. Subsequently, the encoding results of different branches are multi-level fused. , , The number of branch fusions is 3, 2, and 1. The single fusion process is as follows: ; in express The result of a single fusion of a branch. If there are multiple fusions in the branch, replace .right The final fusion result of the branches is convolved to obtain the fused high-resolution hyperspectral image Z. The heterogeneous compensatory sampling module designed in this embodiment fully extracts the spatial similarities between different spectral bands in the hyperspectral image, significantly reducing spatial distortion during the fusion process compared to existing upsampling methods. The multidirectional hybrid modeling module designed in this embodiment effectively models the three-dimensional spatial-spectral structure of the hyperspectral image, reducing spectral distortion. This ultimately achieves high signal-to-noise ratio and high spatial resolution hyperspectral imaging.

[0029] To validate the effectiveness of this embodiment's high signal-to-noise and high spatial resolution hyperspectral fusion imaging method, this embodiment conducted comparative and ablation experiments. In these comparative experiments, the method's performance was compared with several currently advanced hyperspectral fusion imaging methods to verify its overall superiority. In the ablation experiments, the designed heterogeneous compensatory sampling module and multi-directional hybrid modeling module were replaced or removed with existing modules to verify their effectiveness. The CAVE hyperspectral dataset was used as the experimental dataset, consisting of 32 high-resolution hyperspectral images with a size of 512×512×31. The first 20 images in this dataset were used as the training set, while the remaining images were designated as the test set. A Gaussian filter with a standard deviation of 3 and the spectral response function of a Nikon D700 camera were used for spatial and spectral downsampling, with a downsampling factor of 8. The structural similarity index (SSIM), spectral angular error (SAM), peak signal-to-noise ratio (PSNR), and correlation coefficient (CC) were used as evaluation metrics. We selected eight methods for comparison, including HySure, HSRNet, Fusformer, SIGNet, PSRT, U2Net, MIMO-SST, and LRTN. We used an NVIDIA Quadro RTX 6000 and an Intel Xeon Silver 4216 CPU as the experimental setup. Table 1 shows the results of the comparative experiments on the CAVE dataset.

[0030] Table 1: Comparative experimental results on the CAVE dataset

[0031] The experimental results in Table 1 demonstrate that the method of this embodiment achieves optimal fusion performance, outperforming existing methods on all evaluation metrics. The PSNR metric primarily reflects the signal-to-noise ratio of the fused image. Compared with existing methods, the peak signal-to-noise ratio of this embodiment's method is significantly improved, demonstrating its high signal-to-noise ratio and ability to accurately acquire high-resolution hyperspectral images.

[0032] In ablation experiments, the method of this embodiment is compared with three methods: (1) removing the structural information selection branch (N1) in the heterogeneous compensation sampling module; and (2) replacing the multi-directional mixture modeling module with ordinary convolution (N2). Table 2 shows the results of the ablation experiments.

[0033] Table 2: Ablation experiment results

[0034] The experimental results in Table 2 show that compared with the existing basic modules, the heterogeneous compensation sampling module and multi-directional hybrid modeling module designed in this embodiment can effectively improve the performance of hyperspectral fusion imaging.

[0035] In addition, this embodiment also provides a high signal-to-noise and high spatial resolution hyperspectral fusion imaging system, comprising an interconnected microprocessor and memory, wherein the microprocessor is programmed or configured to execute the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method. This embodiment also provides a computer-readable storage medium storing a computer program or instructions, wherein the computer program or instructions are programmed or configured to execute the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method via a processor. This embodiment also provides a computer program product, comprising a computer program or instructions, wherein the computer program or instructions are programmed or configured to execute the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method via a processor.

[0036] Those skilled in the art should understand that the technical solution provided by the present invention may be in the form of a method, a system, or a computer program product. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0037] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A high-signal-to-noise and high-spatial-resolution hyperspectral fusion imaging method, characterized in that: The following steps are included: using a multi-level heterogeneous compensation sampling module to input a low-resolution hyperspectral image Upsampling to obtain multi-scale spectral sampling information ,in is the upsampling multiple Spectral sampling information of corresponding scale; use convolution layer to input multispectral image Perform channel adjustment to obtain initial spatial information ; Multi-scale spectral sampling information With initial spatial information The feature fusion is carried out in the combined residual fusion network to obtain the fused high-resolution hyperspectral image. The heterogeneous compensation sampling module includes a structure information selection branch , basic information maintenance branch , connection module and convolution layer, the feature map of the input heterogeneous compensation sampling module is respectively passed through the structure information selection branch Select structural information and maintain basic information branches Maintain basic information and select branches based on the structural information , basic information maintenance branch The feature maps output by the two are connected through the connection module and sent to the convolution layer for processing to obtain the feature map output by the heterogeneous compensation sampling module.

2. The high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to claim 1, characterized in that: The structural information selection branch Structural information selection includes: S101, input the feature map of the heterogeneous compensation sampling module through the convolution layer The channel dimension changes from C Expand to ,in H and W Indicates the size of the space, C represents the number of spectral channels, The upsampling multiple of the heterogeneous compensation sampling module Relevant hyperparameters; S102: Flatten all pixels in the feature map obtained by expanding the channel dimension into a single-channel spatial matrix ; S103, the size of the convolution kernel is The convolutional layer is a spatial matrix of a single channel Perform feature extraction to obtain feature maps ; S104, feature map Perform spatial downsampling to keep the number of spatial pixels consistent with the input feature map I to obtain structural information selection branch Output feature map.

3. The high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to claim 1, characterized in that: The basic information holds the branch Maintaining basic information includes: inputting the feature map I of the heterogeneous compensation sampling module through the convolution layer Extract features, then upsample the extracted features through bilinear interpolation to obtain the basic information keeping branch The output feature map, where H and W Indicates the size of the space, C Indicates the number of spectral channels.

4. The high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to claim 1, characterized in that: The multi-level heterogeneous compensation sampling module is used to upsample the input low-resolution hyperspectral image X to obtain multi-scale spectral sampling information When , it includes using at least four levels of heterogeneous compensation sampling modules to upsample the input low-resolution hyperspectral image X, and the obtained multi-scale spectral sampling information The upsampling multiple of the spectral sampling information at each scale are 1, 2, 4 and 8 respectively, and the spectral sampling information of the corresponding scales is 、 、 and .

5. The high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to claim 1, characterized in that: The multi-scale spectral sampling information With initial spatial information The feature fusion is carried out in the combined residual fusion network to obtain the fused high-resolution hyperspectral image. When , the combined residual fusion network includes fusion branches, of which Multi-scale spectral sampling information The number of scales, all fusion branches use multi-directional hybrid modeling modules to optimise the input feature maps Extract the fusion features of three directions from CW-H, CH-W and HW-C respectively and perform the fusion on the input feature map. Extract depth features, and splice the depth features and the fusion features of the three directions into splicing features in a way that maintains the original position relationship and three-dimensional spatial spectrum characteristics , and from the splicing features Extract features to obtain the output features of the multi-directional hybrid modeling module ,in H and W Indicates the size of the space, C Indicates the number of spectral channels.

6. The high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to claim 4, characterized in that: The three-directional fusion features extracted from the CW-H, CH-W and HW-C directions include: S201, according to the following formula HW-C , CH-W and CW-H Feature maps of the input in three directions Extract modeling information in three directions: CW-H, CH-W, and HW-C: ； ； ； in, and Respectively represent the CW-H direction rotation and reverse rotation operation, and Respectively represent the rotation and reverse rotation operations in the CH-W direction, is the convolution operation, 、 and The modeling information extracted from the three directions of CW-H, CH-W and HW-C respectively; the feature map of the input Ordinary convolution is used to reduce the number of channels from Expand to 3 C , and then extract the feature sets of three directions through group convolution with a group number of 3 , for the feature set in three directions According to the following formula, the features of the three directions of CW-H, CH-W and HW-C are separated along the channel dimension: ； ； ； in, 、 and They are the characteristics in the three directions of CW-H, CH-W and HW-C. and The feature sets in three directions are Located in the middle 、 and Sub-blocks within the three channel range, is the input feature map The number of channels; S202: The modeling information extracted in the three directions of CW-H, CH-W, and HW-C and the separated features in the three directions of CW-H, CH-W, and HW-C are used to calculate the fusion features of the three directions according to the following formula: ； ； ； in, 、 and They are the fusion features of the three directions of CW-H, CH-W and HW-C, Represents element-by-element multiplication; the feature map of the input The function expression for extracting deep features is: ； in, is the deep feature, is a convolutional layer; the concatenated features Extract features to obtain the output features of the multi-directional hybrid modeling module The function expression is: 。 7. The high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to claim 6, characterized in that: described Among the fusion branches, the input of the first N-1 fusion branches is multi-scale spectral sampling information Spectral sampling information of the corresponding scale , the input of the Nth fusion branch is multi-scale spectral sampling information Spectral sampling information of the corresponding scale and initial spatial information , any The fusion branch includes the splicing module, convolution layer and Cascaded multi-directional hybrid modeling module, The splicing module of the fusion branch is used to convert the spectral sampling information of the corresponding scale and initial spatial information Splicing, the splicing modules of the other fusion branches are used to combine the spectral sampling information of the corresponding scale and the output features of the first-level multi-directional hybrid modeling module in the next fusion branch ,forward Any of the fused branches Output features of the multi-directional hybrid modeling module After upsampling, it is combined with the next fusion branch Output features of the multi-directional hybrid modeling module After concatenation, it is used as the output feature of the backward output, and finally the The first fusion branch Output features of the multi-directional hybrid modeling module After a convolution layer, the output is the fused high-resolution hyperspectral image .

8. A high-signal-to-noise and high-spatial-resolution hyperspectral fusion imaging system comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the high signal-to-noise and high spatial resolution hyperspectral fusion imaging method according to any one of claims 1 to 7 through a processor.

Images