A Hyperspectral and Panchromatic Image Fusion Method Based on Multi-Level Information Extraction

Abstract

A hyperspectral and panchromatic image fusion method based on multi-level information extraction, comprising the following steps: (1) Image preprocessing: Filter the color image corresponding to the hyperspectral image, and then obtain pixels at a step size of r in both the row and column directions of the hyperspectral image to obtain a low-resolution hyperspectral image; perform upsampling operation on LR-HSI using the mean-invariant constraint to obtain UP-HIS; (2) Design the residual activity factor; (3) Design the local residual fusion module; (4) Multi-level information extraction and fusion: Calculate the residual learning ability of the local residual fusion module according to the residual activity factor to determine the number of local residual fusion modules; perform equally spaced spectral band extraction on the adopted UP-HIS, and finally output the reconstructed HR-HSI. The present invention uses multi-level spectral information and feature maps to batch share a hidden layer state, mutually perceive the average information of each channel map, and simultaneously complete feature extraction and fusion. It can improve the correlation between the spectral image and the panchromatic image and reduce spectral distortion.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image fusion, and particularly relates to a hyperspectral and panchromatic image fusion method based on multi-level information extraction. Background Art

[0002] Hyperspectral images (HSIs) characterize the intensity information of electromagnetic radiation on the surface of an object. Spectral technology can obtain information related to light in a substance, and imaging technology can capture two-dimensional image information of a target object. By using spectral imaging technology, the spatial information and spectral information of a target can be obtained simultaneously, resulting in a three-dimensional spectral data cube, thereby achieving the function of distinguishing different substances in a scene. Traditional spectral imaging technology uses a two-dimensional planar detector and it is difficult to obtain 3D hyperspectral data from a single shot. There are problems of mutual limitations in aspects such as spectral resolution, exposure time, spectral energy utilization rate, and signal-to-noise ratio. Usually, an improvement in one index is inevitably accompanied by a decrease in another index, which restricts the development and application of spectral imaging technology. In this case, a flexible alternative is to simultaneously acquire a high-spatial-resolution and low-spectral-resolution multispectral image (HR-MSI) and a low-spatial-resolution and high-spectral-resolution hyperspectral image (LR-HSI) of the same static scene, and then combine the advantages of both to obtain a high-resolution (HR) image in the spatial and spectral domains. The hyperspectral and panchromatic image fusion technology can not only qualitatively and quantitatively implement spectral analysis technology, but also obtain a more accurate and intuitive target status map through optical imaging technology, providing more precise information for applications such as analysis and evaluation, detection and monitoring, and measurement and calculation.

[0003] In recent years, many hyperspectral panchromatic fusion techniques have been proposed. According to whether deep learning is used, these methods can be roughly divided into two categories. The first category is classical traditional methods, mainly including methods based on component substitution, methods based on multi-resolution analysis, and methods based on variational optimization. The second category is deep learning methods that have become popular in recent years. The method based on component substitution uses the panchromatic image to replace the structural information component of the hyperspectral image in the new transform domain. However, the method based on component substitution loses some spectral information when replacing the spectral image structural information, so the fusion result often leads to serious spectral distortion and oversharpening. The method based on multi-resolution analysis injects the spatial information of the panchromatic image at a certain scale into the hyperspectral image at the corresponding scale, and then inverse-transforms to generate the fused image. The method based on multi-resolution analysis can solve the spectral distortion problem of the method based on component substitution to a certain extent. However, in the process of injecting structural information into the multi-resolution panchromatic image, the fusion result will degrade the spatial information. The method based on variational optimization solves the fusion problem by constructing an energy function and designing an optimization algorithm for the model function, but it is easy to fall into a local optimal solution during the optimization process, resulting in the final fused image quality not reaching the optimal. The hyperspectral panchromatic fusion method based on deep learning specifies the network input as LR-HSI and HR-MSI, designs network modules for feature extraction and fusion, and obtains the required hyperspectral data cube. Although deep learning has achieved success in pan-sharpening, it only focuses on the visual effects and objective evaluation indicators of fusing high spatial resolution hyperspectral images (HR-HSI), ignoring the running time problem.

[0004] The patent "Hyperspectral Panchromatic Sharpening Method Based on Deep Detail Injection Network" (Publication No.: CN113284067A, Application No.: 202110602214.6, Application Date: May 31, 2021) applied by Xi'an University of Technology discloses a method for extracting joint features of hyperspectral and panchromatic images using residual dense blocks. This method upsamples the low-resolution hyperspectral image and combines it with the panchromatic image, uses convolution to extract shallow features, and uses residual dense blocks for global feature fusion, solving the problem of limited fusion effect caused by insufficient detail extraction in the existing technology. The deficiency of this patent is that the use of multiple convolutional layers and residual blocks increases the network parameters, and its long running time makes it unable to meet the requirements of real-time imaging.

[0005] The patent "Hyperspectral and Panchromatic Image Fusion Method Based on Deep Learning and Matrix Decomposition" (publication number: CN110428387A, application number: 201910619754.8, application date: July 10, 2019) applied by Xidian University discloses a method for learning high-frequency detail features using a high-frequency information deep convolutional network. This method uses guided filtering to enhance the edge features of the super-resolution hyperspectral image, constructs an optimization equation to fuse the low-spatial-resolution hyperspectral image, panchromatic image, and prior image, and finally solves the optimization equation to obtain the hyperspectral fusion image. The drawback of this patent is that it does not consider the feature information of hyperspectral images at different scales, which is prone to spectral detail loss; it is easy to fall into a local optimal solution during the optimization process, and the complex processing of the two types of images will result in a long running time, which is not conducive to real-time fusion imaging.

[0006] In summary, although the existing methods can achieve good results in the hyperspectral and panchromatic image fusion task, they do not consider the multi-level information of hyperspectral images, the extracted spectral features are not comprehensive, and they all require a large amount of running time and cannot achieve the goal of real-time fusion. Therefore, there is still room for improvement in the hyperspectral and panchromatic image fusion method based on multi-level information extraction. Summary of the Invention

[0007] In order to overcome the above problems existing in the prior art, the purpose of the present invention is to provide a hyperspectral and panchromatic image fusion method based on multi-level information extraction, which uses multi-level spectral information and feature maps to batch share a hidden layer state, mutually perceive the average information of each channel map, and complete feature extraction and fusion simultaneously. When performing feature extraction, LR-HSI and HR-MSI with different depths are selected for layer-by-layer detail feature extraction to improve the correlation between the spectral image and the panchromatic image and reduce spectral distortion. Through the method of multi-level information fusion, the problems of spatial information loss and spectral distortion are effectively solved.

[0008] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0009] A hyperspectral and panchromatic image fusion method based on multi-level information extraction, comprising the following steps;

[0010] (1) Image preprocessing, filtering the high-spatial-resolution and low-spectral-resolution multi-spectral image (HR-MSI), and then obtaining the low-resolution hyperspectral image (LR-HSI) by taking pixels at a step size of r in both the row and column directions of the hyperspectral image; performing an upsampling operation on the LR-HSI using the mean invariant constraint to obtain the upsampled high-resolution hyperspectral image UP-HSI (Upsampled high resolution hyperspectral image);

[0011] (2) Design residual activity factor:

[0012] (3) Design local residual fusion module:

[0013] (4) Multilevel information extraction and fusion. Calculate the residual learning ability of the local residual fusion module according to the residual activity factor, and determine the number of local residual fusion modules; Extract the equally spaced spectral bands of the adopted UP-HSI, and finally output the reconstructed HR-HSI.

[0014] The specific steps of step (1) are as follows:

[0015] (1a) Select the CAVE dataset of hyperspectral images of indoor scenes and the Harvard dataset of hyperspectral images of real scenes. Each group of CAVE datasets contains hyperspectral images HR-HSI and corresponding multispectral images HR-MSI, and each group of Harvard datasets only contains hyperspectral images HR-HSI;

[0016] (1b) Use the spectral response curve of the Nikon700 camera to generate the corresponding color image (HR-MSI) of the hyperspectral image of each group of Harvard datasets;

[0017] (1c) Adopt the commonly used Word's Protocol rule in the field of remote sensing. First, filter the HR-HSI using a Gaussian blur kernel, and then obtain the low-resolution hyperspectral image (LR-HSI) by taking pixels at a step size of r in both the row and column directions of the hyperspectral image;

[0018] (1d) Use the bicubic interpolation algorithm to perform upsampling on the obtained LR-HSI, and at the same time add a constraint condition that the mean remains unchanged to obtain UP-HSI.

[0019] The specific steps of step (2) are as follows:

[0020] (2a) Define the residual activity factor η as follows:

[0021]

[0022] where x represents the input feature of the residual learning module, f(x) represents the mapping fitted by the residual module, and the ratio of the x-norm to the f(x)-norm is defined as the residual activity factor;

[0023] The residual module is a general feature extraction module, and the residual fusion module is a specific fusion module; The residual activity factor can be used to judge the performance of the general residual module;

[0024] (2b) Further characterize the learning activity of the residual module, and characterize η as a number between 0 and 1.0 as follows:

[0025]

[0026] Then, when the residual activity factor η is closer to around 1.0, the learning activity of the residual module is stronger, and the risk of degradation of the residual learning module is lower.

[0027] The specific steps of step (3) are as follows:

[0028] (3a) The first local residual fusion module takes HR-MSI as the input, and outputs feature map 0 after convolution operation. The output images of the remaining local residual modules are concatenated with HR-MSI in the channel dimension and then output feature map 0 after convolution operation;

[0029] (3b) Feature map 0 is concatenated with UP-HSI in the channel dimension, and the channel average mechanism is adopted to calculate the mean value of each channel. Then, through a fully connected neural network with a shared hidden state, a small coefficient is output for each channel. Finally, each channel is multiplied by the corresponding small coefficient to complete channel information perception and output features Figure 1 and features Figure 2 ;

[0030] (3c) Features Figure 1 and features Figure 2 are concatenated in the channel dimension, and feature extraction is performed through convolution and the number of channels is changed through convolution. Then, they are concatenated with feature map 0 with the same number of channels, and feature fusion is completed again using convolution to output the feature fusion image.

[0031] The specific steps of step (4) are as follows:

[0032] (4a) Calculate the residual learning ability of the local residual fusion module in (3a) according to the residual activity factor defined in (2b), and determine the number of local residual fusion modules;

[0033] (4b) The UP-HSI adopted in (3b) is an equally spaced spectral band extraction;

[0034] (4c) A purification module is used for fusion fine-tuning. The purification module includes two groups of convolution operations, and finally the reconstructed HR-HSI is output.

[0035] In (4a), the residual activity factor n is calculated for each added local residual fusion module. When the value of n is close to 1, it represents strong residual activity, indicating that the added local residual fusion module at this time is effective. Based on this, new local residual fusion modules are added in sequence and the number is determined according to the calculated n.

[0036] The beneficial effects of the present invention:

[0037] 1. The present invention designs an upsampling method with the constraint of invariant mean, and constrains the process of bicubic interpolation algorithm for upsampling operation of LR-HSI through this condition;

[0038] 2. The present invention designs a residual activity factor for judging the activity of residual modules, which can be used as the basis for the number of residual modules;

[0039] 3. The present invention designs a fusion module based on local residuals for feature extraction and fusion. This module uses fewer parameters, has better feature extraction and fusion effects, and can reconstruct hyperspectral images at video frame rate;

[0040] 4. The present invention selects LR-HSI cascaded with HR-MSI of different depths, extracts detailed features layer by layer, improves the correlation between hyperspectral images and color images, and reduces spectral distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a hyperspectral and panchromatic image fusion method for multi-level information extraction.

[0042] Figure 2 It is a schematic diagram of extracting block details. DETAILED DESCRIPTION OF THE INVENTION

[0043] The present invention will be further described in detail below with reference to the drawings and embodiments.

[0044] As Figure 1 、 Figure 2 shown: A hyperspectral and panchromatic image fusion method based on multi-level information extraction includes the following steps;

[0045] (1) Image preprocessing:

[0046] (1a) Select two publicly available datasets commonly used in the field of hyperspectral fusion reconstruction, the CAVE dataset containing 32 groups of indoor scene hyperspectral images and the Harvard dataset containing 50 groups of real scene hyperspectral images. Each group of the CAVE dataset contains 31 hyperspectral images HR-HSI and corresponding multispectral images HR-MSI, and each group of the Harvard dataset only contains 31 hyperspectral images HR-HSI;

[0047] (1b) Generate the corresponding color image (HR-MSI) of each hyperspectral image in the Harvard dataset using the spectral response curve of the Nikon700 camera;

[0048] (1c) Using the Word's Protocol rule commonly used in the field of remote sensing, first filter the HR-HSI with a 7×7 Gaussian blur kernel with a mean of 0 and a variance of 2, and then obtain the low-resolution hyperspectral image (LR-HSI) by taking pixels at a step size of r in both the row and column directions of the hyperspectral image;

[0049] (1d) Perform upsampling on the obtained LR-HSI using the bicubic interpolation algorithm, and at the same time add a constraint condition that the mean remains unchanged to obtain the UP-HSI.

[0050] (2) Design the residual activity factor:

[0051] (2a) Define the residual activity factor η as follows:

[0052]

[0053] Where x represents the input feature of the residual learning module, f(x) represents the mapping fitted by the residual module, and the ratio of the x-norm to the f(x)-norm is defined as the residual activity factor;

[0054] (2b) Further characterize the learning activity of the residual module, and characterize η as a number between 0 and 1.0 as follows:

[0055]

[0056] Then, when the residual activity factor η is closer to 1.0, the learning activity of the residual module is stronger, and the risk of degradation of the residual learning module is lower;

[0057] (3) Design the local residual fusion module:

[0058] (3a) Concatenate the output image of the local residual fusion module with the HR-MSI in the channel dimension. For the first local residual fusion module without a previous input, directly process the HR-MSI and output the feature map 0 after a 3×3 convolution operation;

[0059] (3b) Concatenate the feature map 0 with the UP-HSI in the channel dimension, and adopt the channel average mechanism to calculate the mean of each channel. Then, through a fully connected neural network with a shared hidden state, output a small coefficient for each channel. Finally, multiply each channel by the corresponding small coefficient to complete the channel information perception and output the feature Figure 1 and the feature Figure 2 ;

[0060] (3c) The feature Figure 1 and the feature Figure 2Cascade at the channel dimension, perform feature extraction through a 3×3 convolution, then change the number of channels through a 1×1 convolution, and then cascade with the feature map 0 of the same number of channels. Use a 3×3 convolution and a 1×1 convolution again to complete feature fusion and output the feature fusion image;

[0061] (4) Multi-level information extraction and fusion:

[0062] (4a) Calculate the residual learning ability of the local residual fusion module in (3a) according to the residual activity factor defined in (2b), and determine that the number of local residual fusion modules is 5;

[0063] (4b) The UP-HSI used in (3b) is an equally spaced spectral band extraction. The first and second local residual fusion modules use 8-channel UP-HSI, and the third, fourth, and fifth local residual fusion modules use 31-channel UP-HSI; for a dataset containing 31 spectral bands, equally spaced extract the images of the 1st, 5th, 9th, 13th, 17th, 21st, 25th, and 29th channels, a total of 8 channels;

[0064] (4c) The output of the fifth local residual fusion module is cascaded with the 31-channel UP-HSI, and a purification module is used for fusion fine-tuning. The purification module contains two groups of 3×3 and 1×1 convolution operations, and finally outputs the reconstructed HR-HSI.

[0065] In the above (4a), calculate the residual activity factor n for each added local residual fusion module. When the value of n is close to 1, it represents strong residual activity, indicating that the added local residual fusion module at this time is effective. Based on this, add new local residual fusion modules in turn and determine the number according to the calculated n.

[0066] Figure 1 The original inputs are LR-HSI and HR-MSI, where the image resolution of each channel of LR-HSI is low. Although HR-MSI has a high spatial resolution, the number of spectral bands is small. Through the network for feature extraction and purification fusion, the finally output HR-HSI not only contains rich spectral channel information, but each channel image also has a high spatial resolution, and can complete high-quality fusion.

[0067] Figure 2 is Figure 1 A detailed display of the fusion module in the overall network framework.

[0068] The hyperspectral and panchromatic image fusion based on multi-level information extraction of the present invention is a lightweight method for remote sensing image pansharpening. This network can reconstruct hyperspectral images at video frame rate to meet the real-time requirements. By analyzing the working mechanism of the residual module that plays a key role in the existing fusion network, a residual activity factor is defined to determine the ability of the residual module used, thereby determining the number of residual blocks. Based on this principle, a lightweight fusion network is designed. By using the multi-level spectral information and sharing a hidden layer state of the feature maps in batches, the correlation between spectral channels is increased, the average information of each channel map is mutually perceived, and feature extraction and fusion are completed simultaneously. At the same time, LR-HSI and HR-MSI with different depths are selected to extract detailed features layer by layer, improving the correlation between the spectral image and the panchromatic image and reducing spectral distortion.

[0069] According to the above principle, the technical idea of the present invention is as follows: First, an upsampling operation is performed on the LR-HSI using the mean invariant constraint, and the upsampled UP-HSI is equally spaced and cascaded with the HR-MSI. A residual activity factor is defined to quantify the learning ability of the residual module, and the number of residual blocks to be used is selected according to the residual activity factor. A lightweight residual constraint block based on global variance fine-tuning is designed to extract and fuse the spectral feature information of the LR-HSI and the spatial structure information of the HR-MSI. Finally, a fusion purification network is used for fine-tuning and correction to output the final reconstructed high-resolution hyperspectral image.

[0070] The present invention solves the problem of texture detail loss caused by insufficient detail extraction in the process of hyperspectral and panchromatic image fusion of the existing hyperspectral fusion algorithms, and improves the spatial detail information of the reconstructed hyperspectral image;

[0071] The present invention solves the problem of spectral distortion caused by insufficient feature extraction of each spectral band image in the process of hyperspectral and panchromatic image fusion of the existing hyperspectral fusion algorithms, and improves the spectral detail information of the reconstructed hyperspectral image;

[0072] The present invention solves the problem of the number of residual blocks used in the deep learning-based hyperspectral fusion method. The designed residual activity factor is used to judge the number of residual blocks selected in the feature extraction process, realizing the lightweight of the fusion network and improving the reconstruction efficiency.

Claims

1. A hyperspectral and panchromatic image fusion method based on multi-level information extraction, characterized in that, It includes the following steps; (1) Image preprocessing: Filter the high-spatial-resolution and low-spectral-resolution multispectral image (HR-MSI), and then obtain the low-resolution hyperspectral image (LR-HSI) by acquiring pixels at a step size of r in both the row and column directions of the hyperspectral image; Perform upsampling operation on the LR-HSI using the mean-invariant constraint to obtain the upsampled high-resolution hyperspectral image UP-HSI; (2) Design the residual activity factor: (3) Design the local residual fusion module: (4) Multilevel information extraction and fusion: Calculate the residual learning ability of the local residual fusion module according to the residual activity factor to determine the number of local residual fusion modules; Extract the equally spaced spectral bands of the adopted UP-HSI, and finally output the reconstructed HR-HIS; The specific content of step (2) is as follows: (2a) Define the residual activity factor η as follows: where x represents the input feature of the residual learning module, f(x) represents the mapping fitted by the residual module, and the ratio of the x-norm to the f(x)-norm is defined as the residual activity factor; (2b) Further characterize the learning activity of the residual module, and characterize η as a number between 0 and 1.0 as follows: Then, when the residual activity factor η is closer to 1.0, the learning activity of the residual module is stronger, and the risk of degradation of the residual learning module is lower; The specific content of step (3) is as follows: (3a) The first local residual fusion module takes the HR-MSI as the input, and outputs the feature map 0 after convolution operation. The output images of the remaining local residual modules are cascaded with the HR-MSI in the channel dimension and then output the feature map 0 after convolution operation; (3b) Cascade the feature map 0 with the UP-HSI in the channel dimension, and adopt the channel average mechanism to calculate the mean value of each channel. Then, pass through a fully connected neural network with a shared hidden state to output a small coefficient for each channel. Finally, each channel is multiplied by the corresponding small coefficient to complete the channel information perception, and output the feature map 1 and the feature map 2; (3c) Cascade the feature map 1 and the feature map 2 in the channel dimension, perform feature extraction through convolution and change the number of channels through convolution, then cascade with the feature map 0 with the same number of channels, and use convolution again to complete feature fusion, and output the feature fusion image.

2. A hyperspectral and panchromatic image fusion method based on multi-level information extraction according to claim 1, characterized in that The specific content of step (1) is as follows: (1a) Select the CAVE dataset of indoor scene hyperspectral images and the Harvard dataset of real scene hyperspectral images. Each group of CAVE datasets contains the hyperspectral image HR-HSI and has the corresponding multispectral image HR-MSI. Each group of Harvard datasets only contains the hyperspectral image HR-HSI; (1b) Generate the corresponding color image (HR-MSI) of each group of Harvard dataset hyperspectral images using the spectral response curve; (1c) Adopt the Word's Protocol rule in the remote sensing field. First, filter the HR-HSI using a Gaussian blur kernel, and then obtain the low-resolution hyperspectral image (LR-HSI) by acquiring pixels at a step size of r in both the row and column directions of the hyperspectral image; (1d) Upsample the obtained LR-HSI using the bicubic interpolation algorithm, and at the same time add the constraint condition of unchanged mean to obtain UP-HSI.

3. A hyperspectral and panchromatic image fusion method based on multi-level information extraction according to claim 1, characterized in that (4) The specific steps are as follows: (4a) Calculate the residual learning ability of the local residual fusion module in (3a) according to the residual activity factor defined in (2b), and determine the number of local residual fusion modules; (4b) The UP-HSI used in (3b) is an equal-interval spectral band extraction; (4c) Use a purification module for fusion fine-tuning. The purification module includes two groups of convolution operations, and finally output the reconstructed HR-HSI.

4. A hyperspectral and panchromatic image fusion method based on multi-level information extraction according to claim 3, characterized in that, (4a) Calculate the residual activity factor n for each added local residual fusion module. When the value of n is close to 1, it represents strong residual activity, indicating that the added local residual fusion module at this time has an effect. Based on this, add new local residual fusion modules in turn and determine the number according to the calculated n.

Images