Hyperspectral image fusion method, system, device and medium based on dual-branch spectral unmixing network

Abstract

The present invention discloses a hyperspectral image fusion method, system, device, and medium based on a dual-branch spectral unmixing network, relating to the field of hyperspectral imaging technology. The method comprises: obtaining an original hyperspectral image; inputting the original hyperspectral image into a dual-branch spectral unmixing network to obtain a target hyperspectral image; the spatial resolution of the target hyperspectral image is higher than the spatial resolution of the original hyperspectral image; the dual-branch spectral unmixing network comprises a fusion branch and a residual branch, and a total branch connected to both the fusion branch and the residual branch; wherein the fusion branch is used to combine the endmember matrix and resolution abundance of the original hyperspectral image to obtain a linear high-resolution hyperspectral image; the residual branch is used to obtain a residual image; and the total branch is used to reconstruct the linear high-resolution hyperspectral image and the residual image to obtain the target hyperspectral image. The present invention can effectively improve the spatial resolution of hyperspectral images.

Description

Technical Field

[0001] The present invention relates to the field of hyperspectral imaging technology, and in particular to a hyperspectral image fusion method, system, device and medium based on a dual-branch spectral unmixing network. Background Art

[0002] Hyperspectral imaging is an emerging imaging technology that typically uses imaging sensors to capture hundreds of spectral bands, providing wide spectral coverage. However, due to limitations such as signal-to-noise ratio, spatial resolution often needs to be compromised to ensure the quality of hyperspectral images, resulting in relatively low spatial resolution of the acquired hyperspectral images.

[0003] A large number of research results have been published on image fusion methods to enhance the spatial resolution of hyperspectral images. Typical methods include: fusion methods based on pan-sharpening, fusion methods based on model optimization, and fusion methods based on convolutional neural networks (CNNs). The basic idea is to use a fusion framework to integrate the high spatial resolution information of multispectral or panchromatic images of the same scene into the hyperspectral image.

[0004] While the fusion of hyperspectral and panchromatic images improves the spatial resolution of the hyperspectral image, it also suffers from spectral distortion and insufficient enhancement of texture detail. To address this issue, existing techniques primarily extract the endmember matrix of the low-resolution hyperspectral image using vertex component analysis (VCA). Using a CNN, the spatial and spectral information of both the HS and MS images is then simultaneously mined to improve the accuracy of abundance map estimation. This allows for direct reconstruction of the desired high-spatial-resolution hyperspectral image using a linear spectral mixture model.

[0005] Although this method performs well in maintaining spectral fidelity, it also has several drawbacks. First, the computational complexity of using VCA to extract endmembers increases rapidly with the amount of data and the number of endmembers, which limits its ability to process large-scale data. Second, it has shortcomings in exploring spatial information, which may lead to less than ideal visual effects. Finally, it does not consider the impact of various nonlinear factors in the spectral unmixing model on the imaging results, such as observation conditions, imaging system, and atmospheric transmission interference. These factors may produce residual errors, seriously affecting the image fusion effect.

[0006] Therefore, there is currently a lack of solutions to improve the spatial resolution while ensuring the quality of hyperspectral images. Summary of the Invention

[0007] The purpose of the present invention is to provide a hyperspectral image fusion method, system, device and medium based on a dual-branch spectral unmixing network, which can effectively improve the spatial resolution of hyperspectral images.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A hyperspectral image fusion method based on a dual-branch spectral unmixing network, comprising:

[0010] Obtain original hyperspectral images;

[0011] Inputting the original hyperspectral image into a dual-branch spectral unmixing network to obtain a target hyperspectral image; the spatial resolution of the target hyperspectral image is higher than the spatial resolution of the original hyperspectral image; the dual-branch spectral unmixing network includes a fusion branch and a residual branch, and a total branch connected to both the fusion branch and the residual branch;

[0012] The fusion branch is used to combine the endmember matrix and resolution abundance of the original hyperspectral image to obtain a linear high-resolution hyperspectral image; the residual branch is used to obtain a residual image; and the total branch is used to reconstruct the linear high-resolution hyperspectral image and the residual image to obtain a target hyperspectral image.

[0013] Optionally, the fusion branch includes a CNN-A network and a CNN-B network connected in sequence; the residual branch is constructed based on a nonlinear residual network; the residual branch includes a spatial module, a spectral module and a feature fusion module, and the spatial module and the spectral module are both connected to the feature fusion module; the CNN-B and the feature fusion module are both connected to the main branch.

[0014] Optionally, the spatial module and the spectral module are both constructed based on multiple convolutional layers; and the feature fusion module consists of 1×1 convolution and ReLU activation function.

[0015] Optionally, the network of the fusion branch is easily trained using stochastic gradient descent and back-propagation methods; and the network of the residual branch is trained using an adaptive moment estimation optimizer.

[0016] Optionally, the original hyperspectral image is input into a dual-branch spectral unmixing network to obtain a target hyperspectral image, specifically comprising:

[0017] In the fusion branch, a fast linear decomposition hyperspectral image algorithm is used to extract the endmember matrix of the original hyperspectral image, and a CNN-A network is used to extract and fuse the collaborative spatial and spectral information to obtain an intermediate enhanced spectral image. The auxiliary panchromatic image and the intermediate enhanced spectral image are further cascaded and input into the CNN-B network to estimate the required resolution abundance. The endmember matrix and the resolution abundance are then combined using a linear spectral mixture model to obtain a linear high-resolution hyperspectral image.

[0018] In the residual branch, the spatial texture and spectral details in the original hyperspectral image are extracted, and the obtained nonlinear spatial-spectral structural features are mapped to obtain a residual image with the same size and number of channels as the linear high-resolution hyperspectral image;

[0019] In the general branch, the residual image is added to the linear high-resolution hyperspectral image to reconstruct a fused target hyperspectral image.

[0020] The present invention also provides a hyperspectral image fusion system based on a dual-branch spectral unmixing network, comprising:

[0021] Image acquisition module, used to acquire original hyperspectral images;

[0022] a resolution processing module, configured to input the original hyperspectral image into a dual-branch spectral unmixing network to obtain a target hyperspectral image; wherein the spatial resolution of the target hyperspectral image is higher than the spatial resolution of the original hyperspectral image; the dual-branch spectral unmixing network includes a fusion branch and a residual branch, and a total branch connected to both the fusion branch and the residual branch;

[0023] The fusion branch is used to combine the endmember matrix and resolution abundance of the original hyperspectral image to obtain a linear high-resolution hyperspectral image; the residual branch is used to obtain a residual image; and the total branch is used to reconstruct the linear high-resolution hyperspectral image and the residual image to obtain a target hyperspectral image.

[0024] The present invention also provides an electronic device, characterized in that it includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the above-mentioned hyperspectral image fusion method based on the dual-branch spectral unmixing network.

[0025] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the hyperspectral image fusion method based on the dual-branch spectral unmixing network as described above.

[0026] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0027] The present invention discloses a hyperspectral image fusion method, system, device, and medium based on a dual-branch spectral unmixing network. The method comprises obtaining an original hyperspectral image; inputting the original hyperspectral image into the dual-branch spectral unmixing network to obtain a target hyperspectral image; the spatial resolution of the target hyperspectral image is higher than that of the original hyperspectral image; the dual-branch spectral unmixing network comprises a fusion branch, a residual branch, and a main branch connected to both the fusion branch and the residual branch; the fusion branch is used to combine the endmember matrix and resolution abundance of the original hyperspectral image to obtain a linear high-resolution hyperspectral image; the residual branch is used to obtain a residual image; and the main branch is used to reconstruct the linear high-resolution hyperspectral image and the residual image to obtain the target hyperspectral image. The present invention can effectively improve the spatial resolution of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is the main framework diagram for generating the intermediate space enhanced image in this embodiment;

[0030] Figure 2 This is the main framework diagram for generating linear high-resolution HSI in this embodiment;

[0031] Figure 3 This is the main framework diagram of generating a complete HSI after supplementing the residual in this embodiment;

[0032] Figure 4 This is the main framework diagram of the hyperspectral image fusion method based on the dual-branch spectral unmixing network in this embodiment;

[0033] Figure 5 Schematic diagram of the experimental results of the Pavia City Center dataset in this embodiment; wherein, part (a) is a schematic diagram of the experimental results of TBCNN; part (b) is a schematic diagram of the experimental results of DiCNN; part (c) is a schematic diagram of the experimental results of HyPNN; part (d) is a schematic diagram of the experimental results of CCNNF; and part (e) is a schematic diagram of the experimental results of HMISU;

[0034] Figure 6Schematic diagram of the experimental results of the Washington DC Mall dataset in this embodiment; wherein, part (a) is a schematic diagram of the experimental results of TBCNN; part (b) is a schematic diagram of the experimental results of DiCNN; part (c) is a schematic diagram of the experimental results of HyPNN; part (d) is a schematic diagram of the experimental results of CCNNF; and part (e) is a schematic diagram of the experimental results of HMISU;

[0035] Figure 7 Schematic diagram of the experimental results of the University of Houston dataset in this embodiment; wherein, part (a) is a schematic diagram of the experimental results of TBCNN; part (b) is a schematic diagram of the experimental results of DiCNN; part (c) is a schematic diagram of the experimental results of HyPNN; part (d) is a schematic diagram of the experimental results of CCNNF; and part (e) is a schematic diagram of the experimental results of HMISU;

[0036] Figure 8 Schematic diagram of the process of the hyperspectral image fusion method of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] The purpose of the present invention is to provide a hyperspectral image fusion method, system, device and medium based on a dual-branch spectral unmixing network, which can effectively improve the spatial resolution of hyperspectral images.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] like Figures 1-8 As shown, the present invention provides a hyperspectral image fusion method based on a dual-branch spectral unmixing network, comprising:

[0041] Step 100: Acquire an original hyperspectral image;

[0042] Step 200: Inputting the original hyperspectral image into a dual-branch spectral unmixing network to obtain a target hyperspectral image; the spatial resolution of the target hyperspectral image is higher than the spatial resolution of the original hyperspectral image; the dual-branch spectral unmixing network includes a fusion branch and a residual branch, and a total branch connected to both the fusion branch and the residual branch;

[0043] The fusion branch is used to combine the endmember matrix and resolution abundance of the original hyperspectral image to obtain a linear high-resolution hyperspectral image; the residual branch is used to obtain a residual image; and the total branch is used to reconstruct the linear high-resolution hyperspectral image and the residual image to obtain a target hyperspectral image.

[0044] Based on the above technical solution, the following detailed steps are provided:

[0045] As a summary, the purpose of this embodiment is to propose a dual-branch spectral unmixing network, which is divided into a fusion branch and a residual branch. In the fusion branch, the endmember matrix of the low-resolution hyperspectral image is first extracted by the fast linear decomposition hyperspectral image algorithm (FUN). Then, CNN-A is integrated into the method to achieve the extraction and fusion of collaborative spatial information and spectral information, thereby obtaining an intermediate enhanced spectral image. On this basis, an auxiliary panchromatic image is further used to cascade the enhanced spectral image into CNN-B to further explore spatial features and protect the original spectral features, thereby estimating the required high-resolution abundance. The abundance and endmembers are combined through a linear spectral mixture model to obtain a linear high-resolution hyperspectral image. In the residual branch, the high-resolution multispectral image is input into a nonlinear residual network to explore the fine nonlinear spatial-spectral structure features and map them to the residual image. Finally, the residual image is added to the linear high-resolution hyperspectral image to reconstruct a complete high-resolution hyperspectral image. The present invention can more accurately capture spatial information and spectral information, further improving the reconstruction accuracy of high-resolution hyperspectral images.

[0046] For the merge branch:

[0047] First, the FUN algorithm is used to obtain the endmember matrix from the hyperspectral image. The low-spatial-resolution hyperspectral image and the high-spatial-resolution multispectral image are then fused and fed into the CNN-A network to generate spatial details. These spatial details are then incorporated into the low-spatial-resolution hyperspectral image to generate an intermediate spatially enhanced hyperspectral image. The intermediate image is then sharpened and fused with the panchromatic image to obtain the abundance matrix. Finally, the product of the predicted abundance matrix and the endmember matrix, ignoring nonlinear factors, yields a linear high-resolution hyperspectral image.

[0048] For the residual branch:

[0049] The residual branch includes a spatial module and a spectral module, which simultaneously extracts spatial texture and spectral details in multispectral images.

[0050] The spatial module extracts spatial texture and local information from the input feature map. This better captures spatial information in the data, resulting in a more reliable and accurate representation. The spectral module extracts and abstracts the input spectral information to better describe and capture the spectral details in the data. These two features are then concatenated, and skip connections are introduced to produce a residual image with the same size and number of channels as the linear high-resolution hyperspectral image.

[0051] For the main branch:

[0052] The residual map is added to the linear high-resolution hyperspectral image to reconstruct a fused high-resolution hyperspectral image. The network in the fusion branch can be easily trained using stochastic gradient descent (SGD) and backpropagation methods, thereby accelerating abundance extraction. The nonlinear residual network in the residual branch is trained using the Adaptive Moment Estimation (ADAM) optimizer, which has faster convergence and better generalization performance.

[0053] As a specific embodiment, take the following data set as an example.

[0054] Simulated dataset

[0055] (1) Pavia City Center Dataset: This scene image was captured by the Reflection Optical System Imaging Spectrometer (ROSIS). The original image was cropped to a spatial size of 400×400, and 13 noisy spectral bands were discarded as observation data for high-resolution HSI, retaining 102 spectral bands with a wavelength range of 430 to 860 nm. The high-resolution MSI was obtained by averaging the original high-resolution HSI bands (i.e., 440-500, 540-600, 640-700, and 740-800 nm). The high-resolution MSI contains four spectral bands and has the same spatial resolution as the observed high-resolution HSI. At the same time, in order to simulate the corresponding high-resolution PAN image, the Wald protocol

[17] was adopted. According to Wald's scheme, we averaged the 520-900 nm band of the original HSI to simulate the corresponding high-resolution PAN image. Therefore, the fused image has the same resolution as the observed high-resolution HSI. To generate a low-resolution HSI with a spatial size of 100×100×102, we use a Gaussian filter of size 8×8 with a standard deviation of 0.5 to spatially blur the observed high-resolution HSI, and then downsample the result with a scaling factor of 4.

[0056] (2) Washington DC Mall Dataset: This scene data is an aerial hyperspectral image acquired by the Hydice sensor with a spatial size of 1208×307. The original image was cropped to a spatial size of 400×240, ignoring the bands in the atmospheric opaque region. A total of 191 bands covering the visible and near-infrared bands from 400 to 2400 nm were obtained as the observed high-resolution HSI data. The high-resolution MSI was obtained by averaging the original high-resolution HSI bands (i.e., 440-500, 840-900, 1040-1100, 1340-1440, 1540-1600, and 1840-1900 nm). The high-resolution MSI contains 6 spectral bands. According to Wald's scheme, we average the 401-2473 nm band of the original HSI to simulate the corresponding high-resolution PAN image. The observed high-resolution HSI is spatially blurred using a Gaussian filter of size 8×8 and a standard deviation of 0.5, and then downsampled with a scaling factor of 4 to generate a low-resolution HSI with a spatial size of 100×60.

[0057] Real dataset

[0058] (1) University of Houston dataset: This dataset was acquired by the ITRES CASI-1500 sensor. The data size is 349×1905 and contains 144 bands with a spectral range from 380nm to 1050nm. The original high-resolution HSI was clipped to a spatial size of 320×320 as the observed high-resolution HSI. The high-resolution MSI used in this paper was obtained by clipping the original high-resolution MSI image to a spatial size of 400×400. The high-resolution PAN image used was collected by the WorldView-II satellite, with a spatial size of 320×320 and covering a spectral range of 450-800nm. Subsequently, the observed high-resolution HSI was spatially blurred using a Gaussian filter with a size of 7×7 and a standard deviation of 0.5, and then downsampled with a scaling factor of 4 to generate a low-resolution HSI with a spatial size of 80×80.

[0059] In the experiments, five fusion quality evaluation metrics were used: PSNR, SSIM, SAM, ERGAS, and UIQI. Experiments on the Pavia dataset show that the best results are mostly achieved when the number of endmembers D is set to 50, as shown in Table 1. Therefore, the number of endmembers was set to 50 on all datasets. In the fusion branch, the number of training epochs was 200, the learning rate was set to 0.0001, the momentum was set to 0.9, and the batch size was 64. In the residual branch, the number of training epochs was set to 1000, and the initial learning rate was set to 0.003. The decay rate parameters β1 and β2 of ADAM were set to 0.9 and 0.999, respectively.

[0060] The comparison results of the evaluation indicators are shown in Table 2, Table 3 and Table 4 respectively. Figure 5 、 Figure 6 and Figure 7 A visual comparison is shown, with bold indicating the best value in each column.

[0061] Table 1 Experimental results of Pavia City Center dataset with different numbers of endmembers

[0062]

[0063]

[0064] Table 2 Numerical evaluation of different methods using the Pavia City Center dataset

[0065]

[0066] Table 3 Numerical evaluation of different methods using the Washington DC Mall dataset

[0067]

[0068] Table 4 Numerical evaluation of different methods using the University of Houston dataset

[0069]

[0070] Comparative experiment

[0071] In order to verify that the FUN algorithm used in the present invention is indeed superior to other advanced algorithms (such as the VCA algorithm), additional simulations were performed on the same Pavia dataset. As can be seen from Table 5, when the number of end members is 50, the performance of using the FUN algorithm to extract end members and finally obtain the target image is better than that of using the VCA algorithm.

[0072] Table 5 Experimental results of FUN algorithm and VCA algorithm on Pavia City Center dataset

[0073]

[0074] More generally, the accuracy of the pixels selected as endmembers was evaluated by calculating the spectral angles and comparing them with the results obtained when extracting endmembers using VCA. Table 6 shows the average spectral angle (SA) types obtained by the FUN and VCA algorithms at different signal-to-noise ratios (SNRs) and different numbers of endmembers, N. As can be seen, in the pavia dataset, which has a relatively high amount of noise (SNR = 20), the results obtained using the FUN algorithm are very close to those obtained using the VCA algorithm. However, in the pavia dataset, which has less noise, the results obtained by the FUN algorithm are significantly better than those obtained by the VCA algorithm, and the results improve as the number of endmembers increases. It can be concluded that the FUN algorithm is as accurate as the VCA algorithm in terms of the quality of the extracted endmembers, showing similar spectral angle values, and the FUN algorithm tends to be significantly better as the number of endmembers increases.

[0075] Table 6 Comparison of spectral angles obtained between the endmembers extracted by the FUN and VCA algorithms and the real endmembers present in the hyperspectral image

[0076]

[0077]

[0078] Ablation experiments

[0079] In order to verify the effectiveness of the network structure of the present invention, an ablation experiment was conducted to test the performance of the fusion branch, residual branch, and total branch network respectively. Their results on two data sets (simulated and real) are reported in Tables 7 and 8. On the simulated data set, it was observed that the performance of the fusion branch was relatively poor, but it was still competitive with other advanced methods. However, for the real data set, it was found that the performance of the fusion branch was affected to a certain extent. After the residual branch was added, the performance of the final result was improved. This shows that the present invention can definitely achieve excellent fusion performance.

[0080] Table 7 shows the ablation experiments on the simulated Pavia City Center dataset.

[0081]

[0082] Table 8 shows the ablation experiment on the real Houston dataset.

[0083]

[0084] As another specific embodiment, in order to improve the spatial resolution of hyperspectral images, a fusion method with other remote sensing images can be used, such as fusing hyperspectral images with panchromatic images. However, due to the large difference in spatial resolution between the two, and the fact that hyperspectral images contain a large number of redundant spectral bands, directly fusing the two may lead to spatial spectrum distortion problems. If panchromatic images are directly used to perform panchromatic sharpening of hyperspectral images, severe spatial spectrum distortion will result. In order to avoid this situation, considering that the spatial spectrum resolution of multispectral images is between the two, the spatial spectrum resolution of multispectral images can be used to first provide appropriate spatial information for hyperspectral images. Therefore, the fusion branch and residual branch proposed in this embodiment have the following specific algorithms:

[0085] According to the spectral mixture model, the target image is represented in the form of a 2D matrix Low spatial resolution hyperspectral imagery and high spatial resolution multispectral images It can be expressed as:

[0086] X=E X A X +N X (1)

[0087] Y=E Y A Y +N Y (2)

[0088] T=E X A+N (3)

[0089] Where M and N represent the spatial size, Λ and λ are the number of spectral bands, and E X ∈R Λ×D and E Y ∈R λ×D

[0090] is the endmember matrix, and is the abundance matrix, and D is the number of end members. N X and N Y represents other nonlinear effects in the mixed pixel. The spectral response function S can be used to relate the end member matrices of HSI and MSI as follows:

[0091] E Y =S·E X (4)

[0092] Fusion Branch

[0093] First, the FUN algorithm was used to obtain the endmember matrix from hyperspectral images. The FUN algorithm proved to be an effective and useful tool, producing more accurate results than other endmember extraction algorithms. Experimental results showed that the FUN algorithm outperformed the N-FINDR and VCA algorithms in terms of endmember extraction accuracy.

[0094] (1) Intermediate space enhancement results

[0095] First, the low spatial resolution hyperspectral image X and the high spatial resolution multispectral image Y are fused, and the low spatial resolution HSIX is spatially upsampled to generate an image of the same size as the high spatial resolution MSI, denoted as The two images are then concatenated together in a sequential stacking manner along the spectral dimension and divided into image blocks of size P×P×(Λ+λ) to be input into the network.

[0096] The network architecture of CNN-A consists of 4 convolutional layers, each followed by a batch normalization (BN) layer and a leaky rectified linear unit (Leaky ReLU) function with α=0.2. The first three convolutional layers each contain 64 filters of size 3×3, while the last convolutional layer contains Λ filters to generate Λ spatial details. The spatial details are then incorporated into the image X to produce a spatially enhanced image. The network architecture adopts a mirror padding mode to mirror the edges of the input sample blocks and pad around the edges to maintain the original size of the block image and avoid losing edge information. Figure 1 The main framework for generating intermediate space enhanced images is presented.

[0097] (2) Abundance estimation results

[0098] The intermediate space enhanced image obtained by CNN-A Need to be compatible with full-color images (The number of spectral bands is 1) for sharpening and fusion. First, the image blocks divided into P×P×(Λ+1) are sent to the CNN-B network, which also contains four convolutional layers. The first three convolutional layers contain 32 filters of size 3×3, and each convolutional layer is followed by a BN layer and a rectified linear unit (ReLU) function. In order to meet the non-negativity constraint of the abundance and the constraint of the sum being one, the last convolutional layer uses an output layer with a Softmax function, which contains convolution filters, and each pixel produces D abundance scores, thereby obtaining an abundance matrix. Finally, without considering nonlinear factors, the predicted abundance matrix A is the same as the end member matrix E X Reconstruct linear high-resolution HSI, i.e. Figure 2 The main framework for generating linear high-resolution hyperspectral images is presented.

[0099] Residual branch

[0100] Considering the influence of nonlinear factors, the fusion branch alone cannot perfectly reconstruct high-resolution hyperspectral images. Therefore, this paper introduces a nonlinear residual branch to focus on spatial and spectral details to supplement the residual of the hyperspectral image. The residual branch includes a spatial module and a spectral module, which are used to extract spatial texture and spectral details in multispectral images, respectively. The data input to the network is still cropped into patches of size P × P × λ.

[0101] For the spatial module, the MSI spectral channels are first upsampled to 64 using a 2D convolutional layer with a 1×1 kernel size. Subsequently, four convolution operations with a 3×3 kernel size are performed, followed by a ReLU activation function to extract spatial texture and local information from the input feature map. This better captures the spatial information in the data, resulting in a more reliable and accurate representation.

[0102] For the spectral module, a two-dimensional convolutional layer with a convolution kernel size of 1×1 is first used to upsample the MSI spectral channels to 32. Four 1×1 convolutional layers perform feature extraction and abstraction on the input spectral information to better describe and capture the spectral details in the data.

[0103] The present invention simultaneously performs both spatial and spectral module tasks to maximize the preservation and extraction of all important information. These two features are then concatenated to produce a spatial-spectral fusion feature with twice the number of HSI channels. Next, 1×1 convolution and ReLU activation functions are introduced to further fuse the spatial and spectral features while reducing the number of features to the same number of HSI channels. Skip connections are also introduced to increase training speed.

[0104] Finally, a 1×1 convolutional layer is used to map the nonlinear features into a residual image N, which has the same size and number of channels as HSI. After adding the residuals, we get the complete high-resolution HSI. The main framework of generating the complete HSI after supplementing the residuals is shown in Figure 3.

[0105] Main branch

[0106] Figure 4 The main framework of the proposed hyperspectral image fusion method based on a dual-branch spectral unmixing network is presented. To effectively train the proposed network, the network is constrained. The goal of the fusion branch is to minimize the prediction error as shown below:

[0107]

[0108]

[0109]

[0110] The loss function of the fusion branch is then defined as follows:

[0111]

[0112] For the residual branch, the predicted hyperspectral image Estimation loss function with the observed hyperspectral image T:

[0113]

[0114] This loss promotes the learning of residual features and improves the accuracy of fusion. and ||||0 are the F-norm and the 0-norm respectively.

[0115] The advantages of the hyperspectral image fusion method based on the dual-branch spectral unmixing network of this embodiment are as follows: (1) This method combines the linear spectral mixture model and fully utilizes the inherent characteristics of the hyperspectral image. The FUN algorithm is used to complete the endmember extraction, which is stronger than the most advanced algorithms in terms of the quality of the extracted endmembers; (2) The hyperspectral image is first sharpened and fused with the multispectral image, and the obtained intermediate spatial enhanced hyperspectral image is then fused with the panchromatic image to deeply explore the spatial features and protect the original spectral features, thereby completing the super-resolution reconstruction of the hyperspectral image; (3) The nonlinear residual network focuses on fine nonlinear features that are difficult to represent, such as high-frequency local spatial textures and spectral details. The addition of the mapped residual image can improve the high-resolution hyperspectral image.

[0116] Experiments on two widely used simulated datasets and a real remote sensing dataset show that the present invention can achieve fusion performance that is better than or comparable to the current state-of-the-art methods.

[0117] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0118] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A hyperspectral image fusion method based on a dual-branch spectral unmixing network, characterized in that: include: Obtain original hyperspectral images; The dual-branch spectral unmixing network includes a fusion branch and a residual branch, and a main branch connected to both the fusion branch and the residual branch; The fusion branch is used to combine the endmember matrix and resolution abundance of the original hyperspectral image to obtain a linear high-resolution hyperspectral image; In the residual branch, the spatial texture and spectral details in the original hyperspectral image are extracted, and the obtained nonlinear spatial-spectral structural features are mapped to obtain a residual image with the same size and number of channels as the linear high-resolution hyperspectral image; In the main branch, the residual image is added to the linear high-resolution hyperspectral image to reconstruct a fused target hyperspectral image; The more specific processing process for the fusion branch is as follows: Firstly, a fast algorithm of linear decomposition of hyperspectral images is used to obtain the endmember matrix in the hyperspectral image; (1) Intermediate space enhancement results: First low spatial resolution hyperspectral image and high spatial resolution multispectral images Fusion, low spatial resolution HSI Perform spatial upsampling to generate an image of the same size as the high spatial resolution MSI, denoted as ; The two images are then concatenated in a sequential stacking manner along the spectral dimension and input into the CNN-A network to generate a spatially enhanced image The network architecture uses a mirror filling mode, which mirrors the edges of the input sample blocks and fills them around the edges. (2) Abundance estimation results: The intermediate space enhanced image obtained by CNN-A network Need to be compatible with full-color images Perform sharp fusion and cascade input into the CNN-B network to obtain the abundance matrix; finally, without considering nonlinear factors, the predicted abundance matrix and endmember matrix Reconstruct linear high-resolution HSI, .

2. The hyperspectral image fusion method based on a dual-branch spectral unmixing network according to claim 1 is characterized in that: The fusion branch includes a CNN-A network and a CNN-B network connected in sequence; the residual branch is constructed based on a nonlinear residual network; the residual branch includes a spatial module, a spectral module and a feature fusion module, and the spatial module and the spectral module are both connected to the feature fusion module; the CNN-B network and the feature fusion module are both connected to the main branch.

3. The hyperspectral image fusion method based on a dual-branch spectral unmixing network according to claim 2 is characterized in that: The spatial module and the spectral module are both constructed based on multiple convolutional layers; the feature fusion module consists of 1×1 convolution and ReLU activation function.

4. The hyperspectral image fusion method based on a dual-branch spectral unmixing network according to claim 1 is characterized in that: The network of the fusion branch is easily trained using stochastic gradient descent and back-propagation methods; the network of the residual branch is trained using an adaptive moment estimation optimizer.

5. A hyperspectral image fusion system based on a dual-branch spectral unmixing network, applying the method according to any one of claims 1 to 4, characterized in that: include: Image acquisition module, used to acquire original hyperspectral images; a resolution processing module, configured to input the original hyperspectral image into a dual-branch spectral unmixing network to obtain a target hyperspectral image; wherein the spatial resolution of the target hyperspectral image is higher than the spatial resolution of the original hyperspectral image; The dual-branch spectral unmixing network includes a fusion branch and a residual branch, and a main branch connected to both the fusion branch and the residual branch; The fusion branch is used to combine the endmember matrix and resolution abundance of the original hyperspectral image to obtain a linear high-resolution hyperspectral image; The residual branch is used to obtain a residual image; the total branch is used to reconstruct the linear high-resolution hyperspectral image and the residual image to obtain a target hyperspectral image.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the hyperspectral image fusion method based on a dual-branch spectral unmixing network according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that It stores a computer program, which, when executed by a processor, implements the hyperspectral image fusion method based on a dual-branch spectral unmixing network as described in any one of claims 1 to 4.

Images