A method for training network models for hyperspectral image enhancement

Abstract

This invention discloses a method for training a network model for hyperspectral image enhancement, the method comprising the following steps: obtaining a training dataset; and processing the hyperspectral images H in the training dataset. LR and the hyperspectral image H' obtained by upsampling HR The input is fed into the spectral correction branch of the network model to correct the hyperspectral images H in the training dataset. LR Correction is performed to obtain the corrected hyperspectral image, which is then converted into a multispectral image I. HR The input is fed into the spatial reconstruction branch of the network model for decomposition and recombination to extract the multispectral image I. HR The spatial information in the image is injected into the spectral correction branch using learnable weights, enabling it to adaptively inject spatial information into the hyperspectral image H'. HR The hyperspectral image obtained is used to train the network model using a hybrid loss function. This invention solves the problems of existing technologies that ignore the differences in required information for different bands and fail to extract sufficient spatial details, thereby obtaining richer spatial details and ensuring better spatial structure quality.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a network model training method for hyperspectral image enhancement, and more particularly to a hierarchical adaptive learning network training method for hyperspectral image enhancement, specifically for spectral correction and spatial reconstruction. Background Technology

[0002] Due to hardware limitations, hyperspectral images often cannot achieve high spatial resolution. Improving the spatial resolution of hyperspectral images is mostly achieved by enhancing the performance of the acquisition equipment, but this is costly. Therefore, how to inject high-resolution spatial details while preserving spectral information has always been a challenge in hyperspectral image enhancement. Currently, deep learning has been widely applied to various computer vision tasks. Convolutional neural networks (CNNs), as the core method of deep learning, have the ability to understand image information. Through careful design of loss functions or feature extraction modules, CNNs can achieve ideal results in hyperspectral image enhancement tasks.

[0003] However, existing image processing methods still have shortcomings: 1) Previous methods usually link multispectral images and hyperspectral images directly and train them as a whole. However, the features of the two are significantly different, and training them as a whole will bring great obstacles to fusion and reconstruction; 2) In most methods, the role of hyperspectral images is limited to upsampling input data, which may distort the original complete spectral information due to spatial interpolation, further leading to distortion of the reconstructed hyperspectral image. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a network model training method for hyperspectral image enhancement, so as to solve the technical problems mentioned in the prior art.

[0005] A method for training a network model for hyperspectral image enhancement, the method comprising the following steps:

[0006] S1. Acquire multiple pairs of multispectral images with the same angle in multiple scenes. HR and hyperspectral images H LR To obtain the training dataset;

[0007] S2, The hyperspectral images H in the training dataset LR And the hyperspectral image H LR The hyperspectral image H' obtained by upsampling HR The input is fed into the spectral correction branch of the network model to correct the hyperspectral images H in the training dataset. LR Correction is performed to obtain the corrected hyperspectral image.

[0008] Multispectral image I HR The input is fed into the spatial reconstruction branch of the network model for decomposition and recombination to extract the multispectral image I. HR Spatial information in the image is used to obtain the reconstructed multispectral image.

[0009] The extracted spatial information is injected into the spectral correction branch of the corresponding layer using a learnable weight, enabling it to adaptively inject spatial information into the hyperspectral image H'. HR In the process, an enhanced hyperspectral image was obtained.

[0010] S3. Train the network model using a hybrid loss function until it converges.

[0011] Optionally, in S1, the hyperspectral image H' HR With the multispectral image I HR They are the same size.

[0012] Optionally, the acquired hyperspectral image and multispectral image are downsampled by a factor of 4 according to the Wald protocol, and additive noise is added to the downsampled hyperspectral image to simulate a low-resolution hyperspectral image, thus obtaining the hyperspectral image H'. HR ;

[0013] The acquired multispectral image was downsampled by 4 times according to the Wald protocol to obtain the multispectral image I. HR .

[0014] Optionally, the upsampling method is:

[0015] A trainable upsampling module is selected and inserted into the network model. The downsampled hyperspectral image H is then processed by the trainable upsampling module. LR Perform a 4x upsampling.

[0016] Optionally, the trainable upsampling module is a pixel-shuffle module.

[0017] Optionally, in step S2, if the network depth of the spectral correction branch is set to d, then the output result of the i-th layer of the spectral correction branch is:

[0018]

[0019] Where j = i-1, Lecel i Let res be the i-th layer of the spectral correction branch, spe(·) be the residual learning, and spe(·) be the spectral correction. For the hyperspectral image H' of layer j HR Features For the hyperspectral image H of layer j LR Features and These are the outputs of Spe(·), α i Let θ be the learnable weight of the i-th layer. i The parameters that the spectral correction branch of the i-th layer needs to learn;

[0020] When i = 1, Lecel i As the first layer, the network model uses 3×3 convolutions to activate the hyperspectral image H. LR and hyperspectral image H' HR To output the corresponding and

[0021] When i = d, Lecel i As the last layer, the network model uses 3×3 convolutions to respectively... and Reconstruction is performed to obtain a hyperspectral image. and hyperspectral images

[0022] Optionally, in S2, the spectral correction branch has a spectral splicing module and a compression correction module;

[0023] For input sample F x For x∈{H,L}, the input channel c of the spectral correction branch is compressed using a convolution kernel of arbitrary size. Then input sample F x Through the spectral correction branch The input channel receives data four times, and the spectral stitching module sequentially stitches together the sample data from the four inputs to obtain the implicit features.

[0024]

[0025] Among them, in obtaining implicit features When, input sample F H and input sample F L Parameters are shared between them, cat is a cascading operation, F ki Corresponding to the features of different convolution kernels, k1 represents a 1x1 convolution kernel, k2 represents a 3x3 convolution kernel, k3 represents a 5x5 convolution kernel, and k4 represents a 7x7 convolution kernel;

[0026] The input sample F is calculated using the compression correction module. H Channel attention vector V FH and input sample F L Channel attention vector VFL :

[0027]

[0028] The correction vector V is calculated through a correction layer consisting of a 1x1 convolution and a sigmoid function. r :

[0029]

[0030] The hyperspectral image H LR Spectral correction parameter F' x for:

[0031]

[0032] Through the spectral correction parameter F' x Correcting the hyperspectral image H LR The hyperspectral image is obtained.

[0033] Optionally, in step S2, the multispectral image I is extracted using a high-pass filter. HR The high-frequency information in the image is used as prior information to compensate for the deficiencies in spatial information extraction. The spatial reconstruction branch uses the high-frequency information to guide the convolutional neural network to learn and extract multispectral images. HR detail;

[0034]

[0035] Among them, HF cnn The high-frequency information extracted by the high-pass filter, HF non The high-frequency information extracted by the high-pass filter is represented by G, where G is the high-pass filter and HF is the multispectral image I. HR High-frequency information, C(·) represents the convolution operation;

[0036] The HF learned cnn and HF non Reconstruction yields multispectral images

[0037]

[0038] Where Rebuild(·) is the reconstruction layer, and the multispectral image Reconstruction was performed using a 3×3 convolution kernel.

[0039] Optionally, in S2, the network model will assign each layer corresponding to the spatial reconstruction branch to... Injected into the layer corresponding to the spectral reconstruction branch In the middle, then using a jump connection to obtain

[0040]

[0041] Where, α i Let be the learnable weight values ​​of the i-th layer in the spatial reconstruction branch.

[0042] Optionally, in step S3, the hybrid loss function includes a loss function for constrained spatial information. Spa Loss function for constraining spectral information Spe The hybrid loss function Loss is:

[0043] Loss = Loss m +λ1·Loss Spa +λ2·Loss Spe ;

[0044]

[0045] Loss m =||H' HR -H HR ||;

[0046]

[0047] Where λ1 and λ2 are hyperparameters, and Loss is... m For hyperspectral image H' HR and hyperspectral images The loss function between For H x The spectral vector represented in pixels, x∈{HR,LR}, Loss spe For multispectral images With multispectral image I HR The spectral error between them, where Cosine is the cosine similarity. For the i-th endmember in a high-resolution hyperspectral image, To reconstruct the i-th endmember in a high-resolution hyperspectral image, For the i-th endmember in a low-resolution hyperspectral image, To reconstruct the i-th endmember in a low-resolution hyperspectral image, For endmember p and endmember The dot product between p and p, and the range of the dot product is [-1, 1]; ||p|| is the Euclidean norm of the endmember p. For end element The Euclidean norm; where the closer the dot product is to 1, the closer it is to the endmember p and the endmember The more similar they are, the closer their dot products are. -1 indicates that the endmember p and the endmember p are similar. The less similar the dot product, the less likely it is that the endmember p is similar to the endmember 0. Orthogonal.

[0048] The beneficial effects that this invention can produce include:

[0049] The present invention provides a network model training method for hyperspectral image enhancement, which mainly designs two branches: a spectral correction branch and a spatial reconstruction branch. The spectral correction branch includes a spectral stitching module and a compression correction module, which are mainly used to correct the final output hyperspectral image using the spectral information of the original hyperspectral image, thus better avoiding spectral distortion. The spatial reconstruction branch uses a hierarchical structure and learnable weights to adaptively inject the extracted high-frequency information into the enhanced image, thereby solving the problems of existing technologies that ignore the differences in the required information for different bands and insufficient extraction of spatial details, thus obtaining richer spatial details and ensuring better spatial structure quality. At the same time, a hybrid loss function is adopted, and constraints are added to the reconstruction of hyperspectral and multispectral images to ensure the collaborative learning between spectral and spatial information, resulting in higher quality final images. Attached Figure Description

[0050] Figure 1 This is a framework diagram of a network model training method for hyperspectral image enhancement according to the present invention;

[0051] Figure 2 This is a schematic diagram of the network structure of the spectral correction branch of the present invention;

[0052] Figure 3 This is a schematic diagram of the spectral splicing module structure of the spectral correction branch of the present invention;

[0053] Figure 4 This is a schematic diagram of the compressed correction module structure of the spectral correction branch of the present invention;

[0054] Figure 5 This is a schematic diagram of the network structure of the spatial reconstruction branch of the present invention;

[0055] Figure 6 Figure 1 shows the SAM distribution results of this invention on the CAVE dataset, with the color bar range being [0, 0.2]. Among them, Figure (a) is a multispectral image, Figure (b) is the result image of the CNMF method, Figure (c) is the result image of the Bayesian method, Figure (d) is the result image of the GFPCA method, Figure (e) is the result image of the DDLPS method, Figure (f) is the result image of the GuideNet method, Figure (g) is the result image of the ADKNet method, Figure (h) is the result image of the BUSIF method, and Figure (i) is the output image of the network model of this invention.

[0056] Figure 7 This is a graph showing the SAM distribution results of this invention on the Houston dataset, with the color bar range being...

[0057] [0,0.2]; where, Figure (a) is a multispectral image, Figure (b) is the result image of the CNMF method, Figure (c) is the result image of the Bayesian method, Figure (d) is the result image of the GFPCA method, Figure (e) is the result image of the DDLPS method, Figure (f) is the result image of the GuideNet method, Figure (g) is the result image of the ADKNet method, Figure (h) is the result image of the BUSIF method, and Figure (i) is the output result image of the network model of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Please see Figures 1-5 As shown, this invention provides a method for training a network model for hyperspectral image enhancement, the method comprising the following steps:

[0060] Step 1: Acquire multiple pairs of multispectral images at the same angle in multiple scenes. HR and hyperspectral images H LR In order to obtain the training dataset.

[0061] In the above, the network model downsamples the acquired hyperspectral image and multispectral image respectively to obtain the multispectral image I. HR and hyperspectral images H LR ; and the hyperspectral image H LR Upsampling was performed to obtain the hyperspectral image H' HR Hyperspectral image H' HR With multispectral image I HR The dimensions are the same. Specifically, the downsampling method is as follows: according to the Wald protocol, the acquired hyperspectral image and multispectral image are downsampled by a factor of 4, and additive noise is added to the downsampled hyperspectral image to simulate a low-resolution hyperspectral image, thus obtaining the hyperspectral image H'. HR The acquired multispectral image was downsampled by 4 times according to the Wald protocol to obtain multispectral image I. HR The upsampling method is as follows: a trainable upsampling module (Pixel-shuffle module) is selected and inserted into the network model. The upsampling module is then used to process the downsampled hyperspectral image H.LR Perform a 4x upsampling.

[0062] Step 2: Extract the hyperspectral images H from the training dataset. LR And the hyperspectral image H LR The hyperspectral image H' obtained by upsampling HR The input is fed into the spectral correction branch of the network model to correct the hyperspectral images H in the training dataset. LR Correction is performed to obtain the corrected hyperspectral image. Multispectral image I HR The input is fed into the spatial reconstruction branch of the network model for decomposition and recombination to extract the multispectral image I. HR Spatial information in the image is used to obtain the reconstructed multispectral image. The extracted spatial information is injected into the spectral correction branch of the corresponding layer using a learnable weight, enabling it to adaptively inject spatial information into the hyperspectral image H'. HR In the process, an enhanced hyperspectral image was obtained.

[0063] In this embodiment, the network depth of the spectral correction branch is set to d, then the output result of the i-th layer of the spectral correction branch is:

[0064]

[0065] Where j = i-1, Lecel i Let i be the i-th layer of the spectral correction branch, res be the residual learning, and Spe(·) be the spectral correction. For the hyperspectral image H' of layer j HR Features For the hyperspectral image H of layer j LR Features and These are the outputs of Spe(·), α i Let θ be the learnable weight of the i-th layer. i These are the parameters that need to be learned for the i-th layer spectral correction branch; when i=1, Lecel i As the first layer, the network model uses 3×3 convolutions to activate the hyperspectral image H. LR and hyperspectral image H' HR To output the corresponding and When i = d, Lecel i As the last layer, the network model uses 3×3 convolutions to respectively... and Reconstruction is performed to obtain a hyperspectral image. and hyperspectral images

[0066] In the above, the spectral correction branch has a spectral stitching module and a compression correction module; for the input sample F x For x∈{H,L}, the input channel c of the spectral correction branch is compressed using a convolution kernel of arbitrary size. Then input sample F x Through the spectral correction branch The input channel receives data four times, and the sample data from the four inputs are sequentially stitched together by a spectral stitching module to obtain the implicit features.

[0067]

[0068] Among them, in obtaining implicit features When, input sample F H and input sample F L Parameters are shared between them, cat is a cascading operation, F ki Corresponding to the features of different convolution kernels, k1 represents a 1x1 convolution kernel, k2 represents a 3x3 convolution kernel, k3 represents a 5x5 convolution kernel, and k4 represents a 7x7 convolution kernel;

[0069] Then, the input sample F is calculated using the compression correction module. H Channel attention vector V FH and input sample F L Channel attention vector V FL :

[0070]

[0071] The correction vector V is calculated through a correction layer consisting of a 1x1 convolution and a sigmoid function. r :

[0072]

[0073] Hyperspectral image H LR Spectral correction parameter F' x for:

[0074]

[0075] Through the spectral correction parameter F' x Correcting hyperspectral images H LR Obtain hyperspectral images

[0076] In this embodiment, a high-pass filter is used to extract the multispectral image I. HRThe high-frequency information in the image is used as prior information to compensate for the shortcomings of spatial information extraction. The spatial reconstruction branch uses high-frequency information to guide the convolutional neural network to learn and extract multispectral images. HR detail;

[0077]

[0078] Among them, HF cnn The high-frequency information extracted by the high-pass filter, HF non The high-frequency information extracted by the high-pass filter is represented by G, where G is the high-pass filter and HF is the multispectral image I. HR The high-frequency information, C(·) represents the convolution operation; the learned HF cnn and HF non Reconstruction yields multispectral images

[0079] I' HR =Rebuild(HF) cnn +HF non );

[0080] Where Rebuild(·) is the reconstruction layer, and the multispectral image is... A 3×3 convolutional kernel is used for reconstruction. In the above, the network model divides each layer in the spatial reconstruction branch into layers corresponding to... Spectral correction parameters injected into the corresponding layer of the spectral reconstruction branch In the middle, then using a jump connection to obtain

[0081]

[0082] Where, α i Let be the learnable weight values ​​of the i-th layer in the spatial reconstruction branch.

[0083] Step 3: Train the network model using a hybrid loss function until convergence. The hybrid loss function includes a loss function that constrains spatial information (Loss). Spa Loss function for constraining spectral information Spe The mixed loss function Loss is:

[0084] Loss = Loss m +λ1·Loss Spa +λ2·Loss Spe ;

[0085]

[0086] Loss m =||H' HR -H HR ||;

[0087]

[0088] Where λ1 and λ2 are hyperparameters, and Loss is... m For hyperspectral image H' HR and hyperspectral images The loss function between For H x The spectral vector is represented in pixels, x∈{HR,LR}. It should be noted that HR refers to high resolution; LR refers to low resolution; Loss spe For multispectral images With multispectral image I HR The spectral error between them, where Cosine is the cosine similarity. For the i-th endmember in a high-resolution hyperspectral image, To reconstruct the i-th endmember in a high-resolution hyperspectral image, For the i-th endmember in a low-resolution hyperspectral image, To reconstruct the i-th endmember in a low-resolution hyperspectral image, For endmember p and endmember The dot product between p and p, and the range of the dot product is [-1, 1]; ||p|| is the Euclidean norm of the endmember p. For end element The Euclidean norm of p, which is the length; where the dot product is closer to 1, it indicates that the endmember p is closer to the endmember p. The more similar they are, the closer their dot products are. -1 indicates that the endmember p and the endmember p are similar. The less similar the dot product, the less likely it is that the endmember p is similar to the endmember 0. Orthogonal.

[0089] In the above-described network model training method for hyperspectral image enhancement, the present invention mainly designs two branches: a spectral correction branch and a spatial reconstruction branch. The spectral correction branch includes a spectral stitching module and a compression correction module, which are mainly used to correct the final output hyperspectral image using the spectral information of the original hyperspectral image, thus better avoiding spectral distortion. The spatial reconstruction branch uses a hierarchical structure and learnable weights to adaptively inject the extracted high-frequency information into the enhanced image, thereby solving the problems of existing technologies that ignore the differences in the required information for different bands and insufficient extraction of spatial details, thus obtaining richer spatial details and ensuring better spatial structure quality. At the same time, a hybrid loss function is adopted, and constraints are added to the reconstruction of hyperspectral and multispectral images to ensure the collaborative learning between spectral and spatial information, resulting in higher quality final images.

[0090] In this embodiment, a pair of multispectral and hyperspectral images from the same scene at the same angle are acquired as a sample in the test dataset to evaluate the performance of the network model. It should be noted that the training method for the test dataset is the same as that for the training dataset. Specifically, this invention uses multiple sets of image data to verify the effectiveness of the network model's training method. The image data comes from the CAVE and Houston datasets. The CAVE dataset contains images of various real-world materials and objects with controlled lighting in laboratory environments, with a spectral range of 400-700 nm and a spectral resolution of 10 nm. It consists of 32 pairs of hyperspectral and multispectral images; the hyperspectral images have a spatial resolution of 512×512 and contain 31 bands; the multispectral images have a spatial resolution of 512×512 and contain 3 bands. The Houston dataset has a spectral range of 380-1050 nm, a ground sampling distance of 1 m, contains 48 bands, and consists of 14 pairs of hyperspectral and multispectral images. After cropping, the spatial resolution of the hyperspectral images is 600×596, and the spatial resolution of the multispectral images is consistent with that of the hyperspectral images.

[0091] In the above, the method of this invention is mainly compared with traditional Naive Bayes models, matrix factorization-based methods (CNMF), hybrid method-based methods (GFPCA), detail-based hyperspectral image deep Laplacian fusion (DDLPS), blind unsupervised hyperspectral image fusion (GuideNet), source adaptive discriminant kernel-based image fusion (ADKNet), and high-resolution guided hyperspectral image fusion (BUSIF). The comparison process was implemented in Python 3.11 and trained and tested on an NVIDIA RTX 3090 GPU; an adaptive moment estimation algorithm was used for convergence during training. The parameter settings are as follows: the network learning rate is e^(-ε / ε). -4 The initial value of the learnable weights is e. -3 The number of iterations is 10,000, the learning rate decreases by 0.5 every 500 training iterations, and other parameters use default values.

[0092] In the above, after the experiment, the experimental results of the method proposed in this invention and other comparative methods were analyzed. This embodiment analyzes the performance of the experimental results through objective evaluation indicators and visual evaluation; among which, the main objective evaluation indicators include Peak Signal-to-Noise Ratio (PSNR) for evaluating the degree of distortion between the enhanced image and the original image, Spatial Correlation Coefficient (CC) for measuring the similarity between two images, Root Mean Square Error (RMSE) for evaluating the error between the reconstructed image and the original image, Relative Global Correction Error (ERGAS) and Spectral Angle Mapping (SAM) for measuring the global image quality difference, and spectral angle mapping between corresponding pixels. It should be noted that SAM is limited to evaluating the overall spectral distortion of the image and does not provide information about the distribution of spectral distortion. To overcome this limitation, this embodiment introduces Standard Deviation (STD) to evaluate the distribution of SAM values ​​on each pixel in the entire image, thereby enabling the evaluation of the main characteristic distribution of the influence of spectral distortion within the image.

[0093] Table 1:

[0094]

[0095] In this embodiment, Table 1 shows the objective index results obtained by training the proposed method of this invention on the CAVE dataset and comparing it with other comparative methods. Figure 6The distribution of SAM values ​​for a subset of data in the CAVE dataset is shown using different contrast methods. The last column shows relatively small variations in the distribution of SAM values. In the second row, there is significant spectral distortion at the flower stems in Figures (b)-(h), while the distortion in Figure (i) is less pronounced. In the third row, Figures (b)-(g) show a higher degree of spectral distortion in local details, while Figures (h) and (i) perform better in detail processing. In the fifth row, Figures (b)-(h) show significant distortion along the edges of the color swatches, while Figure (i) maintains lower SAM values ​​in these areas. In the sixth row, there is significant spectral distortion in the stamens of Figures (b)-(f), with severe distortion at the edges. Although Figures (g) and (h) in the sixth row show negligible spectral distortion in the details of the stamens, edge distortion is still present; in contrast, Figure (i) shows a low degree of distortion overall. Therefore, as shown in Table 1, the method of this invention achieves optimal performance across all metrics. Compared to BUSIF, which is superior among the remaining comparative methods, the experimental results of this invention show that PSNR is improved by 3.25%, SAM by 20.16%, CC by 0.13%, ERGAS by 17.88%, RMSE by 9.30%, and STD by 48.97%. Furthermore, the method of this invention maintains the lowest STD value. It should be noted that since STD represents the overall distribution of spectral distortion, this indicates that the network model of this invention not only reduces the degree of local spectral distortion when training image data, but also ensures a smoother distortion distribution and smaller edge distortion.

[0096] Table 2:

[0097]

[0098] In this embodiment, Table 2 shows the objective index results obtained by training the proposed method and other comparative methods on the Houston dataset, while also referring to... Figure 7 As can be seen, the spectral distortion in Figure (i) is significantly reduced, while Figures (b) and (d) exhibit severe local spectral distortion, especially at the edges. Although Figures (f) and (g) mostly show low spectral distortion, they still exhibit higher spectral distortion in specific areas (such as roofs of different colors) compared to Figure (i). In Table 2, except for CC, the method of this invention performs best on all indicators. Notably, in the experimental results of this invention, SAM outperforms the second-best comparative method by 19.72%. Overall, the above results demonstrate that the method proposed in this invention has superior performance.

Claims

1. A method for training a network model for hyperspectral image enhancement, characterized in that, The method includes the following steps: S1. Acquire multiple pairs of multispectral images with the same angle in multiple scenes. HR and hyperspectral images H LR To obtain the training dataset; S2, The hyperspectral images H in the training dataset LR And the hyperspectral image H LR The hyperspectral image H' obtained by upsampling HR The input is fed into the spectral correction branch of the network model to correct the hyperspectral images H in the training dataset. LR Correction is performed to obtain the corrected hyperspectral image. The spectral correction branch includes a spectral splicing module and a compression correction module. For input sample F x For x∈{H,L}, the input channels of the spectral correction branch are compressed using convolution kernels of arbitrary size, and then the input samples F are processed. x The data is input in several steps through the input channel of the spectral correction branch, and the sample data from these input steps are sequentially stitched together by the spectral stitching module to obtain the implicit features. The input sample F is calculated using the compression correction module squeze. H Channel attention vector V FH and input sample F L Channel attention vector V FL : The correction vector V is calculated through a correction layer consisting of a 1x1 convolution and a sigmoid function. r : The hyperspectral image H LR Spectral correction parameter F' x for: Through the spectral correction parameter F' x Correcting the hyperspectral image H LR The hyperspectral image is obtained. Multispectral image I HR The input is fed into the spatial reconstruction branch of the network model for decomposition and recombination to extract the multispectral image I. HR Spatial information in the image is used to obtain the reconstructed multispectral image. The multispectral image I is extracted using a high-pass filter. HR The high-frequency information in the image is used as prior information to compensate for the deficiencies in spatial information extraction. The spatial reconstruction branch uses the high-frequency information to guide the convolutional neural network to learn and extract multispectral images. HR detail; Among them, HF cnn The high-frequency information extracted by the high-pass filter, HF non The high-frequency information extracted by the high-pass filter is represented by G, where G is the high-pass filter and HF is the multispectral image I. HR High-frequency information, C(·) represents the convolution operation; Learn HF cnn and HF non Reconstruction yields multispectral images Where Rebuild(·) is the reconstruction layer, and the multispectral image Reconstruction was performed using a 3×3 convolution kernel; The extracted spatial information is injected into the spectral correction branch of the corresponding layer using a learnable weight, enabling it to adaptively inject spatial information into the hyperspectral image H'. HR In the process, an enhanced hyperspectral image was obtained. S3. Train the network model using a hybrid loss function until it converges.

2. The method for training a network model for hyperspectral image enhancement according to claim 1, characterized in that, In S1, the hyperspectral image H' HR With the multispectral image I HR They are the same size.

3. The method for training a network model for hyperspectral image enhancement according to claim 2, characterized in that, In step S1, the acquired hyperspectral image and multispectral image are downsampled by a factor of 4 according to the Wald protocol, and additive noise is added to the downsampled hyperspectral image to simulate a low-resolution hyperspectral image, thus obtaining the hyperspectral image H'. HR ; The acquired multispectral image was downsampled by 4 times according to the Wald protocol to obtain the multispectral image I. HR .

4. The method for training a network model for hyperspectral image enhancement according to claim 2, characterized in that, The upsampling method is as follows: A trainable upsampling module is selected and inserted into the network model. The downsampled hyperspectral image H is then processed by the trainable upsampling module. LR Perform a 4x upsampling.

5. The network model training method for hyperspectral image enhancement according to claim 4, characterized in that, The trainable upsampling module is a pixel-shuffle module.

6. The method for training a network model for hyperspectral image enhancement according to claim 1, characterized in that, In S2, if the network depth of the spectral correction branch is set to d, then the output result of the i-th layer of the spectral correction branch is: Where j = i-1, Lecel i Let res be the i-th layer of the spectral correction branch, spe(·) be the residual learning, and spe(·) be the spectral correction. For the hyperspectral image H' of layer j HR Features For the hyperspectral image H of layer j LR Features and These are the outputs of Spe(·), α i Let θ be the learnable weight of the i-th layer. i The parameters that the spectral correction branch of the i-th layer needs to learn; When i = 1, Lecel i As the first layer, the network model uses 3×3 convolutions to activate the hyperspectral image H. LR and hyperspectral image H' HR To output the corresponding and When i = d, Lecel i As the last layer, the network model uses 3×3 convolutions to respectively... and Reconstruction is performed to obtain a hyperspectral image. and hyperspectral images 7. The method for training a network model for hyperspectral image enhancement according to claim 1, characterized in that, In S2, for the input sample F x For x∈{H,L}, the input channel c of the spectral correction branch is compressed using a convolution kernel of arbitrary size. Then input sample F x Through the spectral correction branch The input channel receives data four times, and the spectral stitching module sequentially stitches together the sample data from the four inputs to obtain the implicit features. Among them, in obtaining implicit features When, input sample F H and input sample F L Parameters are shared between them, cat is a cascading operation, F ki Corresponding to the features of different convolution kernels, k1 represents a 1x1 convolution kernel, k2 represents a 3x3 convolution kernel, k3 represents a 5x5 convolution kernel, and k4 represents a 7x7 convolution kernel.

8. A network model training method for hyperspectral image enhancement according to claim 7, characterized in that, In S2, the network model will assign each layer corresponding to the spatial reconstruction branch to... Injected into the layer corresponding to the spectral reconstruction branch In the middle, then using a jump connection to obtain Where, α i Let be the learnable weight values ​​of the i-th layer in the spatial reconstruction branch.

9. The method for training a network model for hyperspectral image enhancement according to claim 1, characterized in that, In S3, the hybrid loss function includes the loss function Loss for constrained spatial information. Spa Loss function for constraining spectral information Spe The hybrid loss function Loss is: Loss=Loss m +λ1·Loss Spa +λ2·Loss Spe ; Loss m =||H' HR -H HR ||; Where λ1 and λ2 are hyperparameters, and Loss is... m For hyperspectral image H' HR and hyperspectral images The loss function between For H x The spectral vector represented in pixels, x∈{HR,LR}, Loss spe For multispectral images With multispectral image I HR The spectral error between them, where Cosine is the cosine similarity. For the i-th endmember in a high-resolution hyperspectral image, To reconstruct the i-th endmember in a high-resolution hyperspectral image, For the i-th endmember in a low-resolution hyperspectral image, To reconstruct the i-th endmember in a low-resolution hyperspectral image, For endmember p and endmember The dot product between p and p, and the range of the dot product is [-1, 1]; ||p|| is the Euclidean norm of the endmember p. For end element The Euclidean norm; where the closer the dot product is to 1, the closer it is to the endmember p and the endmember The more similar they are, the closer their dot products are. -1 indicates that the endmember p and the endmember p are similar. The less similar the dot product, the less likely it is that the endmember p is similar to the endmember 0. Orthogonal.

Images