Network model training method for hyperspectral image enhancement

Abstract

The invention discloses a network model training method for hyperspectral image enhancement. The method comprises the following steps: acquiring a training data set; inputting the hyperspectral image HLR in the training data set and the hyperspectral image H 'HR obtained by up-sampling into a spectral correction branch of the network model so as to correct the hyperspectral image HLR in the training data set, obtaining a corrected hyperspectral image # imgabs0 #, inputting the multispectral image IHR into a spatial reconstruction branch of the network model for decomposition and recombination, and obtaining the hyperspectral image HLR. Spatial information in a multispectral image IHR is extracted and is injected into a spectral correction branch by using a learnable weight, so that the spatial information is adaptively injected into a hyperspectral image H 'HR to obtain a hyperspectral image # imgabs1 #, and a mixed loss function is adopted to train a network model. According to the method, the problems that in the prior art, information needed by different wave bands is ignored to be different, and spatial detail extraction is insufficient are solved, so that richer spatial details are obtained, and good spatial structure quality is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and specifically to a network model training method for hyperspectral image enhancement, in particular to a hierarchical adaptive learning spectral correction and spatial reconstruction network training method for hyperspectral image enhancement. Background Art

[0002] Due to hardware limitations, hyperspectral images often cannot obtain high spatial resolution. Most methods improve the spatial resolution of hyperspectral images by improving the performance of acquisition equipment, but this is expensive. Therefore, in hyperspectral image enhancement, how to inject high-resolution spatial details while maintaining spectral information has always been a difficult problem. At present, deep learning has been widely used in various computer vision tasks. As the core method of deep learning, convolutional neural networks have the ability to understand image information. By carefully designing loss functions or feature extraction modules, convolutional neural networks can achieve ideal results in hyperspectral image enhancement tasks.

[0003] However, existing image processing methods still have defects: 1) Previous methods usually directly link multispectral images with hyperspectral images and train them as a whole, but there are obvious differences in the characteristics of the two. Training them as a whole will bring great obstacles to fusion reconstruction; 2) In most methods, the role of hyperspectral images is limited to serving as upsampled input data, which may distort the original complete spectral information due to spatial interpolation, further causing distortion of the reconstructed hyperspectral image. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a network model training method for hyperspectral image enhancement to solve the technical problems mentioned in the prior art.

[0005] A network model training method for hyperspectral image enhancement, the method comprising the following steps:

[0006] S1. Obtain multiple pairs of multispectral images of the same angle in multiple scenes I HR and the hyperspectral image H LR , to obtain the training data set;

[0007] S2, the hyperspectral image H in the training data set LR And the hyperspectral image H LR The hyperspectral image H' obtained by upsampling HR Input into the spectral correction branch of the network model to correct the hyperspectral image H in the training data set LR Correction is performed to obtain the corrected hyperspectral image

[0008] The multispectral image I HR Input into the spatial reconstruction branch of the network model for decomposition and reorganization to extract the multispectral image I HR The spatial information in the image is obtained by reconstructing the multispectral image.

[0009] A learnable weight is used to inject the extracted spatial information into the spectral correction branch of the corresponding layer, so that it can adaptively inject the spatial information into the hyperspectral image H' HR The enhanced hyperspectral image is obtained

[0010] S3. Use the mixed loss function to train the network model until convergence.

[0011] Optionally, in S1, the hyperspectral image H' HR With the multispectral image I HR The same size.

[0012] Optionally, the acquired hyperspectral image and multispectral image are respectively downsampled by 4 times according to the Wald protocol, and additive noise is added to the downsampled hyperspectral image to simulate a low-resolution hyperspectral image, so as to obtain the hyperspectral image H' HR ;

[0013] The acquired multispectral image is downsampled by 4 times according to the Wald protocol to obtain the multispectral image I HR .

[0014] Optionally, the upsampling method is:

[0015] A trainable upsampling module is selected and inserted into the network model, and the hyperspectral image H obtained by downsampling is processed by the trainable upsampling module. LR Upsample by a factor of 4.

[0016] Optionally, the trainable upsampling module is a Pixel-shuffle module.

[0017] Optionally, in S2, the network depth of the spectral correction branch is set to d, and the output result of the i-th layer of the spectral correction branch is:

[0018]

[0019] Where j = i-1, Lecel i is the i-th layer of the spectral correction branch, res is residual learning, Spe(·) is spectral correction, is the j-th layer hyperspectral image H' HR Features, is the j-th layer hyperspectral image H LR Features, and are the output results of Spe(·), α i is the learnable weight of the i-th layer, θ i The parameters that need to be learned for the spectrum correction branch of the i-th layer;

[0020] When i=1, Lecel i As the first layer, the network model uses 3×3 convolution to activate the hyperspectral image H LR and hyperspectral image H' HR , to output the corresponding and

[0021] When i = d, Lecel i As the last layer, the network model uses 3×3 convolution to and Reconstruct and obtain a hyperspectral image and hyperspectral images

[0022] Optionally, in S2, the spectrum correction branch has a spectrum splicing module and a compression correction module;

[0023] For the input sample F x ,x∈{H,L}, the input channel c of the spectral correction branch is compressed using a convolution kernel of any size as Then input sample F x Through the spectral correction branch The input channel is input four times, and the sample data of the four inputs are sequentially spliced ​​together through the spectrum splicing module to obtain the implicit feature

[0024]

[0025] Among them, in obtaining implicit features When inputting sample F H And the input sample F L Share parameters between them, cat is a cascade operation, F ki Corresponding to the features of different convolution kernels, k1 represents the convolution kernel of 1x1, k2 represents the convolution kernel of 3x3, k3 represents the convolution kernel of 5x5, and k4 represents the convolution kernel of 7x7;

[0026] The compression correction module calculates the input sample F H The channel attention vector V FH And the input sample F L The channel attention vector VFL :

[0027]

[0028] The correction vector V is calculated through the Correction layer consisting of 1x1 convolution + sigmoid function r :

[0029]

[0030] The hyperspectral image H LR The spectral correction parameter F' x for:

[0031]

[0032] The spectral correction parameter F' x Correct the hyperspectral image H LR , and obtain the hyperspectral image

[0033] Optionally, in S2, the multispectral image I is extracted by a high-pass filter. HR The high-frequency information in the image is used as prior information to make up for the deficiency of spatial information extraction. The spatial reconstruction branch uses the high-frequency information to guide the convolutional neural network to learn and extract the multispectral image I HR detail;

[0034]

[0035] Among them, HF cnn HF is the high-frequency information extracted by the high-pass filter. non is the non-high frequency information extracted by the high-pass filter, G is the high-pass filter, and HF is the multispectral image I HR The high-frequency information of , C(·) represents the convolution operation;

[0036] The HF you have learned cnn and HF non Reconstruct multispectral images

[0037]

[0038] Where Rebuild(·) is the reconstruction layer, the multispectral image A 3×3 convolution kernel is used for reconstruction.

[0039] Optionally, in S2, the network model converts the corresponding The spectral reconstruction branch is injected into the corresponding layer Then, we use skip connections to get

[0040]

[0041] Among them, α i is the learnable weight value of the i-th layer in the spatial reconstruction branch.

[0042] Optionally, in S3, the hybrid loss function includes a loss function Loss of constrained spatial information Spa And the loss function Loss of constrained spectral information Spe , the mixed loss function Loss is:

[0043] Loss=Loss m +λ1·Loss Spa +λ2·Loss Spe ;

[0044]

[0045] Loss m =||H' HR -H HR ||;

[0046]

[0047] Among them, λ1 and λ2 are hyperparameters, Loss m is the hyperspectral image H' HR and hyperspectral images The loss function between H x The spectral vector represented by pixels, x∈{HR,LR}, Loss spe For multispectral images With multispectral image I HR The spectral error between them, Cosine is the cosine similarity, is the i-th endmember in the high-resolution hyperspectral image, To reconstruct the i-th endmember in a high-resolution hyperspectral image, is the i-th end member in the low-resolution hyperspectral image, To reconstruct the i-th endmember in the low-resolution hyperspectral image, is the end member p and the end member The dot product between , and the dot product range is [-1, 1]; ||p|| is the Euclidean norm of the end member p, End member The Euclidean norm of ; where the dot product is closer to 1, the closer the end member p is to the end member The more similar, the closer the dot product is to -1, indicating that the end member p is The more dissimilar, the dot product is 0, which means that the end member p and the end member Orthogonal.

[0048] The beneficial effects that the present invention can produce include:

[0049] The network model training method for hyperspectral image enhancement provided by the present invention mainly designs two branches, namely a spectral correction branch and a spatial reconstruction branch, wherein the spectral correction branch includes a spectral splicing module and a compression correction module, which is mainly used to use the spectral information of the original hyperspectral image to correct the final output hyperspectral image to better avoid spectral distortion; the spatial reconstruction branch uses a hierarchical structure and learnable weights to adaptively inject the extracted high-frequency information into the enhanced image to solve the problem that the prior art ignores the differences in information required for different bands and insufficient extraction of spatial details, thereby obtaining richer spatial details and ensuring better spatial structure quality; at the same time, a mixed loss function is used to add constraints on the reconstruction of hyperspectral images and multispectral images to ensure the collaborative learning between spectral information and spatial information, so that the final image obtains higher quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A framework diagram of a network model training method for hyperspectral image enhancement according to the present invention;

[0051] Figure 2 A schematic diagram of the network structure of the spectrum correction branch of the present invention;

[0052] Figure 3 It is a schematic diagram of the structure of the spectrum splicing module of the spectrum correction branch of the present invention;

[0053] Figure 4 It is a schematic diagram of the structure of the compression correction module of the spectrum correction branch of the present invention;

[0054] Figure 5 A schematic diagram of the network structure of the space reconstruction branch of the present invention;

[0055] Figure 6 This is the SAM distribution result diagram of the present invention on the CAVE data set, and the color bar range is [0,0.2]; wherein, Figure (a) is a multispectral image, Figure (b) is the result image of the CNMF method, Figure (c) is the result image of the Bayesian method, Figure (d) is the result image of the GFPCA method, Figure (e) is the result image of the DDLPS method, Figure (f) is the result image of the GuideNet method, Figure (g) is the result image of the ADKNet method, Figure (h) is the result image of the BUSIF method, and Figure (i) is the network model output result image of the present invention.

[0056] Figure 7 This is the SAM distribution result diagram of the present invention on the Houston data set, and the color bar range is

[0057] [0,0.2]; wherein, Figure (a) is a multispectral image, Figure (b) is a result image of the CNMF method, Figure (c) is a result image of the Bayesian method, Figure (d) is a result image of the GFPCA method, Figure (e) is a result image of the DDLPS method, Figure (f) is a result image of the GuideNet method, Figure (g) is a result image of the ADKNet method, Figure (h) is a result image of the BUSIF method, and Figure (i) is an output result image of the network model of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1-Figure 5 As shown, the present invention provides a network model training method for hyperspectral image enhancement, the method comprising the following steps:

[0060] Step 1: Obtain multiple pairs of multispectral images of the same angle in multiple scenes I HR and the hyperspectral image H LR , to obtain the training data set.

[0061] In the above, the network model downsamples the acquired hyperspectral image and multispectral image respectively to obtain the multispectral image I HR and the hyperspectral image H LR ; and for the hyperspectral image H LR Upsample to get the hyperspectral image H' HR , hyperspectral image H' HR With multispectral image I HR Specifically, the downsampling method is: according to the Wald protocol, the acquired hyperspectral image and multispectral image are downsampled by 4 times, and additive noise is added to the downsampled hyperspectral image to simulate a low-resolution hyperspectral image, and the hyperspectral image H' is obtained. HR ; According to the Wald protocol, the acquired multispectral image is downsampled by 4 times to obtain the multispectral image I HR The upsampling method is as follows: select a trainable upsampling module (Pixel-shuffle module) and insert it into the network model, and use the trainable upsampling module to perform the downsampling on the hyperspectral image HLR Upsample by a factor of 4.

[0062] Step 2: The hyperspectral image H in the training dataset LR And for the hyperspectral image H LR The hyperspectral image H' obtained by upsampling HR Input into the spectral correction branch of the network model to correct the hyperspectral image H in the training dataset LR Correction is performed to obtain the corrected hyperspectral image The multispectral image I HR Input to the spatial reconstruction branch of the network model for decomposition and recombination to extract the multispectral image I HR The spatial information in the image is obtained by reconstructing the multispectral image. A learnable weight is used to inject the extracted spatial information into the spectral correction branch of the corresponding layer, so that it can adaptively inject the spatial information into the hyperspectral image H' HR The enhanced hyperspectral image is obtained

[0063] In this embodiment, the network depth of the spectral correction branch is set to d, and the output result of the i-th layer of the spectral correction branch is:

[0064]

[0065] Where j = i-1, Lecel i is the i-th layer of the spectral correction branch, res is residual learning, Spe(·) is spectral correction, is the j-th layer hyperspectral image H' HR Features, is the j-th layer hyperspectral image H LR Features, and are the output results of Spe(·), α i is the learnable weight of the i-th layer, θ i is the parameter that needs to be learned for the spectral correction branch of the i-th layer; when i = 1, Lecel i As the first layer, the network model uses 3×3 convolution to activate the hyperspectral image H LR and hyperspectral image H' HR , to output the corresponding and When i = d, Lecel i As the last layer, the network model uses 3×3 convolution to and Reconstruct and obtain a hyperspectral image and hyperspectral images

[0066] In the above, the spectrum correction branch has a spectrum splicing module and a compression correction module; for the input sample F x ,x∈{H,L}, the input channel c of the spectral correction branch is compressed using a convolution kernel of any size as Then input sample F x By spectral correction branch The input channel is input four times, and the sample data of the four inputs are spliced ​​together in sequence through the spectrum splicing module to obtain the implicit feature

[0067]

[0068] Among them, in obtaining implicit features When inputting sample F H And the input sample F L Share parameters between them, cat is a cascade operation, F ki Corresponding to the features of different convolution kernels, k1 represents the convolution kernel of 1x1, k2 represents the convolution kernel of 3x3, k3 represents the convolution kernel of 5x5, and k4 represents the convolution kernel of 7x7;

[0069] Then the input sample F is calculated by the compression correction module H The channel attention vector V FH And the input sample F L The channel attention vector V FL :

[0070]

[0071] The correction vector V is calculated through the Correction layer consisting of 1x1 convolution + sigmoid function r :

[0072]

[0073] Hyperspectral image H LR The spectral correction parameter F' x for:

[0074]

[0075] By spectral correction parameter F' x Correction of hyperspectral image H LR , and obtain the hyperspectral image

[0076] In this embodiment, the multispectral image I is extracted by a high-pass filter. HRThe high-frequency information in the image is used as prior information to make up for the lack of spatial information extraction. The spatial reconstruction branch uses high-frequency information to guide the convolutional neural network to learn and extract multispectral images. HR detail;

[0077]

[0078] Among them, HF cnn HF is the high-frequency information extracted by the high-pass filter. non is the non-high frequency information extracted by the high-pass filter, G is the high-pass filter, and HF is the multispectral image I HR high-frequency information, C(·) represents the convolution operation; the learned HF cnn and HF non Reconstruct multispectral images

[0079] I' HR =Rebuild(HF cnn +HF non );

[0080] Among them, Rebuild(·) is the reconstruction layer, multispectral image A 3×3 convolution kernel is used for reconstruction. In the above, the network model converts the corresponding layers of each layer in the spatial reconstruction branch into Spectral correction parameters injected into the corresponding layer of the spectral reconstruction branch Then, we use skip connections to get

[0081]

[0082] Among them, α i is the learnable weight value of the i-th layer in the spatial reconstruction branch.

[0083] Step 3: Use the mixed loss function to train the network model until convergence. The mixed loss function includes the loss function Loss of the constrained spatial information Spa And the loss function Loss of constrained spectral information Spe , the mixed loss function Loss is:

[0084] Loss=Loss m +λ1·Loss Spa +λ2·Loss Spe ;

[0085]

[0086] Loss m =||H' HR -H HR ||;

[0087]

[0088] Among them, λ1 and λ2 are hyperparameters, Loss m is the hyperspectral image H' HR and hyperspectral images The loss function between H x The spectral vector represented by pixels, x∈{HR,LR}, it should be noted that HR refers to high resolution; LR refers to low resolution; Loss spe For multispectral images With multispectral image I HR The spectral error between them, Cosine is the cosine similarity, is the i-th endmember in the high-resolution hyperspectral image, To reconstruct the i-th endmember in a high-resolution hyperspectral image, is the i-th end member in the low-resolution hyperspectral image, To reconstruct the i-th endmember in the low-resolution hyperspectral image, is the end member p and the end member The dot product between , and the dot product range is [-1, 1]; ||p|| is the Euclidean norm of the end member p, End member The Euclidean norm of , which is the length; the closer the dot product is to 1, the closer the end member p is to the end member The more similar, the closer the dot product is to -1, indicating that the end member p is The more dissimilar, the dot product is 0, which means that the end member p and the end member Orthogonal.

[0089] In the above, a network model training method for hyperspectral image enhancement of the present invention is mainly designed with two branches, namely, a spectral correction branch and a spatial reconstruction branch, wherein the spectral correction branch includes a spectral splicing module and a compression correction module, which is mainly used to utilize the spectral information of the original hyperspectral image to correct the final output hyperspectral image, so as to better avoid spectral distortion; the spatial reconstruction branch utilizes a hierarchical structure and learnable weights to adaptively inject the extracted high-frequency information into the enhanced image to solve the problem that the prior art ignores the differences in information required for different bands and insufficient extraction of spatial details, thereby obtaining richer spatial details and ensuring better spatial structure quality; at the same time, a hybrid loss function is adopted, and constraints on the reconstruction of hyperspectral images and multispectral images are added to ensure the collaborative learning between spectral information and spatial information, so that the final image obtains higher quality.

[0090] In this embodiment, a pair of multispectral images and hyperspectral images at the same angle in the same scene are obtained as a sample in the test data set to evaluate the performance of the network model. It should be noted that the training method of the test data set is the same as the training method of the training data set. Specifically, the present invention uses multiple sets of image data to verify the effectiveness of the training method of the network model, wherein the image data comes from the CAVE data set and the Houston data set; it should be noted that the images in the CAVE data set contain various real-world materials and objects with controlled lighting in a laboratory environment, with a spectral range of 400-700nm and a spectral resolution of 10nm, and are composed of 32 pairs of hyperspectral images-multispectral images; wherein the spatial resolution of the hyperspectral image is 512×512, including 31 bands; the spatial resolution of the multispectral image is 512×512, including 3 bands. The spectral range of the Houston dataset is 380-1050nm, the ground sampling distance is 1m, it contains 48 bands, and consists of 14 pairs of hyperspectral images and multispectral images; after cropping, the spatial resolution of the hyperspectral image is 600×596, and the spatial resolution of the multispectral image is consistent with that of the hyperspectral image.

[0091] In the above, the method of the present invention is mainly compared with the traditional naive Bayes model, a method based on matrix decomposition (CNMF), a method based on a hybrid method (GFPCA), a detail-based deep Laplace fusion method for hyperspectral images (DDLPS), a blind unsupervised hyperspectral image fusion method (GuideNet), an image fusion method based on a source adaptive discriminant kernel (ADKNet), and a hyperspectral image fusion method based on high-resolution guidance (BUSIF). The comparison process was implemented in the Python 3.11 environment, and was trained and tested on the NVIDIA RTX 3090 GPU; the adaptive moment estimation algorithm was used for convergence during the training process. The parameters are set as follows: the network learning rate is e -4 , the initial value of the learnable weight is e -3 The number of iterations is 10,000, the learning rate decays by 0.5 every 500 training times, and other parameters use the default values.

[0092] In the above, after the experiment, the experimental results of the method proposed in the present invention and other comparative methods are analyzed respectively. This embodiment analyzes the various performances of the experimental results through objective evaluation indicators and visual evaluation; wherein, the main objective evaluation indicators include the peak signal-to-noise ratio (PSNR) for evaluating the degree of distortion between the enhanced image and the original image, the spatial correlation coefficient (CC) for measuring the similarity between the two images, the root mean square error (RMSE) for evaluating the error between the reconstructed image and the original image, the relative global correction error (ERGAS) and the spectral angle mapping (SAM) for measuring the global image quality difference, and the spectral angle between the corresponding pixels. It should be noted that SAM is limited to evaluating the overall spectral distortion of the image, and does not provide information about the distribution of spectral distortion. In order to solve this limitation, this embodiment introduces the standard deviation (STD) to evaluate the distribution of SAM values ​​on each pixel in the entire image, so as to evaluate the main feature distribution affected by the spectral distortion in the image.

[0093] Table 1:

[0094]

[0095] In this embodiment, Table 1 shows the objective index results obtained by training the method proposed in the present invention and other comparative methods on the CAVE dataset, and Figure 6The distribution of SAM values ​​of some data in the CAVE dataset for different comparison methods is shown. Among them, the distribution of SAM values ​​in the last column shows relatively small changes. In the second row, there is obvious spectral distortion in the flower stems of Figures (b)-(h), while the distortion in Figure (i) is obviously less prominent. In the third row, Figures (b)-(g) show a higher degree of spectral distortion in local details, while Figures (h) and (i) have better performance in detail processing. In the fifth row, Figures (b)-(h) show obvious distortion along the edges of the color plate, while Figure (i) maintains a low SAM value in these areas. In the sixth row, Figures (b)-(f) have obvious spectral distortion in the stamens, and the edge parts are severely distorted. Although Figures (g) and (h) in the sixth row show negligible spectral distortion in the details of the stamens, edge distortion still exists. In contrast, Figure (i) shows a low degree of distortion overall. Therefore, from the evaluation results in Table 1, the method of the present invention has achieved the best performance in all indicators. Compared with the better BUSIF in the remaining comparison methods, in the experimental results of the present invention, PSNR is improved by 3.25%, SAM is improved by 20.16%, CC is improved by 0.13%, ERGAS is improved by 17.88%, RMSE is improved by 9.30%, and STD is improved by 48.97%. At the same time, in terms of STD, the method of the present invention maintains the lowest value. It should be noted that since STD represents the overall distribution of spectral distortion, it is shown that the network model of the present invention not only reduces the degree of local spectral distortion when training image data, but also ensures a smoother distortion distribution and smaller edge distortion.

[0096] Table 2:

[0097]

[0098] In this embodiment, Table 2 shows the objective index results obtained by training the method proposed in the present invention and other comparative methods on the Houston dataset. Figure 7 It can be seen that the spectral distortion in Figure (i) is significantly reduced, while Figures (b) and (d) show severe local spectral distortion, especially at the edges. Although most of Figures (f) and (g) show lower spectral distortion, they still show higher spectral distortion in specific areas (such as roofs of different colors) compared with Figure (i). In Table 2, except for CC, the method of the present invention performs best in all indicators. It is worth noting that in the experimental results of the present invention, SAM performs 19.72% higher than the second best comparison method. In general, the above results show that the method proposed in the present invention has better performance.

Claims

1. A network model training method for hyperspectral image enhancement, characterized in that: The method comprises the following steps: S1. Obtain multiple pairs of multispectral images of the same angle in multiple scenes I HR and the hyperspectral image H LR , to obtain the training data set; S2, the hyperspectral image H in the training data set LR And the hyperspectral image H LR The hyperspectral image H' obtained by upsampling HR Input into the spectral correction branch of the network model to correct the hyperspectral image H in the training data set LR Correction is performed to obtain the corrected hyperspectral image The multispectral image I HR Input into the spatial reconstruction branch of the network model for decomposition and reorganization to extract the multispectral image I HR The spatial information in the image is obtained by reconstructing the multispectral image. A learnable weight is used to inject the extracted spatial information into the spectral correction branch of the corresponding layer, so that it can adaptively inject the spatial information into the hyperspectral image H' HR The enhanced hyperspectral image is obtained S3. Use the mixed loss function to train the network model until convergence.

2. A network model training method for hyperspectral image enhancement according to claim 1, characterized in that: In S1, the hyperspectral image H' HR With the multispectral image I HR The same size.

3. A network model training method for hyperspectral image enhancement according to claim 2, characterized in that: In S1, the acquired hyperspectral image and multispectral image are respectively downsampled by 4 times according to the Wald protocol, and additive noise is added to the downsampled hyperspectral image to simulate a low-resolution hyperspectral image, thereby obtaining the hyperspectral image H' HR ; The acquired multispectral image is downsampled by 4 times according to the Wald protocol to obtain the multispectral image I HR .

4. A network model training method for hyperspectral image enhancement according to claim 2, characterized in that: The upsampling method is: A trainable upsampling module is selected and inserted into the network model, and the hyperspectral image H obtained by downsampling is processed by the trainable upsampling module. LR Upsample by a factor of 4.

5. A network model training method for hyperspectral image enhancement according to claim 4, characterized in that: The trainable upsampling module is a Pixel-shuffle module.

6. A network model training method for hyperspectral image enhancement according to claim 1, characterized in that: In S2, the network depth of the spectral correction branch is set to d, and the output result of the i-th layer of the spectral correction branch is: Where j = i-1, Lecel i is the i-th layer of the spectral correction branch, res is residual learning, Spe(·) is spectral correction, is the j-th layer hyperspectral image H' HR Features, is the j-th layer hyperspectral image H LR Features, and are the output results of Spe(·), α i is the learnable weight of the i-th layer, θ i The parameters that need to be learned for the spectrum correction branch of the i-th layer; When i=1, Lecel i As the first layer, the network model uses 3×3 convolution to activate the hyperspectral image H LR and hyperspectral image H' HR , to output the corresponding and When i = d, Lecel i As the last layer, the network model uses 3×3 convolution to and Reconstruct and obtain a hyperspectral image and hyperspectral images 7. A network model training method for hyperspectral image enhancement according to claim 6, characterized in that: In S2, the spectrum correction branch has a spectrum splicing module and a compression correction module; For the input sample F x ,x∈{H,L}, the input channel c of the spectral correction branch is compressed using a convolution kernel of any size as Then input sample F x Through the spectral correction branch The input channel is input four times, and the sample data of the four inputs are sequentially spliced ​​together through the spectrum splicing module to obtain the implicit feature Among them, in obtaining implicit features When inputting sample F H And the input sample F L Share parameters between them, cat is a cascade operation, F ki Corresponding to the features of different convolution kernels, k1 represents the convolution kernel of 1x1, k2 represents the convolution kernel of 3x3, k3 represents the convolution kernel of 5x5, and k4 represents the convolution kernel of 7x7; The compression correction module squeezes the input sample F H The channel attention vector V FH And the input sample F L The channel attention vector V FL : The correction vector V is calculated through the Correction layer consisting of 1x1 convolution + sigmoid function r : The hyperspectral image H LR The spectral correction parameter F' x for: The spectral correction parameter F' x Correct the hyperspectral image H LR , and obtain the hyperspectral image 8. A network model training method for hyperspectral image enhancement according to claim 7, characterized in that: In S2, the multispectral image I is extracted by a high-pass filter. HR The high-frequency information in the image is used as prior information to make up for the deficiency of spatial information extraction. The spatial reconstruction branch uses the high-frequency information to guide the convolutional neural network to learn and extract the multispectral image I HR detail; Among them, HF cnn HF is the high-frequency information extracted by the high-pass filter. non is the non-high frequency information extracted by the high-pass filter, G is the high-pass filter, and HF is the multispectral image I HR The high-frequency information of , C(·) represents the convolution operation; The HF you have learned cnn and HF non Reconstruct multispectral images Where Rebuild(·) is the reconstruction layer, the multispectral image A 3×3 convolution kernel is used for reconstruction.

9. A network model training method for hyperspectral image enhancement according to claim 8, characterized in that: In S2, the network model converts the corresponding The spectral reconstruction branch is injected into the corresponding layer Then, we use skip connections to get Among them, α i is the learnable weight value of the i-th layer in the spatial reconstruction branch.

10. A network model training method for hyperspectral image enhancement according to claim 1, characterized in that: In S3, the hybrid loss function includes a loss function Loss of constrained spatial information Spa And the loss function Loss of constrained spectral information Spe , the mixed loss function Loss is: Loss=Loss m +λ1·Loss Spa +λ2·Loss Spe ; Loss m =||H' HR -H HR ||; Among them, λ1 and λ2 are hyperparameters, Loss m is the hyperspectral image H' HR and hyperspectral images The loss function between H x The spectral vector represented by pixels, x∈{HR,LR}, Loss spe For multispectral images With multispectral image I HR The spectral error between them, Cosine is the cosine similarity, is the i-th endmember in the high-resolution hyperspectral image, To reconstruct the i-th endmember in a high-resolution hyperspectral image, is the i-th end member in the low-resolution hyperspectral image, To reconstruct the i-th endmember in the low-resolution hyperspectral image, is the end member p and the end member The dot product between , and the dot product range is [-1, 1]; ||p|| is the Euclidean norm of the end member p, End member The Euclidean norm of ; where the dot product is closer to 1, the closer the end member p is to the end member The more similar, the closer the dot product is to -1, indicating that the end member p is The more dissimilar, the dot product is 0, which means that the end member p and the end member Orthogonal.

Images