Spectral super-resolution method for joint learning of spectral physical degradation and depth prior

Abstract

The invention discloses a spectral super-resolution method based on spectral physical degradation and depth prior combined learning. The method comprises the following steps: establishing a color image spectral super-resolution variation model based on depth prior regularization and spectral degradation physical model combined learning; optimizing the variation model by adopting an auxiliary variable and a variable separation algorithm to obtain an image super-resolution variation model from the color image to the hyperspectral image; performing network expansion on the iteration process of variation model optimization, and executing the optimized network iteration process; designing an adjacent operator for the hyperspectral image and the spectral response function of the potential space; and training the network by taking the L2 norm of the reconstruction error of the hyperspectral image and the consistency loss and observation loss of the physical degradation model as a joint loss function. The method has the capability of simultaneously representing the degradation model and the data prior of the hyperspectral image in the network, realizes the fusion of the local features and the non-local features of the space and the spectrum, and effectively enhances the fidelity capability of the spectrum and the spatial features.

Description

Technical Field

[0001] The present invention relates to spectral super-resolution technology, and in particular to a spectral super-resolution method for joint learning of spectral physical degradation and deep priori. Background Art

[0002] Hyperspectral images can capture a wide range of electromagnetic spectra and retain rich spectral information. However, the practical application of hyperspectral images is usually subject to various constraints. As the spectral resolution of hyperspectral images increases, the spatial resolution will inevitably decrease. In addition, the acquisition of hyperspectral images through various sensors requires a long exposure time and complex equipment configuration, which limits the feasibility of hyperspectral images in practical applications.

[0003] Early spectral super-resolution technology mainly used sparse coding or shallow learning models. When using sparse coding, it is necessary to select appropriate dictionaries and sparse representation methods. Different dictionaries and sparse representation methods may produce different results. In addition, sparse coding algorithms usually require a lot of calculations, which leads to high computational complexity and longer computing time. Shallow learning models may face overfitting problems, especially when the amount of data is insufficient or the feature dimension is high. In addition, hyperspectral data usually has high dimensionality and complex structural characteristics, while shallow learning models have limited expression capabilities and cannot capture the complex relationships in spectral data.

[0004] With the improvement of computer hardware performance, deep neural networks have taken a dominant position in many problems in the field of image vision. Deep neural networks have excellent performance in processing various image problems due to their highly abstract representation, complex modeling and good data adaptability. The feature extraction of image targets is also more comprehensive and effective, which can solve the problem that sparse coding methods and shallow learning models have weak feature extraction and expression capabilities. Therefore, more and more spectral super-resolution methods based on deep learning methods have been proposed, becoming the focus of attention in related fields in recent years. For example, Li et al. [J.Li, S.Du, R.Song, C.Wu, Y.Li, and Q.Du, "Hasic-net: Hybrid attentional convolutional neural network with structure information consistency for spectral super-resolution of color images," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.] proposed a network using residual networks and attention modules to improve the quality of reconstructed images by fusing features of different depths. However, if the hyperspectral data provided is not sufficient, the CNN-based method will also face the problem of overfitting; in addition, the CNN network is more focused on extracting local features and is lacking in extracting global features and modeling long-range dependencies.

[0005] As the application of Vision Transformer in the field of images expands, some researchers have tried to use Transformer for spectral super-resolution to make up for the disadvantages of deep neural networks in extracting global features. However, using multi-head self-attention in the spatial dimension to build a spectral super-resolution network will increase the computational complexity and require more training time. In contrast, multi-head self-attention in the spectral dimension does not require a lot of time to train. For example, Cai et al. [Y.Cai, J.Lin, Z.Lin, H.Wang, Y.Zhang, H.Pfister, R.Timofte, and LV an Gool, "Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp.745–755.] proposed a Transformer network framework that uses multi-head self-attention in the spectral dimension to extract global features of the image and complete spectral super-resolution of color images. In addition, the introduction of graph neural networks can not only reduce time complexity, but also has great advantages in extracting non-local features. However, how to fuse two different types of data is a difficulty that cannot be ignored. The deep learning-based method adapts to the input data through learning and adjusts its own parameters such as weights and biases, which lacks interpretability to a certain extent. By reading a lot of information, it can be found that previous spectral super-resolution algorithms usually use the implicit learning ability of deep learning networks to directly hide the solution process of the spectral response function (SRF), or directly use SRF as prior knowledge without additional optimization and making full use of SRF, which makes the optimization process of hyperspectral images unclear. Summary of the invention

[0006] The object of the present invention is to provide a spectral super-resolution method for joint learning of spectral physical degradation and deep prior.

[0007] The technical solution to achieve the purpose of the present invention is: a spectral super-resolution method of spectral physical degradation and deep prior joint learning, comprising:

[0008] The first step is to establish a variational model for color image spectral super-resolution by jointly learning the deep prior regularization and spectral degradation physical model. That is, according to the degradation model of hyperspectral images, the data fidelity term and regularization term are designed, and the objective function of color image spectral super-resolution with deep prior regularization is established under the variational model framework.

[0009] In the second step, auxiliary variables and variable separation algorithms are used to optimize the variational model to obtain an image super-resolution variational model from color images to hyperspectral images. That is, auxiliary variables and variable separation and alternating optimization methods are used to optimize the variational model in the first step to obtain an iterative solution process.

[0010] The third step is to expand the network of the iterative process of variational model optimization and perform the optimized network iterative process, that is, to design a color image super-resolution network and perform the iterative process of variational model optimization, in which both the data prior and the spectral degradation process of the hyperspectral spectrum can be represented in the network;

[0011] The fourth step is to design a neighbor operator for the hyperspectral image and spectral response function in the latent space, that is, to design a neighbor operator to solve the approximate solution of the hyperspectral image and spectral response function;

[0012] In the fifth step, the network is trained using the L2 norm of the reconstruction error of the hyperspectral image and the consistency loss and observation loss of the physical degradation model as the joint loss function.

[0013] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the computer program is loaded into the processor.

[0014] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0015] A computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0016] The method of the present invention has a simple structure. Compared with the prior art, its significant features are: (1) based on the variational regularization of the depth prior, a spectral super-resolution model of the color image is designed and networked, and the fusion network is interpretable; (2) the degradation model and the depth prior of the hyperspectral image are represented in the super-resolution network, and the data prior can be fully utilized by learning from the data through network training, and the image degradation process can be explicitly solved; (3) a unique neighboring operator is designed to make full use of the local and non-local features of the image prior, thereby improving the accuracy of the reconstructed image; (4) the loss function is improved by using the explicitly solved degradation matrix, thereby improving the training efficiency.

[0017] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural diagram of the method of the present invention.

[0019] Figure 2 It is a neighboring operator structure diagram of image depth prior in the method of the present invention.

[0020] Figure 3 This is the fusion result of different methods on the ICVL dataset, (a) HDnet, (b) HInet, (c) HRnet, (d) HSCNN+, (e) MST++, (f) Ours, (g) benchmark image.

[0021] Figure 4 yes Figure 3 Error graph of fusion results, (a) HDnet, (b) Hinet, (c) HRnet, (d) HSCNN+, (e) MST++, (f) Ours. DETAILED DESCRIPTION

[0022] The present invention proposes a spectral super-resolution method of jointly learning spectral physical degradation and deep prior. The method establishes a fusion model of deep prior regularization, learns the degradation model and deep prior of hyperspectral images in a large amount of data, mines the local and non-local features of the image space-spectral domain, reconstructs the hyperspectral image, and has excellent super-resolution performance.

[0023] Combination Figure 1 , the implementation process of the present invention is described in detail below, the steps are as follows:

[0024] The first step is to establish a variational model for color image spectral super-resolution that jointly learns the deep prior regularization and spectral degradation physical model. That is, according to the degradation model of the hyperspectral image, the data fidelity term and the regularization term are designed, and the objective function of color image spectral super-resolution with deep prior regularization is established under the variational model framework. Assume that the hyperspectral image is Where C, H, and W are the number of spectral bands, height, and width respectively; the color image is Where c, H, and W are the number, height, and width of its spectral bands respectively; the spectral response function Given the color image Y and the spectral response function R, the hyperspectral image X can be obtained from the following variational model:

[0025]

[0026] in, is the approximate solution of the hyperspectral image X in formula (1), ||.|| Frepresents the L2 norm, measuring data fidelity, J(·) is the regularization term, representing the prior knowledge of the hyperspectral image and spectral response function, and the parameters λ1 and λ2 are the weights of the regularization term.

[0027] In the second step, auxiliary variables and variable separation algorithms are used to optimize the variational model to obtain the image super-resolution variational model from color image to hyperspectral image. That is, the variational model in the first step is optimized by using auxiliary variables and variable separation and alternating optimization to obtain an iterative solution process. Specifically, auxiliary variables and variable separation algorithms are used to introduce auxiliary variables V and K. Formula (1) is converted to:

[0028]

[0029] Among them, μ1 and μ2 are the weights of the auxiliary variable penalty term, J deep (X) represents the depth prior of the hyperspectral image, J deep (R) represents the depth prior of the spectral response function. Using the alternating iterative algorithm, formula (2) can be converted into a sub-problem for iterative solution:

[0030]

[0031] In formula (4) The approximate solution of the spectral response function R is represented by the semi-quadratic splitting method, and then the X and R sub-problems are transformed, and the s-th iteration is:

[0032] 1) V- and K-subproblems

[0033]

[0034] Use the gradient descent method to get V s , K s

[0035]

[0036] Among them, V s and K s They represent the auxiliary variables of the approximate solution of the hyperspectral image and the approximate solution of the spectral response function in the iterative solution process, respectively. ε1 and ε2 are the step sizes of the gradient descent.

[0037] 2) X- and R-subproblems

[0038]

[0039] The solution of formula (9) (10) in the sth stage of the iterative solution process can be written as Respectively represent V s and K sThe neighboring operator at K s We use a multilayer perceptron (MLP) to solve the neighboring operator at V s We designed a graph self-attention model (Graph Transformer) to solve the neighboring operator.

[0040] The third step is to expand the network of the iterative process of variational model optimization and perform the optimized network iterative process, that is, to design a color image super-resolution network and perform the iterative process of variational model optimization, in which the data prior and the spectral degradation process of the hyperspectral spectrum can be represented in the network. Specifically, K in formula (7) s-1 and Respectively represent spectral transformation. In the network, the feature fusion in formula (7) and (8) is realized by matrix multiplication and element-by-element addition. The neighboring operator in formula (9) and (10) can be implemented by any denoising convolutional neural network.

[0041] The fourth step is to design the neighboring operator for the hyperspectral image and spectral response function in the latent space, that is, to design the neighboring operator to solve the approximate solution of the hyperspectral image and the spectral response function. Specifically, the neighboring operator in formula (9) uses a separately designed Graph Transformer, and the neighboring operator in formula (10) is implemented using MLP. The MLP consists of four fully connected layers and nonlinear activation functions. The input layer and the output layer have cC neurons, c represents the number of spectral bands of the color image, C represents the number of spectral bands of the hyperspectral image, and the two hidden layers have 32 neurons. The Graph Transformer is composed of a convolutional neural network (CNN) and a parallel Transformer branch and a graph convolutional neural network (GCN) in series. The CNN network consists of two convolutional layers and nonlinear activation functions, with a convolution kernel size of 1×1 and a feature number of C. The Transformer branch consists of two Transformers using skip connections. The Transformer consists of two layers of normalization, a self-attention calculation, and a feedforward network, where the attention calculation uses multi-head self-attention in the spectral dimension. Assume that the input features are Then there is

[0042] Q=W Q X,K=W K X,V=W V X (11)

[0043]

[0044] Output = W (Concat (v i ))+p(X) (13)

[0045] in Represents a token, represents the weight matrix, A i represents the attention matrix, d h represents the scaling factor of the attention score, v i represents the single-head feature output by the self-attention calculation in the multi-head self-attention mechanism, p(·) represents the position information embedding, Concat(·) represents the connection operation along the channel dimension, W in formula (13) represents the weight of the linear mapping processing of the Concat feature, and Output represents the output feature of the Transformer branch.

[0046] Assume that the input features of GCN Considering each pixel as a node, we have

[0047]

[0048] H=WX (15)

[0049]

[0050] H l =f((D l ) -(1 / 2) A l (D l ) -(1 / 2) H l-1 W l ) (17)

[0051] Output=Reshape(H) (18)

[0052] Among them, M i,j represents the adjacency relationship between nodes. represents the node matrix, represents the adjacency matrix of the lth layer, represents the weight matrix, D l Indicates A l The degree matrix of Transformer is represented by I, the identity matrix, ⊙ represents matrix multiplication, Reshape represents the dimensional transformation of feature H, changing H from C×HW to C×H×W, and the Output of formula (18) represents the output feature of the graph convolutional neural network GCN branch; the outputs of Transformer and GCN are connected along the channel dimension and passed through a convolutional layer to obtain an approximate solution of the hyperspectral image prior.

[0053] In the fifth step, the network is trained using the L2 norm of the reconstruction error of the hyperspectral image and the consistency loss and observation loss of the physical degradation model as the joint loss function. Specifically, the consistency loss represents the L2 loss between the degraded image of the reconstructed image and the color image, the observation loss represents the L2 loss between the degraded image of the reference image and the color image, and the physical degradation process is completed using the optimized spectral response function.

[0054]

[0055] in represents the i-th reconstructed hyperspectral image sample, represents the i-th benchmark hyperspectral image sample, N represents the total number of samples, represents a color image, represents the spectral degradation matrix, α and β represent the trade-off factors of the two loss terms. The ADAM optimization algorithm is used to optimize the loss function.

[0056] The effect of the present invention can be further illustrated by the following simulation experiment:

[0057] (1) Simulation conditions

[0058] The experimental data uses the ICVL dataset, which contains 201 sets of images, each of which contains 31 bands. The spectral range is 400-700nm and the spatial size is 1300×1392.

[0059] (2) Analysis of simulation experiment results

[0060] Table 1 shows the super-resolution performance of the method of the present invention and its comparative method in simulation experiments on ICVL and CAVE datasets. Figure 1 It is a structural diagram of the present invention; Figure 2 The structural diagram of the neighboring operator of the hyperspectral image, (a) neighboring operator structure, (b) Transformer structure diagram, (c) spectral-wise multi-head self-attention (SMSA), (d) feedforward neural network structure. Figure 3 This is the fusion result of different methods on the ICVL dataset, (a) HDnet, (b) HInet, (c) HRnet, (d) HSCNN, (e) MST++, (f) Ours, (g) benchmark image, Figure 4 yes Figure 3Error graph of fusion results, (a) HDnet, (b) HInet, (c) HRnet, (d) HSCNN, (e) MST++, (f) Ours. The darker the color of the error graph, the smaller the error. Intuitively, the image fusion result error of the method of the present invention is smaller. In order to further quantify the performance of the model, PSNR (peak signal-to-noise ratio), SAM (spectral angle), SSIM (structural similarity), ERGAS (relative error of overall dimension), CC (correlation coefficient) are used as image evaluation indicators. The results show that the method of the present invention exceeds the other five advanced methods in all indicators, indicating the effectiveness of the method.

[0061] Table 1

[0062]

[0063] From the experimental results, the present invention has achieved the best results among the comparison methods on the ICVL dataset. The experimental results show that the peak signal-to-noise ratio of the reconstructed high-resolution image can reach 46.3081dB on the ICVL dataset. The present method has excellent performance when applied to spectral super-resolution of color images. Figure 3 As shown, the fusion result of the present invention has higher fidelity than other methods. Figure 4 It shows that the reconstruction error of this method is also lower than that of other methods.

Claims

1. A spectral super-resolution method based on joint learning of spectral physical degradation and deep prior, characterized in that: The following steps are involved: The first step is to establish a variational model for color image spectral super-resolution by jointly learning the deep prior regularization and spectral degradation physical model. That is, according to the degradation model of hyperspectral images, the data fidelity term and regularization term are designed, and the objective function of color image spectral super-resolution with deep prior regularization is established under the variational model framework. In the second step, auxiliary variables and variable separation algorithms are used to optimize the variational model to obtain an image super-resolution variational model from color images to hyperspectral images. That is, auxiliary variables and variable separation and alternating optimization methods are used to optimize the variational model in the first step to obtain an iterative solution process. The third step is to expand the network of the iterative process of variational model optimization and perform the optimized network iterative process, that is, to design a color image super-resolution network and perform the iterative process of variational model optimization, in which both the data prior and the spectral degradation process of the hyperspectral spectrum can be represented in the network; The fourth step is to design a neighbor operator for the hyperspectral image and spectral response function in the latent space, that is, to design a neighbor operator to solve the approximate solution of the hyperspectral image and spectral response function; In the fifth step, the network is trained using the L2 norm of the reconstruction error of the hyperspectral image and the consistency loss and observation loss of the physical degradation model as the joint loss function.

2. The spectral super-resolution method of spectral physical degradation and deep prior joint learning according to claim 1 is characterized in that: The first step is to establish a variational model for color image spectral super-resolution by jointly learning the deep prior regularization and spectral degradation physical model. That is, according to the degradation model of the hyperspectral image, the data fidelity term and the regularization term are designed, and the objective function of the color image spectral super-resolution with deep prior regularization is established under the variational model framework. Assume that the hyperspectral image is Where C, H, and W are the number of spectral bands, height, and width respectively; the color image is Where c, H, and W are the number, height, and width of its spectral bands respectively; the spectral response function Given the color image Y and the spectral response function R, the hyperspectral image X is obtained from the following variational model: in The approximate solution of the hyperspectral image X in formula (1) measures the data fidelity, ||.|| F represents the L2 norm, J(·) is the regularization term, represents the prior knowledge of the hyperspectral image and the spectral response function, and the parameters λ1 and λ2 are the weights of the regularization term.

3. The spectral super-resolution method of spectral physical degradation and deep prior joint learning according to claim 2 is characterized in that: In the second step, auxiliary variables and variable separation algorithms are used to optimize the variational model to obtain the image super-resolution variational model from color image to hyperspectral image. That is, the variational model in the first step is optimized by using auxiliary variables and variable separation and alternating optimization to obtain an iterative solution process. Specifically, auxiliary variables and variable separation algorithms are used to introduce auxiliary variables V and K. Formula (1) is converted to: Among them, μ1 and μ2 are the weights of the auxiliary variable penalty term, J deep (X) represents the depth prior of the hyperspectral image, J deep (R) represents the depth prior of the spectral response function. Using the alternating iterative algorithm, formula (2) can be converted into an iterative solution of sub-problems: in represents the approximate solution of the spectral response function R; Using the semi-quadratic splitting method, the X and R sub-problems are transformed and the s-th iteration is: 1) V- and K-subproblems Use the gradient descent method to get V s , K s Among them, V s and K s They represent the auxiliary variables of the approximate solution of the hyperspectral image and the approximate solution of the spectral response function in the iterative solution process, respectively. ε1 and ε2 are the step sizes of the gradient descent. 2) X- and R-subproblems The solution of formula (9) (10) in the sth stage of the iterative solution process can be written as Respectively represent V s and K s The neighboring operator at .

4. The spectral super-resolution method of spectral physical degradation and deep prior joint learning according to claim 3 is characterized in that: The third step is to expand the network of the iterative process of variational model optimization and perform the optimized network iterative process, that is, to design a color image super-resolution network and perform the iterative process of variational model optimization, in which the data prior and the spectral degradation process of the hyperspectral spectrum are represented in the network; K in formula (7) s-1 and They represent spectral transformation respectively. In the network, the feature fusion in formulas (7) and (8) is realized by matrix multiplication and element-by-element addition. The neighboring operator in formulas (9) and (10) is realized by using a denoising convolutional neural network.

5. The spectral super-resolution method of spectral physical degradation and deep prior joint learning according to claim 4 is characterized in that: The fourth step is to design a neighboring operator for the hyperspectral image and spectral response function in the latent space, that is, to design a neighboring operator to solve the approximate solution of the hyperspectral image and spectral response function; Formula (9) The neighbor operator in is implemented using Graph Transformer, and the neighbor operator in formula (10) is implemented using MLP; The MLP consists of four fully connected layers and a nonlinear activation function. The input and output layers have cC neurons, where c represents the number of spectral bands of color images and C represents the number of spectral bands of hyperspectral images. The two hidden layers have 32 neurons. The Graph Transformer consists of a convolutional neural network and a parallel Transformer branch connected in series with a graph convolutional neural network. The CNN network consists of two convolutional layers and a nonlinear activation function, with a convolution kernel size of 1×1 and a feature number of C. The Transformer branch consists of two Transformers using skip connections; the Transformer consists of two layer normalizations, a self-attention calculation, and a feedforward network, where the attention calculation uses multi-head self-attention in the spectral dimension; assuming the input feature is Then there is Q=W Q X,K=W K X,V=W V X (11) Output=W(Concat(v i ))+p(X) (13) Among them, Q, K, Represents a token, W Q , W K , W V , represents the weight matrix, A i represents the attention matrix, d h represents the scaling factor of the attention score, v i represents the single-head feature output by the self-attention calculation in the multi-head self-attention mechanism, p(·) represents the position information embedding, Concat(·) represents the connection operation along the channel dimension, W in formula (13) represents the weight of the linear mapping processing of the Concat feature, and Output represents the output feature of the Transformer branch; Assuming the input features of GCN Considering each pixel as a node, we have H=WX (15) H l =f((D l ) -(1 / 2) A l (D l ) -(1 / 2) H l-1 W l ) (17) Output=Reshape(H) (18) Among them, M i,j represents the adjacency relationship between nodes. represents the node matrix, represents the adjacency matrix of the lth layer, W, represents the weight matrix, D l Indicates A l The degree matrix of I represents the identity matrix, ⊙ represents matrix multiplication, Reshape represents the dimension transformation of feature H, changing H from C×HW to C×H×W, and the Output of formula (18) represents the output feature of the graph convolutional neural network GCN branch; The outputs of Transformer and GCN are concatenated along the channel dimension and passed through a convolutional layer to obtain an approximate solution of the hyperspectral image prior.

6. The spectral super-resolution method of spectral physical degradation and deep prior joint learning according to claim 5, characterized in that: In the fifth step, the network is trained using the L2 norm of the reconstruction error of the hyperspectral image and the consistency loss and observation loss of the physical degradation model as the joint loss function; the consistency loss represents the L2 loss between the degraded image of the reconstructed image and the color image, and the observation loss represents the L2 loss between the degraded image of the reference image and the color image. The physical degradation process is completed using the optimized spectral response function: in represents the i-th reconstructed hyperspectral image sample, represents the i-th benchmark hyperspectral image sample, N represents the total number of samples, represents a color image, represents the spectral degradation matrix, α and β represent the trade-off factors of the two loss terms; the ADAM optimization algorithm is used to optimize the loss function.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images