Hyperspectral image super-resolution reconstruction method

Abstract

The invention provides a hyperspectral image super-resolution reconstruction method, which is characterized in that a super-resolution reconstruction model comprises a tensor construction module used for generating a high-resolution multispectral image tensor and a low-resolution hyperspectral image tensor; the gating cross-keeping sharing module is used for obtaining a first output corresponding to the low-resolution hyperspectral image tensor and a second output corresponding to the high-resolution multispectral image tensor according to the low-resolution hyperspectral image tensor and the high-resolution multispectral image tensor; the hyperspectral channel attention module is used for performing feature extraction on the first output to obtain a channel attention feature map; the multispectral spatial attention module is used for performing feature extraction on the second output to obtain a spatial attention feature map; and the fusion module is used for performing multi-level feature fusion on the channel attention feature map and the space attention feature map to obtain a high-resolution hyperspectral image. In a word, the method can generate a high-precision high-resolution hyperspectral image.

Description

Technical Field

[0001] The invention belongs to the field of hyperspectral image super-resolution, and in particular relates to a hyperspectral image super-resolution reconstruction method. Background Art

[0002] Hyperspectral image super-resolution is a method of reconstructing low spatial resolution hyperspectral images into high spatial-spectral resolution images. Hyperspectral super-resolution methods are mainly divided into two categories: single hyperspectral image super-resolution and image fusion-based super-resolution. The training images of the single hyperspectral image super-resolution method are derived from a single sensor and cannot achieve a more comprehensive perception of the scene. The training images of the image fusion-based super-resolution method come from two or more sensors imaging the same scene, and can achieve high-precision super-resolution reconstruction of hyperspectral images through information complementarity. Therefore, the fusion-based method has greater potential in improving the spatial-spectral resolution of hyperspectral images. The fusion-based methods mainly include traditional machine learning-based methods and deep learning-based methods.

[0003] Hyperspectral super-resolution fusion methods based on traditional machine learning mainly include the extension of pan-sharpening technology, spatial unmixing method, Bayesian sparse learning, matrix decomposition method and tensor decomposition method, etc. Although the methods based on traditional machine learning are generally easy to operate, they rely on manual priors and are limited in improving the accuracy of spatial resolution.

[0004] Hyperspectral super-resolution fusion methods based on deep learning are mainly divided into two categories: supervised and unsupervised.

[0005] In practice, supervised methods rely on labeled hyperspectral image datasets to achieve high-precision reconstruction effects, but it is often difficult to collect a large number of high-resolution annotated and registered hyperspectral images.

[0006] Unsupervised hyperspectral image super-resolution methods can achieve good training results without relying on a large number of data sets, but there are still the following problems: On the one hand, the encoder used to reconstruct the image fails to effectively use the spatial spectral information of hyperspectral and multispectral images for feature complementarity to improve the fusion accuracy. On the other hand, the decoder lacks enhancement of different modal features. In addition, the network used to estimate the degradation parameters is not accurate enough. Summary of the invention

[0007] The present invention is made to solve the above-mentioned problem, and aims to provide a hyperspectral image super-resolution reconstruction method.

[0008] The present invention provides a hyperspectral image super-resolution reconstruction method, which is used for obtaining a high-resolution hyperspectral image corresponding to a low-resolution hyperspectral image according to a low-resolution hyperspectral image and a high-resolution multispectral image of the same scene, and has the following characteristics, comprising the following steps: step S1, constructing a training data set according to existing image data; step S2, calculating a first parameter and a second parameter according to the training data set by an implicit characterization blind estimation degradation network; step S3, constructing a super-resolution reconstruction model, and training the super-resolution reconstruction model according to the training data set, the first parameter and the second parameter to obtain a trained super-resolution reconstruction model; step S4, inputting the low-resolution hyperspectral image and the high-resolution multispectral image into the trained super-resolution reconstruction model to obtain a high-resolution hyperspectral image, wherein The super-resolution reconstruction model includes: a tensor construction module, which is used to generate a high-resolution multispectral image tensor according to the high-resolution multispectral image, and to generate a low-resolution hyperspectral image tensor according to the low-resolution hyperspectral image; a gated cross-hold sharing module, which is used to obtain a first output corresponding to the low-resolution hyperspectral image tensor and a second output corresponding to the high-resolution multispectral image tensor according to the low-resolution hyperspectral image tensor and the high-resolution multispectral image tensor; a hyperspectral channel attention module, which is used to extract features from the first output to obtain a channel attention feature map; a multispectral spatial attention module, which is used to extract features from the second output to obtain a spatial attention feature map; a fusion module, which is used to perform multi-level feature fusion on the channel attention feature map and the spatial attention feature map to obtain a high-resolution hyperspectral image.

[0009] In the hyperspectral image super-resolution reconstruction method provided by the present invention, it can also have the following characteristics: wherein, the gated cross-hold sharing module includes: a gated cross-hold unit, which is used to obtain an initial first output and an initial second output according to a low-resolution hyperspectral image tensor and a high-resolution multispectral image tensor; a first feedforward unit, which is used to perform layer normalization, full connection processing and activation function processing on the initial first output to obtain a first output; and a second feedforward unit, which is used to perform layer normalization, full connection layer processing and activation function processing on the initial second output to obtain a second output.

[0010] The hyperspectral image super-resolution reconstruction method provided by the present invention may also have the following characteristics: wherein, in the gated cross-hold unit, the expression for generating the initial first output is: EY =Embedding(Y),F LY =LN(F EY ), Q Y =F LY W Q D, F EZ =Embedding(Z),F LZ =LN(FEZ ), V Z =F LZ ·W v , F WY =(Q Y K Z T ☉D)V Z , F GY =GroupNorm(F WY )F RY =Reshape(F GY ), F SY =(swish(F RY W G )☉F EY )W O , where Y is the low-resolution hyperspectral image tensor, Embedding is the Embedding dimension reduction operation, LN is the layer normalization operation, and W Q , W K and W v are learnable parameters, D is a diagonal matrix with the same dimension as the low-resolution hyperspectral image tensor, Z is the high-resolution multispectral image tensor, is the conjugate tensor of the diagonal matrix D, T is the transpose, GroupNorm is the normalization operation, Reshape is the reshaping operation, W G and W O is a learnable parameter, swish is the swish activation function, F SY is the initial first output. In the gated cross-hold unit, the expression for generating the initial second output is: EZ =Embedding(Z),F LZ =LN(F EZ ), Q Z =F LZ W Q D, F EY =Embedding(Y),F LY =LN(F EY ), V Y =F LY ·W v , F WZ =(Q Z K Y T ☉D)V Y , F GZ =GroupNorm(F WZ ), F RZ =Reshape(F GZ ), FSZ =(swish(F RZ W G )☉F EZ )W O , where W Q , W K and W v are learnable parameters, D is a diagonal matrix with the same dimensions as the low-resolution hyperspectral image tensor, is the conjugate tensor of the diagonal matrix D, W G and W O is a learnable parameter, F SZ is the initial second output.

[0011] The hyperspectral image super-resolution reconstruction method provided by the present invention may also have the following characteristics: in the first feed-forward unit, the initial first output is sequentially subjected to layer normalization, first full connection processing, activation function processing, and second full connection processing, and the corresponding output is residually connected with the initial first output to obtain the first output; in the second feed-forward unit, the initial second output is sequentially subjected to layer normalization, first full connection processing, activation function processing, and second full connection processing, and the corresponding output is residually connected with the initial second output to obtain the second output.

[0012] The hyperspectral image super-resolution reconstruction method provided by the present invention may also have the following characteristics: wherein, in the hyperspectral channel attention module, the expression for generating the channel attention feature map is: Y1 =LeakyRelu(FullyConnected(Y)),F Y2 =LeakyRelu(FullyConnected(F Y2 )),W Y1 =GlobalAvgPool(F Y2 )+GlobalMaxPool(F Y2 ), W Y2 =sigmoid(W Y1 ), F1′=W Y2 *F Y2 , where Y′ is the first output, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, GlobalAvgPool is the global average pooling operation, GlobalMaxPool is the maximum pooling operation, sigmoid is the sigmoid activation function processing, and F1' is the channel attention feature map.

[0013] The hyperspectral image super-resolution reconstruction method provided by the present invention may also have the following characteristics: wherein, in the multispectral spatial attention module, the expression for generating the spatial attention feature map is: Z =LeakyRelu(FullyConnected(Z')),W Z1 =cat(mean(F Z ), max(F Z )),W Z2 =sigmoid(Conv(W Z1 ), F2′=W Z2 *F Z , where Z′ is the second output, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, mean is the average operation, max is the maximum operation, cat is the concatenation operation, sigmoid is the sigmoid activation function processing, and F2' is the spatial attention feature map.

[0014] The hyperspectral image super-resolution reconstruction method provided by the present invention may also have the following characteristics: wherein, in the fusion module, the expression for generating a high-resolution hyperspectral image is: E1 =concat(F1′, F2′), F E2 =LeakyRelu(FullyConnected(F E1 )), F E3 =concat(F E2 , F1′, F2), F E =LeakyRelu(FullyConnected(F3),F H =Conv(F E ', C), F H '=Clamp(F H ), where F1' is the channel attention feature map, F2' is the spatial attention feature map, concat is the dimension concatenation operation, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, C is the learnable matrix, Conv is the two-dimensional convolution operation, Clamp is the clamping function Clamp processing, F H ′ is a high-resolution hyperspectral image.

[0015] In the hyperspectral image super-resolution reconstruction method provided by the present invention, it can also have the following characteristics: wherein, in step S2, the implicit characterization blind estimation degradation network includes: a position encoding module, which is used to generate a one-dimensional position encoding and a two-dimensional position encoding according to the number of channels of the low-resolution hyperspectral image and the high-resolution multispectral image; a multi-layer perception module, including a multi-layer perceptron M1 and a multi-layer perceptron M2, which is used to generate an initial first parameter according to the one-dimensional position encoding, and to generate an initial second parameter according to the two-dimensional position encoding; a loss module, including a first loss function, which is used to calculate a first loss through a first loss function according to the low-resolution hyperspectral training images and the high-resolution multispectral training images in the training data set, as well as the initial first parameter and the initial second parameter; an iterative optimization module, which stores a preset termination condition, and is used to optimize the parameters of the position encoding module, the multi-layer perceptron M1 module, and the multi-layer perceptron M2 module according to the first loss until the preset termination condition is reached, the first parameter is the initial first parameter generated by the optimized multi-layer perceptron M1 module, and the second parameter is the initial second parameter generated by the optimized multi-layer perceptron M2 module.

[0016] The hyperspectral image super-resolution reconstruction method provided by the present invention may also have the following characteristics: wherein, in the position coding module, the expression for generating the one-dimensional position coding and the two-dimensional position coding is: PE 1d =[PE1, PE2, A Y ], PE3=sin(2 L ·Π·A U ), PE4=cos((2 L ·Π·A V ), PE 2d =[PE3,PE4,A U ,,A V ], where A Y The inspace function generates a uniform distribution in [0, N Y -1], N Y is the number of channels of the low-resolution hyperspectral image, Π is the circumference, N Zi is the coding level index, N Z is the number of channels of the high-resolution multispectral image, sin() is the sine function, cos() is the cosine function, PE 1d is a one-dimensional position encoding, A U A is an array uniformly distributed in [0, U-1] generated by the inspace function. Vis an array uniformly distributed in [0, V-1] generated by the inspace function, U and V are the width and height of the spatial blur kernel, L is the number of encoding layers, and PE 2d For two-dimensional position encoding, in the multi-layer perception module, the expression for generating the initial first parameter is: Where SRF is the initial first parameter, LeakyReLU is the LeakyRelu activation function, M is the number of nodes in the current layer of the multilayer perceptron M1, and W jk is the weight of neuron k in the jth layer of the multilayer perceptron M1, For the one-dimensional position encoding of neuron k in the jth layer of the multilayer perceptron M1, the expression for generating the initial second parameter in the multilayer perceptron module is: Where PSF is the initial second parameter, N is the number of nodes in the current layer of the multilayer perceptron M2, and W jk is the weight of neuron k in the i-th layer of the multilayer perceptron M2, For the two-dimensional position encoding of neuron k in the i-th layer of the multilayer perceptron M2, the expression of the first loss function is: Loss'=||M'-H'|| 1,1 , H'≈S·HSI, M'≈MSI·PD, where Loss' is the first loss, ||1,1 is regularization, HSI is the low-resolution hyperspectral training image, MSI is the high-resolution multispectral training image, S is the initial first parameter, P is the initial second parameter, and D is the spatial downsampling.

[0017] The hyperspectral image super-resolution reconstruction method provided by the present invention may also have the following characteristics: wherein, in step S3, the parameters of the super-resolution reconstruction model are iteratively optimized by a second loss function, and the expression of the second loss function is: Loss = ||Y-HSI'|| 1,1 +γ||Z-MSI'|| 1,1 , HSI'=XPD, MSI'=SX, where Y is the low-resolution hyperspectral training image in the training dataset, Z is the high-resolution multispectral training image in the training dataset, X is the output generated by the super-resolution reconstruction model based on the low-resolution hyperspectral training image Y and the high-resolution multispectral training image Z, S is the initial first parameter, P is the initial second parameter, and D is the spatial downsampling.

[0018] Functions and Effects of the Invention

[0019] According to the hyperspectral image super-resolution reconstruction method involved in the present invention, first, the shared structure between the two inputs is effectively learned through the gated cross-maintaining sharing module to achieve information complementarity; second, through the hyperspectral channel attention module and the multi-spectral spatial attention module, channel attention and spatial attention are used for feature processing, and the spatial spectral resolution is further improved through the information fusion of the fusion module; third, the accuracy of the first parameter and the second parameter is improved by implicitly characterizing the blind estimation degradation network, and then the reconstruction accuracy of the image is further improved in combination with the second loss function. Therefore, the hyperspectral image super-resolution reconstruction method of the present invention can generate high-precision high-resolution hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic flow chart of a method for super-resolution reconstruction of a hyperspectral image in an embodiment of the present invention;

[0021] Figure 2 is a working block diagram of an implicitly characterized blind estimation degradation network in an embodiment of the present invention;

[0022] Figure 3 is a block diagram of a super-resolution reconstruction model in an embodiment of the present invention;

[0023] Figure 4 is a working block diagram of a gated cross-hold sharing module in an embodiment of the present invention;

[0024] Figure 5 It is a working block diagram of the hyperspectral channel attention module, the multispectral spatial attention module and the fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments and accompanying drawings specifically illustrate the hyperspectral image super-resolution reconstruction method of the present invention.

[0026] In this embodiment, a hyperspectral image super-resolution reconstruction method is provided, which is used to obtain a high-resolution hyperspectral image corresponding to a low-resolution hyperspectral image based on a low-resolution hyperspectral image and a high-resolution multispectral image of the same scene. In this embodiment, the high-resolution multispectral image is a plurality of high-resolution spectral images corresponding to different bands, the low-resolution hyperspectral image has rich spectral information, and the high-resolution spectral image has rich spatial information.

[0027] Figure 1 It is a schematic diagram of the process of the hyperspectral image super-resolution reconstruction method in an embodiment of the present invention.

[0028] like Figure 1 As shown, the hyperspectral image super-resolution reconstruction method includes the following steps:

[0029] Step S1, constructing a training data set based on existing image data.

[0030] Step S2, according to the training data set, by implicitly characterizing the blind estimation of the degradation network, the first parameter and the second parameter are calculated.

[0031] Figure 2 It is a working block diagram of the implicit characterization blind estimation degradation network in an embodiment of the present invention.

[0032] like Figure 2 As shown, the implicit representation blind estimation degradation network 100 includes a position encoding module 11, a multi-layer perception module 12, a loss module 13 and an iterative optimization module (not shown in the figure).

[0033] The position coding module 11 is used to generate one-dimensional position coding and two-dimensional position coding according to the number of channels of the low-resolution hyperspectral image and the high-resolution multispectral image.

[0034] Among them, the expression for generating one-dimensional position coding and two-dimensional position coding is:

[0035]

[0036] PE 1d =[PE1, PE2, A Y ],

[0037]

[0038] PE3=sin(2 L ·Π·A U ),

[0039] PE4=cos(2 L ·Π·A V ),

[0040] PE 2d =[PE3, PE4, A U ,A V ],

[0041] Where A Y The inspace function generates a uniform distribution in [0, N Y -1], N Y is the number of channels of the low-resolution hyperspectral image, Π is the circumference, N Zi is the coding level index, N Z is the number of channels of the high-resolution multispectral image, sin() is the sine function, cos() is the cosine function, PE 1d is a one-dimensional position encoding, A UA is an array uniformly distributed in [0, U-1] generated by the inspace function. V is an array uniformly distributed in [0, V-1] generated by the inspace function, U and V are the width and height of the spatial blur kernel, L is the number of encoding layers, and PE 2d Two-dimensional position encoding.

[0042] In this embodiment, PE1, PE2, PE3 and PE4 are position features of different scales captured by sine function and cosine function respectively. The size of the spatial blur kernel in this embodiment is UxVx1, and the number of coding layers L=1.

[0043] The multi-layer perception module 12 includes a multi-layer perceptron M1, i.e., a multi-layer perceptron M ψ and multilayer perceptron M2, i.e., multilayer perceptron M φ , used to generate an initial first parameter according to the one-dimensional position encoding and the multilayer perceptron M1, and to generate an initial second parameter according to the two-dimensional position encoding and the multilayer perceptron M2. In this embodiment, the multilayer perceptron M1 is composed of five layers of linear full connection, and the multilayer perceptron M2 is composed of five layers of two-dimensional convolutional full connection.

[0044] Among them, the expression for generating the initial first parameter is:

[0045]

[0046] Where SRF is the initial first parameter, LeakyReLU is the LeakyRelu activation function, M is the number of nodes in the current layer of the multilayer perceptron M1, and W jk is the weight of neuron k in the jth layer of the multilayer perceptron M1, is the one-dimensional position code of neuron k in the jth layer of the multilayer perceptron M1.

[0047] The expression for generating the initial second parameter is:

[0048]

[0049] Where PSF is the initial second parameter, N is the number of nodes in the current layer of the multilayer perceptron M2, and W jk is the weight of neuron k in the i-th layer of the multilayer perceptron M2, is the two-dimensional position encoding of neuron k in the i-th layer of the multilayer perceptron M2.

[0050] The loss module 13 includes a first loss function, which is used to calculate a first loss through the first loss function according to the low-resolution hyperspectral training images and the high-resolution multispectral training images in the training data set, as well as the initial first parameters and the initial second parameters.

[0051] Among them, the expression of the first loss function is:

[0052] Loss'=||M'-H'|| 1,1

[0053] H'≈S·HSI,

[0054] M'≈MSI·PD,

[0055] Where Loss' is the first loss, ||1,1 is regularization, HSI is the low-resolution hyperspectral training image, MSI is the high-resolution multispectral training image, S is the initial first parameter, P is the initial second parameter, and D is the spatial downsampling.

[0056] The iterative optimization module stores a preset termination condition, which is used to optimize the parameters of the position encoding module, the multilayer perceptron M1 module, and the multilayer perceptron M2 module according to the first loss until the preset termination condition is reached. In this embodiment, the preset termination condition is that the number of iterations reaches a preset number of times, and the preset number of times in this embodiment is 30,000 times.

[0057] The first parameter is an initial first parameter generated by the optimized multilayer perceptron M1 module, and the second parameter is an initial second parameter generated by the optimized multilayer perceptron M2 module.

[0058] Step S3, constructing a super-resolution reconstruction model, and training the super-resolution reconstruction model according to the training data set, the first parameter and the second parameter to obtain a trained super-resolution reconstruction model.

[0059] In step S3, the parameters of the super-resolution reconstruction model are iteratively optimized by a second loss function, and the expression of the second loss function is:

[0060] Loss=||Y-HSI'|| 1，1 +γ||Z-MSI'|| 1,1 ,

[0061] HSI'=XPD,

[0062] MSI' = SX,

[0063] Where Y is the low-resolution hyperspectral training image in the training dataset, Z is the high-resolution multispectral training image in the training dataset, X is the output generated by the super-resolution reconstruction model based on the low-resolution hyperspectral training image Y and the high-resolution multispectral training image Z, S is the initial first parameter, P is the initial second parameter, and D is the spatial downsampling.

[0064] Figure 3 is a block diagram of a super-resolution reconstruction model in an embodiment of the present invention.

[0065] like Figure 3 As shown, the super-resolution reconstruction model 200 includes a tensor construction module 21, a gated cross-preserving sharing module 22, a hyperspectral channel attention module 23, a multispectral spatial attention module 24 and a fusion module 25.

[0066] The tensor construction module 21 is used to generate a high-resolution multispectral image tensor according to the high-resolution multispectral image, and to generate a low-resolution hyperspectral image tensor according to the low-resolution hyperspectral image.

[0067] The gated cross-hold sharing module 22 is used to obtain a first output corresponding to the low-resolution hyperspectral image tensor and a second output corresponding to the high-resolution multispectral image tensor according to the low-resolution hyperspectral image tensor and the high-resolution multispectral image tensor.

[0068] Figure 4 It is a working block diagram of the gated cross-hold sharing module in an embodiment of the present invention.

[0069] like Figure 4 As shown, the gated cross-holding sharing module 22 includes a gated cross-holding unit 221 , a first feed-forward unit 222 and a second feed-forward unit 223 .

[0070] The gated cross-hold unit 221 is used to obtain an initial first output and an initial second output according to the low-resolution hyperspectral image tensor and the high-resolution multispectral image tensor.

[0071] Among them, the expression for generating the initial first output is:

[0072] F EY =Embedding(Y),

[0073] F LY =LN(F EY ),

[0074] Q Y =F LY W Q D,

[0075] F EZ =Embedding(Z),

[0076] F LZ =LN(F EZ ),

[0077]

[0078] V Z =F LZ ·W v ,

[0079] F WY =((Q Y K Z T ☉D)V Z ,

[0080] F GY =GroupNorm(F WY ),

[0081] F RY =RReshape(F GY ),

[0082] F SY =(swish(F RY W G )☉F EY )W O ,

[0083] Where Y is the low-resolution hyperspectral image tensor, Embedding is the Embedding dimension reduction operation, LN is the layer normalization operation, and W Q , W K and W v are learnable parameters, D is a diagonal matrix with the same dimension as the low-resolution hyperspectral image tensor, Z is the high-resolution multispectral image tensor, is the conjugate tensor of the diagonal matrix D, T is the transpose, GroupNorm is the normalization operation, Reshape is the reshaping operation, W G and W O is a learnable parameter, swish is the swish activation function, F SY is the initial first output.

[0084] The expression that generates the initial second output is:

[0085] F EZ =Embedding(Z),

[0086] F LZ =LN(F EZ ),

[0087] Q Z =F LZ W Q D,

[0088] F EY =Embedding(Y),

[0089] F LY =LN(F EY ),

[0090]

[0091] V Y =F LY ·W v ,

[0092] F WZ =(Q Z K Y T ☉D)V Y ,

[0093] F GZ =GroupNorm(F WZ )

[0094] F RZ =Reshape(F GZ ),

[0095] F SZ =(swish(F RZ W G )☉F EZ )W O ,

[0096] Where W Q , W K and W v are learnable parameters, D is a diagonal matrix with the same dimensions as the low-resolution hyperspectral image tensor, is the conjugate tensor of the diagonal matrix D, W G and W O is a learnable parameter, F SZ is the initial second output.

[0097] In this embodiment, W Q , W K and W v They are used to compute representations of the query Q, key K, and value V, respectively, which are used to compute the attention weights and ultimately generate the self-attention output.

[0098] In this embodiment, by querying the tensor Q Z With key tensor K Y Sum value vector V Y Generate F WZ , and query the tensor Q by Y With key tensor K Z Sum value vector V Z Generate F WY , thereby achieving complementarity and enhancement of input information.

[0099] In this embodiment, W G and W OIt is a complex type of learnable parameter, so the gated cross-holding unit 221 introduces complex type linear transformation and nonlinear transformation, thereby enhancing the model's ability to represent input image data of different modalities.

[0100] The first feedforward unit 222 is used to perform layer normalization, full connection processing and activation function processing on the initial first output to obtain the first output.

[0101] Among them, in the first feedforward unit 222, the initial first output, that is, F SY It is processed in sequence through layer normalization, first full connection processing, activation function processing i.e. LeakyReLU and second full connection processing, and the corresponding output i.e. F FY Perform a residual connection with the initial first output to obtain the first output, namely Y'.

[0102] The second feedforward unit 223 is used to perform layer normalization, full connection layer processing and activation function processing on the initial second output to obtain a second output.

[0103] Among them, in the second feedforward unit 223, the initial second output, namely F SZ It is processed in sequence through layer normalization, first full connection processing, activation function processing i.e. LeakyReLU and second full connection processing, and the corresponding output i.e. F FZ Perform a residual connection with the initial second output to obtain the second output Z′.

[0104] Figure 5 It is a working block diagram of the hyperspectral channel attention module, the multispectral spatial attention module and the fusion module in an embodiment of the present invention.

[0105] like Figure 5 As shown, in this embodiment, the hyperspectral channel attention module 23, the multispectral spatial attention module 24 and the fusion module 25 together constitute a multi-level spatial channel attention parallel fusion decoder, which processes the first output and the second output to generate a high-resolution hyperspectral image.

[0106] The hyperspectral channel attention module 23 is used to extract features from the first output to obtain a channel attention feature map.

[0107] Among them, the expression for generating the channel attention feature map is:

[0108] F Y1 =LeakyRelu(FullyConnected(Y')),

[0109] F Y2 =LeakyRelu(FullyComnected(F Y2 )),

[0110] W Y1 =GlobalAvgPool(F Y2 )+GlobalMaxPool(F Y2 ),

[0111] W Y2 =sigmoid(W Y1 ),

[0112] F1'=W Y2 *F Y2 '

[0113] Where Y′ is the first output, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, GlobalAvgPool is the global average pooling operation, GlobalMaxPool is the maximum pooling operation, sigmoid is the sigmoid activation function processing, and F1' is the channel attention feature map.

[0114] The multi-spectral spatial attention module 24 is used to extract features from the second output to obtain a spatial attention feature map.

[0115] Among them, the expression for generating the spatial attention feature map is:

[0116] F Z =LeakyRelu(FullyConnected(Z)),

[0117] W Z1 =cat(mean(F Z ), max(F Z )),

[0118] W Z2 =sigmoid(Conv(W Z1 ),

[0119] F2'=W Z2 *F Z ,

[0120] Where Z' is the second output, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, mean is the average operation, max is the maximum operation, cat is the concatenation operation, sigmoid is the sigmoid activation function processing, and F2' is the spatial attention feature map.

[0121] The fusion module 25 is used to perform multi-level feature fusion on the channel attention feature map and the spatial attention feature map to obtain a high-resolution hyperspectral image.

[0122] Among them, the expression for generating high-resolution hyperspectral images is:

[0123] F E1 =concat(F1', F2'),

[0124] F E2 =LeakyRelu(FullyConnected(F E1 )),

[0125] F E3 =concat(F E2 , F1', F2'),

[0126] F E =LeakyRelu(FullyConnected(F3),

[0127] F H =Conv(F E ', C),

[0128] F H ′=Clamp(F H ),

[0129] Where F1' is the channel attention feature map, F2' is the spatial attention feature map, concat is the dimension concatenation operation, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, C is the learnable matrix, Conv is the two-dimensional convolution operation, Clamp is the clamping function Clamp processing, and F H ' is a high-resolution hyperspectral image.

[0130] In this embodiment, the feature map F E2 , F'1 and F2' are concatenated and fused at multiple levels to strengthen the spatial structure and channel correlation. C is the artificially designed convolution kernel parameter as a learnable matrix. The clamping function Clamp ensures that the pixel value range of the high-resolution hyperspectral image is between [0,1] to avoid interference of too large or too small values ​​on the super-resolution task.

[0131] Step S4, inputting the low-resolution hyperspectral image and the high-resolution multispectral image into the trained super-resolution reconstruction model to obtain a high-resolution hyperspectral image.

[0132] In this embodiment, the hyperspectral image super-resolution reconstruction method and the present method are respectively trained and tested on the existing hyperspectral image super-resolution methods, namely, the SSFCNN method, the SSRnet method, the TFnet method, the MCT method, the MIAE method, the Fusformer method and the UAL method, on the existing WashingtonDC data set, and the data of the four performance indicators of each method, namely, peak signal-to-noise ratio PSNR, root mean square error RMSE, spectral angle mapper SAM and relative dimensionless global synthesis error ERGAS, are obtained as shown in the following table:

[0133] PSNR RMSE SAM ERGAS SSFCNN 38.89 0.0079 0.64 0.36 SSRnet 39.32 0.0076 0.70 0.34 TFnet 41.82 0.0057 0.46 0.26 MCT 42.30 0.0053 0.48 0.24 MIAE 46.75 0.0112 0.72 0.26 Fusformer 40.34 0.0067 0.54 0.30 UAL 45.64 0.0036 0.31 0.16 This method 53.87 0.0034 0.24 0.08

[0134] The first column in the above table is each method, and the second to fifth columns are the values ​​of peak signal-to-noise ratio PSNR, root mean square error RMSE, spectral angle mapper SAM and relative dimensionless global synthesis error ERGAS corresponding to each method. For example, the cell in the second row and second column indicates that the value of the SSFCNN method on the peak signal-to-noise ratio PSNR is 38.89. The larger the peak signal-to-noise ratio PSNRUI, the better the method performance, the smaller the root mean square error RMSE, the better the method performance, the smaller the spectral angle mapper SAM, the better the method performance, and the smaller the relative dimensionless global synthesis error ERGAS, the better the method performance. It can be seen that this method has the best performance effect on the WashingtonDC dataset compared with the existing methods.

[0135] In this embodiment, the implicit representation blind estimation degradation network 100 and the second loss function are used as a priori-based blind estimation, the gated cross-keeping sharing module 22 is used as a shared encoder, the hyperspectral channel attention module 23, the multispectral spatial attention module 24 and the fusion module 25 are used as adaptive fusion decoders, and a model with corresponding configurations is constructed through ablation experiments to verify the effectiveness of the priori-based blind estimation, shared encoder and adaptive fusion decoder. Among them, the basic model is model 1, the basic model configured with a shared encoder is model 2, the basic model configured with a shared encoder and an adaptive fusion decoder is model 3, and the basic model configured with a priori-based blind estimation, a shared encoder and an adaptive fusion decoder is model 4. The performance indicators of each model on the existing data set are shown in the following table:

[0136] PSNR RMSE SAM ERGAS Model 1 46.79 0.0046 0.32 0.11 Model 2 49.62 0.0039 0.29 0.09 Model 3 52.85 0.0035 0.24 0.08 Model 4 53.61 0.0029 0.20 0.06

[0137] In the above table, the first column is each model, and the second to fifth columns are the values ​​of peak signal-to-noise ratio (PSNR), root mean square error (RMSE), spectral angle mapper (SAM), and relative dimensionless global synthesis error (ERGAS) corresponding to each model. For example, the cell in the second row and second column indicates that the value of peak signal-to-noise ratio (PSNR) of model 1 is 46.79. It can be seen that the prior-based blind estimation, shared encoder, and adaptive fusion decoder can all improve the performance of reconstructed images, and the joint application of the three can achieve more outstanding performance improvements.

[0138] Functions and Effects of the Embodiments

[0139] According to the hyperspectral image super-resolution reconstruction method involved in this embodiment, first, the shared structure between the two inputs is effectively learned through the gated cross-maintaining sharing module to achieve information complementarity; second, through the hyperspectral channel attention module and the multi-spectral spatial attention module, channel attention and spatial attention are used for feature processing, and the spatial spectral resolution is further improved through information fusion of the fusion module; third, the accuracy of the first parameter and the second parameter is improved through the implicit representation blind estimation degradation network, and then the second loss function is combined to further improve the reconstruction accuracy of the image. In short, this method can generate high-precision high-resolution hyperspectral images.

[0140] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A hyperspectral image super-resolution reconstruction method, for obtaining a high-resolution hyperspectral image corresponding to a low-resolution hyperspectral image based on a low-resolution hyperspectral image and a high-resolution multispectral image of the same scene, characterized in that: The following steps are involved: Step S1, constructing a training data set based on existing image data; Step S2, calculating a first parameter and a second parameter by implicitly characterizing and blindly estimating a degradation network according to the training data set; Step S3, constructing a super-resolution reconstruction model, and training the super-resolution reconstruction model according to the training data set, the first parameter and the second parameter to obtain a trained super-resolution reconstruction model; Step S4, inputting the low-resolution hyperspectral image and the high-resolution multispectral image into the trained super-resolution reconstruction model to obtain the high-resolution hyperspectral image, Wherein, the super-resolution reconstruction model includes: A tensor construction module, for generating a high-resolution multispectral image tensor according to the high-resolution multispectral image, and generating a low-resolution hyperspectral image tensor according to the low-resolution hyperspectral image; a gated cross-hold sharing module, configured to obtain, according to the low-resolution hyperspectral image tensor and the high-resolution multispectral image tensor, a first output corresponding to the low-resolution hyperspectral image tensor and a second output corresponding to the high-resolution multispectral image tensor; A hyperspectral channel attention module, configured to extract features from the first output to obtain a channel attention feature map; a multispectral spatial attention module, configured to extract features from the second output to obtain a spatial attention feature map; A fusion module is used to perform multi-level feature fusion on the channel attention feature map and the spatial attention feature map to obtain the high-resolution hyperspectral image.

2. The hyperspectral image super-resolution reconstruction method according to claim 1, Features: in, The gated cross-hold sharing module comprises: A gated cross-hold unit, configured to obtain an initial first output and an initial second output according to the low-resolution hyperspectral image tensor and the high-resolution multispectral image tensor; A first feedforward unit, used for performing layer normalization, full connection processing and activation function processing on the initial first output to obtain the first output; The second feedforward unit is used to perform layer normalization, full connection layer processing and activation function processing on the initial second output to obtain the second output.

3. The hyperspectral image super-resolution reconstruction method according to claim 2, characterized in that: in, In the gated cross-hold unit, the expression for generating the initial first output is: F EY =Embedding(Y), F LY =LN(F EY ), Q Y =F LY W Q D, F EZ =Embedding(Z), F LZ =LN(F EZ ), V Z =F LZ ·W v , F WY =(Q Y K Z T ⊙D)V Z , F GY =GroupNorm(F WY ), F RY =Reshape(F GY ), F SY =(swigh(F RY W G )⊙F EY )W O , Where Y is the low-resolution hyperspectral image tensor, Embedding is the Embedding dimension reduction operation, LN is the layer normalization operation, and W Q , W K and W v is a learnable parameter, D is a diagonal matrix with the same dimension as the low-resolution hyperspectral image tensor, Z is the high-resolution multispectral image tensor, is the conjugate tensor of the diagonal matrix D, T is the transpose, GroupNorm is the normalization operation, Reshape is the reshaping operation, W G and W O is a learnable parameter, swish is the swish activation function, F SY is the initial first output, In the gated cross-hold unit, the expression for generating the initial second output is: F EZ =Embedding(Z), F LZ =LN(F EZ ), Q Z =F LZ W Q D, F EY =Embedding(Y), F LY =LN(F EY ), V Y =F LY ·W v , F WZ =(Q Z K Y T ⊙D)V Y , F GZ =GroupNorm(F WZ ), F RZ =Reshape(F GZ ), F SZ =(swish(F RZ W G )⊙F EZ )W O , Where W Q , W K and W v are learnable parameters, D is a diagonal matrix with the same dimension as the low-resolution hyperspectral image tensor, is the conjugate tensor of the diagonal matrix D, W G and W O is a learnable parameter, F SZ is the initial second output.

4. The hyperspectral image super-resolution reconstruction method according to claim 2, characterized in that: in, In the first feedforward unit, the initial first output is sequentially subjected to layer normalization, first full connection processing, activation function processing, and second full connection processing, and the corresponding output is residually connected with the initial first output to obtain the first output. In the second feedforward unit, the initial second output is sequentially subjected to layer normalization, first full connection processing, activation function processing, and second full connection processing, and the corresponding output is residually connected with the initial second output to obtain the second output.

5. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: in, In the hyperspectral channel attention module, the expression for generating the channel attention feature map is: F Y1 =LeakyRelu(FullyConnected(Y')), F Y2 =LeakyRelu(FullyConnected(F Y2 )), W Y1 =GlobalAvgPool(F Y2 )+GlobalMaxPool(F Y2 ), W Y2 =sigmoid(W Y1 ), F1'=W Y2 *F Y2 , Wherein Y' is the first output, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, GlobalAvgPool is the global average pooling operation, GlobalMaxPool is the maximum pooling operation, sigmoid is the sigmoid activation function processing, and F1' is the channel attention feature map.

6. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: in, In the multispectral spatial attention module, the expression for generating the spatial attention feature map is: F Z =LeakyRelu(FullyConnected(Z')), W Z1 =cat(mean(F Z ),max(F Z )), W Z2 =sigmoid(Conv(W Z1 ), F2'=W Z2 *F Z , Wherein Z' is the second output, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, mean is the average operation, max is the maximum operation, cat is the concatenation operation, sigmoid is the sigmoid activation function processing, and F2' is the spatial attention feature map.

7. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: in, In the fusion module, the expression for generating the high-resolution hyperspectral image is: F E1 =concat(F1',F2'), F E2 =LeakyRelu(FullyConnected(F E1 )), F E3 =concat(F E2 ,F1',F2'), F E =LeakyRelu(FullyConnected(F3), F H =Conv(F E ',C), F H '=Clamp(F H ), Where F1' is the channel attention feature map, F2' is the spatial attention feature map, concat is the dimension concatenation operation, FullyConnected is the fully connected layer processing, LeakyRelu is the LeakyRelu activation function processing, C is the learnable matrix, Conv is the two-dimensional convolution operation, Clamp is the clamping function Clamp processing, F H ' is the high-resolution hyperspectral image.

8. The hyperspectral image super-resolution reconstruction method according to claim 1, Features: in, In the step S2, the implicit characterization blind estimation degradation network includes: A position coding module, used for generating a one-dimensional position coding and a two-dimensional position coding according to the number of channels of the low-resolution hyperspectral image and the high-resolution multispectral image; A multi-layer perception module, comprising a multi-layer perceptron M1 and a multi-layer perceptron M2, for generating an initial first parameter according to the one-dimensional position coding, and generating an initial second parameter according to the two-dimensional position coding; A loss module, comprising a first loss function, for calculating a first loss through the first loss function according to the low-resolution hyperspectral training images and the high-resolution multispectral training images in the training data set, as well as the initial first parameter and the initial second parameter; an iterative optimization module storing a preset termination condition, for optimizing the parameters of the position encoding module, the multilayer perceptron M1 module and the multilayer perceptron M2 module according to the first loss until the preset termination condition is reached, The first parameter is the initial first parameter generated by the optimized multilayer perceptron M1 module, The second parameter is the initial second parameter generated by the optimized multilayer perceptron M2 module.

9. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: in, In the position coding module, the expressions for generating one-dimensional position coding and two-dimensional position coding are: INSTEAD 1d =[PAGE1,PA2,A Y ], PE3=sin(2 L ·P·A U ), PE4=cos(2 L ·P·A V ), ON 2d =[PE3,PE4,A U ,A V ], Where A Y The inspace function generates a uniform distribution in [0,N Y -1], N Y is the number of channels of the low-resolution hyperspectral image, Π is pi, N Zi is the coding level index, N Z is the number of channels of the high-resolution multispectral image, sin() is the sine function, cos() is the cosine function, PE 1d is the one-dimensional position encoding, A U A is an array uniformly distributed in [0, U-1] generated by the inspace function. V is an array uniformly distributed in [0, V-1] generated by the inspace function, U and V are the width and height of the spatial blur kernel, L is the number of encoding layers, and PE 2d is the two-dimensional position code, In the multi-layer perception module, the expression for generating the initial first parameter is: Where SRF is the initial first parameter, LeakyReLU is the LeakyRelu activation function, M is the number of nodes in the current layer of the multilayer perceptron M1, and W jk is the weight of neuron k in the jth layer of the multilayer perceptron M1, is the one-dimensional position code of neuron k in the jth layer of multilayer perceptron M1, In the multi-layer perception module, the expression for generating the initial second parameter is: Where PSF is the initial second parameter, N is the number of nodes in the current layer of the multilayer perceptron M2, and W jk is the weight of neuron k in the i-th layer of the multilayer perceptron M2, is the two-dimensional position encoding of neuron k in the i-th layer of the multilayer perceptron M2, The expression of the first loss function is: Loss’=||M-H’|| 1,1 , H'≈S·HSI, M'≈MSI·PD, Where Loss' is the first loss, || 1,1 is regularization, HSI is the low-resolution hyperspectral training image, MSI is the high-resolution multispectral training image, S is the initial first parameter, P is the initial second parameter, and D is spatial downsampling.

10. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: in, In step S3, the parameters of the super-resolution reconstruction model are iteratively optimized using a second loss function. The expression of the second loss function is: Loss=||Y-HSI'|| 1,1 +γ||Z-MSI'|| 1,1 , HSI'=XPD, MSI' = SX, Wherein Y is the low-resolution hyperspectral training image in the training data set, Z is the high-resolution multispectral training image in the training data set, X is the output generated by the super-resolution reconstruction model according to the low-resolution hyperspectral training image Y and the high-resolution multispectral training image Z, S is the initial first parameter, P is the initial second parameter, and D is spatial downsampling.

Images