Hyperspectral image super-resolution reconstruction method based on long and short term memory network and hybrid convolution

Abstract

The invention discloses a hyperspectral image super-resolution reconstruction method based on a long and short term memory network and hybrid convolution. The method comprises the following steps: S1, preprocessing a data set of a hyperspectral image to generate a plurality of pairs of training sample sets of high-resolution image blocks and low-resolution image blocks; s2, constructing a hyperspectral image super-resolution network model based on a long and short term memory network and hybrid convolution; s3, performing penalty constraint on the trained network model by using a composite loss function, and calculating total loss; and S4, training the training sample set through the network model until the total loss is converged. Image information is effectively extracted based on the long-short-term memory network, and the quality of the reconstructed image is improved through hybrid convolution. And the inner product operator can adaptively allocate weights, so that visual elements with the most abundant information in a hyperspectral image spatial domain are preferentially ordered, the efficiency and the performance of the convolutional neural network are improved, and efficient feature extraction and reasoning are ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer vision enhancement, and in particular to a hyperspectral image super-resolution reconstruction method based on a long short-term memory network and hybrid convolution. Background Art

[0002] Hyperspectral imaging uses a hyperspectral camera to capture detailed spectral information of an object across multiple narrow wavelength bands. The technology records spectral data in a sequence of wavelengths, typically spanning the visible to near-infrared spectrum, with potentially hundreds of bands. Unlike traditional monochrome images, hyperspectral images provide rich, detailed spectral features that reveal an object's optical properties, chemical composition, surface reflectivity, and temperature distribution. This makes hyperspectral imaging valuable in remote sensing, geological surveys, agriculture, environmental monitoring, and life sciences. However, limited by sensor resolution, high spectral resolution degrades spatial quality, resulting in mixed pixels that impair image performance. Super-resolution (SR) reconstruction can restore high-resolution images from low-resolution images without adjusting hardware, improving image quality.

[0003] Hyperspectral image super-resolution (HSI-SR) is divided into fusion-based super-resolution and single image super-resolution. Fusion-based hyperspectral image super-resolution combines low-resolution images with high-resolution images from other sources (such as panchromatic or RGB images) to improve spatial details. In recent years, deep learning has developed rapidly, and convolutional neural networks (CNNs) have been widely used in the field of image processing, which has also brought new breakthroughs in image super-resolution. By introducing deep learning models, complex mapping relationships can be effectively learned from a large amount of training data, and high-resolution images can be predicted based on this. Hyperspectral image super-resolution methods based on convolutional neural networks (CNNs) can be divided into three types: 2D CNN, which focuses on spatial details but may ignore spectral continuity; 3D CNN, which can improve spectral accuracy but may reduce spatial resolution; hybrid 2D / 3D CNN, which aims to balance spectral and spatial learning to obtain better reconstruction results.

[0004] In recent years, deep learning-based hyperspectral image super-resolution (HSI-SR) has made significant progress. However, most methods rely on two-dimensional or three-dimensional convolution to extract features, often processing all spectral bands at the same time, while ignoring the inherent correlation between adjacent bands. In addition, due to the high dimensionality of hyperspectral images, the massive data they carry, and the complex spectral patterns, the difficulty of processing them is greatly increased. Existing methods mainly emphasize the image space spectral domain, but pay less attention to the image frequency domain, resulting in significant differences between the reconstructed results and the original images. Summary of the invention

[0005] The object of the present invention is to provide a hyperspectral image super-resolution reconstruction method based on long short-term memory network and hybrid convolution to solve the technical problems raised in the background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] S1. Preprocess the hyperspectral image dataset to generate several pairs of high-resolution image blocks I GT ' and low-resolution image block I LR ′’s training sample set;

[0008] S2. Using the training sample set, a hyperspectral image super-resolution network model based on long short-term memory network and hybrid convolution is constructed. The model consists of three parts in the following order: shallow feature extraction, deep feature extraction, upsampling and reconstruction of feature image output F up ;

[0009] S3. Use the composite loss function to penalize the trained network model and calculate the total loss L totle ;

[0010] S4. Train the training sample set through the network model and determine the total loss L totle Whether it converges;

[0011] If so, the trained long short-term memory network and the hybrid convolutional hyperspectral image super-resolution network are obtained;

[0012] If not, return to step S2;

[0013] S5. Use the trained network model to reconstruct a super-resolution hyperspectral image from the low-resolution hyperspectral image to be reconstructed.

[0014] Furthermore, the specific steps of S1 are:

[0015] S11, high-resolution image I in the hyperspectral image dataset GT As the original image, the original image is subjected to Gaussian kernel blur downsampling and interpolation processing to obtain a low-resolution image I LR , and establish high-low resolution image pairs on the hyperspectral image dataset;

[0016] S12, the high-resolution and low-resolution images are divided into a plurality of image blocks by a sliding window partitioning method, and a pair of high-resolution image blocks I is generated. GT ′ and low-resolution image block I LR ′, each pair of high-low resolution image blocks is a training sample, and several training samples form a training sample set.

[0017] Furthermore, the specific steps of S2 are:

[0018] S21, shallow feature extraction: low-resolution hyperspectral image block I LR ′ is shaped into a four-dimensional space, and then a three-dimensional convolution is used to transform the low-resolution hyperspectral image block I LR ′ performs shallow feature extraction, then the shallow feature F o It can be expressed as:

[0019] F o =f cinv3D (unsqueeze(I LR ′))

[0020] In the above formula, unsqueeze(.) is used for dimension expansion, f conv3D (.) represents a three-dimensional convolution operation, I LR ' represents a low-resolution hyperspectral image patch;

[0021] S22. Deep feature extraction: Extracting low-resolution hyperspectral image patches using long short-term memory networks and hybrid convolutional networks I LR 'Deep features;

[0022] S23, extract the shallow features F o Combined with the deep features, the combined feature F D It can be expressed as:

[0023] F D =F o +R D (R D-1 (...R1(F o )))

[0024] In the above formula, R D Corresponding to the Dth recurrent convolutional network, R D-1 Corresponding to the D-1th recurrent convolutional network, R1 corresponds to the first recurrent convolutional network, that is, the network containing long short-term memory network units, two-dimensional units and inner product operator units;

[0025] S24, upsampling and reconstruction: Upsample the combined feature FD with an upsampling factor u, and then use the transposed three-dimensional convolution layer to process the image reconstructed according to the combined feature FD and adjust it to the required size. The corresponding feature output F after resizing up It can be expressed as:

[0026] F up =squeeze(f conv3D (upscale(F D ,u))

[0027] In the above formula, upscale(.) represents the global upsampling operation, u is the upsampling scale factor, squeeze(.) represents the amplification function, and f conv3D Represents a 3D convolution operation.

[0028] Furthermore, the specific steps of S3 are:

[0029] S31. The L1 loss function is used to calculate the loss of pixels in the spatial domain, effectively penalizing small errors. The formula of the L1 loss function is:

[0030]

[0031] In the above formula, Represents the model's nth time to the low-resolution image block I LR 'Reconstruct high-resolution images, represents the nth high-resolution image block, N represents the total number of training images in a batch, Represents a set of network parameters The loss function for calculating the loss of pixels in the spatial domain is ||.||1, which means calculating the 1 norm;

[0032] S32, add the total spatial spectrum change loss L to the L1 loss function SSTV , through L SSTV The loss is calculated in the spectral domain, then

[0033]

[0034] In the above formula, Respectively represent the calculation of horizontal, vertical and spectral gradients, N represents the total number of training images in a batch, In the network parameter set The loss function for calculating the total spatial spectrum change loss is: represents the nth reconstruction result of the model, and ||.||1 represents the calculation of the 1 norm;

[0035] S33, introducing frequency perception loss L fre To learn the spectral features of high-resolution images, where L fre Including amplitude loss L amp and phase loss L pha , then L fre , L amp and L pha Respectively expressed as:

[0036] L fre =α1L amp +α2L pha

[0037]

[0038] In the above formula, and They refer to the amplitude and phase of the i-th reconstructed image, respectively. and represents the amplitude and phase of the i-th high-resolution image block, α1 and α2 are hyperparameters, N represents the total number of training images in a batch, and ||.||1 represents the calculation of the 1-norm;

[0039] S34. Calculate the total loss L totle , the calculation formula is:

[0040] L totle =L1+γ1L SSTV +γ2L fre

[0041] In the above formula, γ1 and γ2 are used to balance different losses and both γ1 and γ2 are set to 0.001.

[0042] Furthermore, the hybrid convolutional network in S22 is composed of a two-dimensional unit and an inner product operator unit in parallel; the long short-term memory network (LSTM) unit in S23 is composed of ConvLSTM to extract meaningful hyperspectral image features; the two-dimensional (2D) unit is composed of a convolutional layer, a Relu function and a residual connection, and the output of the nth two-dimensional unit of the two-dimensional unit is:

[0043]

[0044] In the above formula, represents the output of the nth two-dimensional unit, represents the output of the n-1th two-dimensional unit, f conv2D (.) is a two-dimensional convolution operation, and σ(.) represents the ReLU activation function.

[0045] Furthermore, the ConvLSTM adds the processing of forward and backward information.

[0046] Furthermore, an additional cross-space residual connection F is introduced into the constructed model c , upsample the input low-resolution image block to the nearest neighbor interpolation of the high-resolution image block, convert the size of the high-resolution hyperspectral image reconstructed by the model to L*tW*tH, and obtain the final reconstruction result of the model I SR , where F c and I SR The expressions are:

[0047] F c =F nearest (I LR '↑,t)∈RL*tH*tw

[0048] I SR =F c +F up

[0049] In the above formula, F nearest (I LR '↑,t) represents the input high-spectral low-resolution image block I LR 'Upsampling operation with a scaling factor of t is performed through nearest neighbor interpolation, R L*tH*tw express L*tH*tw dimensional vector space, F up Represents the feature image output after upsampling and reconstruction of the model.

[0050] Beneficial effects:

[0051] 1. The present invention uses spectral correlation based on long short-term memory network (LSTM) to effectively extract image information, and hybrid convolution enhances image spatial domain learning, thereby improving the quality of reconstructed images. In order to solve the problem of reduced data processing efficiency and greater difficulty in feature extraction caused by dimensionality mutation caused by the high dimensionality of hyperspectral images, a hybrid convolutional network is constructed in parallel by two-dimensional (2D) units and inner product operators (Involution). The inner product operator (Involution) replaces the traditional convolution operation and can adaptively assign weights, thereby prioritizing the most information-rich visual elements in the spatial domain of hyperspectral images, improving the efficiency and performance of the convolutional neural network (CNN), and ensuring efficient feature extraction and reasoning.

[0052] 2. The present invention introduces an additional cross-space residual connection F c To reduce the workload of the model, this connection involves upsampling the nearest neighbor interpolation of the input low-resolution image patch to the high-resolution image. This allows the model to more effectively learn the mapping relationship between the input low-resolution image patch and the output high-resolution image, thereby improving the quality and accuracy of the model's reconstruction of high-resolution hyperspectral images.

[0053] 3. The present invention designs a composite loss function that mixes L1 loss, frequency perception loss and spatial-spectral total variation loss to accurately capture spatial and spectral details. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions implemented in the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0055] Figure 1 Flow chart of a hyperspectral image super-resolution network reconstruction method based on a long short-term memory network (LSTM) and hybrid convolution in an embodiment of the present invention;

[0056] Figure 2 A flowchart for constructing and training a hyperspectral image super-resolution network based on a long short-term memory network (LSTM) and hybrid convolution in an embodiment of the present invention;

[0057] Figure 3 This is a framework diagram of a hyperspectral image super-resolution network reconstruction method based on a long short-term memory network (LSTM) and hybrid convolution provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] like Figure 1-Figure 3 As shown, the present invention provides a hyperspectral image super-resolution reconstruction method based on long short-term memory network and hybrid convolution, and the specific steps are as follows:

[0060] S1. Preprocess the hyperspectral image dataset to generate several pairs of high-resolution image blocks I GT ' and low-resolution image block I LR 'Training sample set;

[0061] In this embodiment, in order to illustrate the implementation of S1, the specific steps are as follows:

[0062] S11, high-resolution image I in the hyperspectral image dataset GT As the original image, the original image is subjected to Gaussian kernel blur downsampling and interpolation processing to obtain a low-resolution image I LR , and establish high-low resolution image pairs on the hyperspectral image dataset, where I GT ∈R L×rH×rW , I LR ∈R L×H×W , r is the sampling rate of Gaussian kernel blur downsampling, L is the number of spectral bands, H and W are the height and width of the image respectively, R L×rH×rW represents a L×rH×rW dimensional vector space, R L×H×WRepresenting L×H×W dimensional vector space, the high-dimensional spatial resolution image is processed into a low-dimensional spatial resolution image by Gaussian kernel blur downsampling and interpolation processing. The high-low resolution image pair in the present invention can be represented as: (x1, y1), (x2, y2),…, (x i ,y i ),…,(x n ,y n ), where y i is x i The corresponding low-resolution image, i represents the i-th pair of high-low resolution images, i = 1, 2, ..., n;

[0063] S12, the high-resolution and low-resolution images are divided into a plurality of image blocks by a sliding window partitioning method, and a pair of high-resolution image blocks I is generated. GT ' and low-resolution image block I LR ', each pair of high-low resolution image blocks is a training sample, and several training samples form a training sample set. The sliding window block method can improve the effect of subsequent image processing and the visual presentation effect of the model, improve the image processing efficiency and accuracy, and thus achieve better image analysis and model display effects.

[0064] S2. Using the training sample set, a hyperspectral image super-resolution network model based on long short-term memory network and hybrid convolution is constructed. The model consists of three parts in the following order: shallow feature extraction, deep feature extraction, upsampling and reconstruction of feature image output F up ;

[0065] In this embodiment, in order to illustrate the implementation of S2, the specific steps are as follows:

[0066] S21, shallow feature extraction: low-resolution hyperspectral image block I LR 'Shape it into a four-dimensional space, and then use three-dimensional convolution to the low-resolution hyperspectral image block I LR 'Perform shallow feature extraction, then the shallow feature F o It can be expressed as:

[0067] F o =f conv3D (unsqueeze(I LR '))

[0068] In the above formula, unsqueeze(.) is used for dimension expansion, f conv3D (.) represents a three-dimensional convolution operation, I LR ' represents a low-resolution hyperspectral image patch;

[0069] S22. Deep feature extraction: Extracting low-resolution hyperspectral image patches using long short-term memory networks and hybrid convolutional networks I LR 'Deep features, the hybrid convolutional network is composed of two-dimensional (2D) units and inner product operator (Involution) units in parallel, and the long short-term memory network (LSTM) is used to evaluate the spectral correlation between adjacent channels of low-resolution hyperspectral image patches to effectively extract the feature information of low-resolution hyperspectral image patches of the input image, and then the hybrid convolutional network is used to enhance the model capacity and strengthen the spatial correlation analysis;

[0070] S23, extract the shallow features F o Combined with deep features, the model stability and information flow are further improved. The combined feature F D It can be expressed as:

[0071] F D =F o +R D (R D-1 (...R1(F o )))

[0072] In the above formula, R D Corresponding to the Dth recurrent convolutional network, R D-1 Corresponding to the D-1th recurrent convolutional network, R1 corresponds to the first recurrent convolutional network, that is, a network containing a long short-term memory network (LSTM) unit, a two-dimensional (2D) unit and an inner product operator (Involution) unit, wherein the two-dimensional (2D) unit and the inner product operator (Involution) unit in parallel constitute a hybrid convolutional network.

[0073] Among them, the long short-term memory network (LSTM) unit: consists of ConvLSTM (convolutional LSTM network). The ConvLSTM architecture combines the characteristics of CNN (convolutional neural network) and LSTM, replacing the fully connected layer in the traditional LSTM with a convolutional layer. It accepts three-dimensional tensor input, enabling it to capture data dependencies in space and time at the same time.

[0074] The present invention can use ConvLSTM to take each spectral channel as a time step or frame to capture the dependencies between spectral channels. ConvLSTM updates the hidden state h at time t at each time step or spectral channel. t and the cell state c t These states will be based on the input of the current time step, the previous hidden state h t-1 and the previous cell state c t-1The ConvLSTM unit contains convolution operations and long short-term memory network (LSTM) operations, which enables it to capture the spatial and temporal dependencies in the spectral data. The hidden state h t represents the features or representations learned at each spectral channel, capturing relevant information from previous channels. Similarly, the cell state c t Keep updating and passing content or context information through time steps. By utilizing these hidden states and cell states, ConvLSTM effectively models the relationship between spectral channels, allowing it to extract meaningful hyperspectral image features, where the hidden state h t and the cell state c t The expression is:

[0075] h t ,c t =ConvLSTM(h t-1 ,c t-1 ,F t ,)

[0076] In the above formula, ConvLSTM implicitly captures the motion between previous states: the previous (t-1 time) hidden state h t-1 , the previous (t-1 time) cell state c t-1 And the feature map F of the low-resolution image input at the current time t t .

[0077] In order to effectively utilize the global context, the present invention adds the processing of forward and reverse information in ConvLSTM. It first reverses the input high-spectral low-resolution image data, and then passes it through the forward and reverse ConvLSTM models respectively. Finally, the outputs of the forward model and the reverse model are spliced, and the 1x1 convolution layer is reduced to obtain the final output data.

[0078] Two-dimensional (2D) unit: The present invention designs a two-dimensional unit consisting of a convolutional layer, a ReLU function and a residual connection. The two-dimensional convolution and the residual connection are outstanding in spatial feature extraction and network performance improvement. The output of the nth two-dimensional unit of the two-dimensional unit is:

[0079]

[0080] In the above formula, represents the output of the nth two-dimensional unit, represents the output of the n-1th two-dimensional unit, f conv2D (.) is a two-dimensional convolution operation, and σ(.) represents the ReLU activation function.

[0081] Involution unit: The inner product operator is an extension of the traditional convolution. It not only considers the information of the current feature point of the input hyperspectral image, but also introduces the global information of the neighboring feature points. This enables the model to obtain more contextual information of the input hyperspectral image, thereby improving the ability to understand complex structures. The inner product operator can adaptively assign weights to prioritize the most informative visual elements in the spatial domain of the hyperspectral image.

[0082] S24, upsampling and reconstruction: Use an upsampling scale factor u to reconstruct the combined features F D Up-sample, and then use the transposed three-dimensional convolution layer to combine the features F D The reconstructed image is processed and resized to the required size, and the corresponding feature output after resizing is F up It can be expressed as:

[0083] F up =squeeze(f conv3D (upscale(F D ,u))

[0084] In the above formula, upscale(.) represents the global upsampling operation, u is the upsampling scale factor, squeeze(.) represents the amplification function, and f conv3D Represents a 3D convolution operation.

[0085] Due to the high similarity between the input low-resolution image and the output high-resolution image, an additional cross-space residual connection F is introduced. c To reduce the workload of the model, this connection involves upsampling the nearest neighbor interpolation of the input low-resolution image block to the high-resolution image. This allows the model to more effectively learn the mapping relationship between the input low-resolution image block and the output high-resolution image, thereby improving the quality and accuracy of the model's reconstruction of high-resolution hyperspectral images. The size of the high-resolution hyperspectral image reconstructed by the model is converted to L*tW*tH to obtain the final reconstruction result of the model I SR , where F c and I SR The expressions are:

[0086] F c =F nearest (I LR '↑,t)∈R L*tH*tw

[0087] I SR =F c +F up

[0088] In the above formula, F nearest (ILR '↑,t) represents the input high-spectral low-resolution image block I LR 'Upsampling operation with a scaling factor of t is performed through nearest neighbor interpolation, R L*tH*tw express L*tH*tw dimensional vector space, F up Represents the feature image output after upsampling and reconstruction of the model.

[0089] The goal of this invention is to use a super-resolution network model to transform the input low-resolution hyperspectral image block I LR 'Reconstruct and generate high-resolution hyperspectral image I SR .

[0090] S3. Use the composite loss function to penalize the trained network model and calculate the total loss L totle , optimize the model from the spatial, spectral and frequency domains, and gradually refine the characteristics of the network model to improve image quality;

[0091] In this embodiment, in order to illustrate the implementation of step S3, the specific steps are as follows:

[0092] S31. The L1 loss function is used to calculate the loss of pixels in the spatial domain, effectively penalizing small errors. The formula of the L1 loss function is:

[0093]

[0094] In the above formula, Represents the model's nth time to the low-resolution image block I LR 'Reconstruct high-resolution images, represents the nth high-resolution image block, N represents the total number of training images in a batch, Represents a set of network parameters The loss function is used to calculate the loss of pixels in the spatial domain. ||.||1 means to find the 1 norm.

[0095] S32, add the total spatial spectrum change loss L to the L1 loss function SSTV , through L SSTV The loss is calculated in the spectral domain, then

[0096]

[0097] In the above formula, Respectively represent the calculation of horizontal, vertical and spectral gradients, N represents the total number of training images in a batch, In the network parameter set The loss function for calculating the total spatial spectrum change loss is: represents the nth reconstruction result of the model, and ||.||1 means calculating the 1-norm.

[0098] Although the L1 loss effectively preserves the spatial details of the super-resolution image output, it ignores the spectral aspect and may cause spectral distortion. To address this issue, the present invention incorporates the total spatial spectral variation (SSTV) into the loss function. SSTV is an improvement over the traditional total variation model that takes into account both the spatial details and spectral distortion of the image output.

[0099] S33, introducing frequency perception loss L fre To learn the spectral features of high-resolution images, where L fre Including amplitude loss L amp and phase loss L pha , then L fre , L amp and L pha Respectively expressed as:

[0100] L fre =α1L amp +α2L pha

[0101]

[0102] In the above formula, and They refer to the amplitude and phase of the i-th reconstructed image, respectively. and represents the amplitude and phase of the i-th high-resolution image patch, α1 and α2 are hyperparameters, N represents the total number of training images in a batch, and ||.||1 means calculating the 1-norm.

[0103] S34. Calculate the total loss L totle , the calculation formula is:

[0104] L totle =L1+γ1L SSTV +γ2L fre

[0105] In the above formula, γ1 and γ2 are used to balance different losses. In the present invention, γ1 and γ2 are both set to 0.001.

[0106] The present invention designs a composite loss function that mixes L1 loss, frequency-aware loss and spatial-spectral total variation loss to accurately capture spatial and spectral details.

[0107] S4. Train the training sample set through the network model and determine the total loss L totle Whether it converges;

[0108] If so, the trained long short-term memory network and the hybrid convolutional hyperspectral image super-resolution network are obtained;

[0109] If not, return to step S2; train the training sample set through the network model until the total loss L totle Convergence, obtain the trained long short-term memory network (LSTM) and hybrid convolutional hyperspectral image super-resolution network, by repeating steps S2 to S3, update the convolution and weights of the network model until the total loss converges, complete the training of the network model, and judge the closeness of the predicted result to the actual image by the convergence of the total loss. The convergence of the total loss means that the difference between the reconstruction result and the high-resolution image is minimal.

[0110] S5. Use the trained hyperspectral image super-resolution network model based on long short-term memory network and hybrid convolution to reconstruct the super-resolution hyperspectral image from the low-resolution hyperspectral image to be reconstructed.

[0111] Due to the high-dimensional characteristics of hyperspectral images, there are still challenges in making full use of spectral correlation and spatial information. In order to make the network obtain better performance, the present invention discloses a hyperspectral image super-resolution reconstruction method based on long short-term memory network and hybrid convolution, which uses spectral correlation based on long short-term memory network (LSTM) to effectively extract image information, and hybrid convolution enhances image spatial domain learning, thereby improving the quality of reconstructed images. In order to solve the problem of reduced data processing efficiency and greater difficulty in feature extraction caused by dimensionality mutation caused by the high dimensionality of hyperspectral images, the present invention constructs a hybrid convolutional network in parallel through two-dimensional (2D) units and inner product operators (Involution). The inner product operator (Involution) replaces the traditional convolution operation and can adaptively assign weights, thereby prioritizing the most information-rich visual elements in the spatial domain of hyperspectral images, improving the efficiency and performance of convolutional neural networks (CNNs), and ensuring efficient feature extraction and reasoning.

[0112] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0113] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A hyperspectral image super-resolution reconstruction method based on long short-term memory network and hybrid convolution, characterized in that: The following steps are involved: S1. Preprocess the hyperspectral image dataset to generate several pairs of high-resolution image blocks I GT ' and low-resolution image block I LR 'Training sample set; S2. Using the training sample set, a hyperspectral image super-resolution network model based on long short-term memory network and hybrid convolution is constructed. The model consists of three parts in the following order: shallow feature extraction, deep feature extraction, upsampling and reconstruction of feature image output F up ; S3. Use the composite loss function to penalize the trained network model and calculate the total loss L totle ; S4. Train the training sample set through the network model and determine the total loss L totle Whether it converges; If so, the trained long short-term memory network and the hybrid convolutional hyperspectral image super-resolution network are obtained; If not, return to step S2; S5. Use the trained network model to reconstruct a super-resolution hyperspectral image from the low-resolution hyperspectral image to be reconstructed.

2. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: The specific steps of S1 are: S11, high-resolution image I in the hyperspectral image dataset GT As the original image, the original image is subjected to Gaussian kernel blur downsampling and interpolation processing to obtain a low-resolution image I LR , and establish high-low resolution image pairs on the hyperspectral image dataset; S12, the high-resolution and low-resolution images are divided into a plurality of image blocks by a sliding window partitioning method, and a pair of high-resolution image blocks I is generated. GT ' and low-resolution image block I LR ', each pair of high-low resolution image blocks is a training sample, and several training samples form a training sample set.

3. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: The specific steps of S2 are: S21, shallow feature extraction: low-resolution hyperspectral image block I LR 'Shape it into a four-dimensional space, and then use three-dimensional convolution to the low-resolution hyperspectral image block I LR 'Perform shallow feature extraction, then the shallow feature F o It can be expressed as: F o =f conv3D (unsqueeze(I LR ')) In the above formula, unsqueeze(.) is used for dimension expansion, f conv3D (.) represents a three-dimensional convolution operation, I LR ' represents a low-resolution hyperspectral image patch; S22. Deep feature extraction: Extracting low-resolution hyperspectral image patches using long short-term memory networks and hybrid convolutional networks I LR 'Deep features; S23, extract the shallow features F o Combined with the deep features, the combined feature F D It can be expressed as: F D =F o +R D (R D-1 (...R1(F o ))) In the above formula, R D Corresponding to the Dth recurrent convolutional network, R D-1 Corresponding to the D-1th recurrent convolutional network, R1 corresponds to the first recurrent convolutional network, that is, the network containing long short-term memory network units, two-dimensional units and inner product operator units; S24, upsampling and reconstruction: Use an upsampling scale factor u to reconstruct the combined features F D Up-sample, and then use the transposed three-dimensional convolution layer to combine the features F D The reconstructed image is processed and resized to the required size, and the corresponding feature output after resizing is F up It can be expressed as: F up =squeeze(f conv3D (upscale(F D ,u)) In the above formula, upscale(.) represents the global upsampling operation, u is the upsampling scale factor, squeeze(.) represents the amplification function, and f conv3D Represents a 3D convolution operation.

4. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: The specific steps of S3 are: S31. The L1 loss function is used to calculate the loss of pixels in the spatial domain, effectively penalizing small errors. The formula of the L1 loss function is: In the above formula, Represents the model's nth time to the low-resolution image block I LR 'Reconstruct high-resolution images, represents the nth high-resolution image block, N represents the total number of training images in a batch, Represents a set of network parameters The loss function for calculating the loss of pixels in the spatial domain is ||.||1, which means calculating the 1 norm; S32, add the total spatial spectrum change loss L to the L1 loss function SSTV , through L SSTV The loss is calculated in the spectral domain, then In the above formula, Respectively represent the calculation of horizontal, vertical and spectral gradients, N represents the total number of training images in a batch, In the network parameter set The loss function for calculating the total spatial spectrum change loss is: represents the nth reconstruction result of the model, and ||.||1 represents the calculation of the 1 norm; S33, introducing frequency perception loss L fre To learn the spectral features of high-resolution images, where L fre Including amplitude loss L amp and phase loss L pha , then L fre , L amp and L pha Respectively expressed as: L fre =α1L amp +α2L pha In the above formula, and They refer to the amplitude and phase of the i-th reconstructed image, respectively. and represents the amplitude and phase of the i-th high-resolution image block, α1 and α2 are hyperparameters, N represents the total number of training images in a batch, and ||.||1 represents the calculation of the 1-norm; S34. Calculate the total loss L totle , the calculation formula is: L totle =L1+γ1L SSTV +γ2L fre In the above formula, γ1 and γ2 are used to balance different losses and both γ1 and γ2 are set to 0.

001.

5. The hyperspectral image super-resolution reconstruction method according to claim 3, characterized in that: The hybrid convolutional network in S22 is composed of two-dimensional units and inner product operator units in parallel; The long short-term memory network unit in S23 is composed of ConvLSTM to extract meaningful hyperspectral image features; The two-dimensional unit consists of a convolutional layer, a ReLU function, and a residual connection. The output of the nth two-dimensional unit is: In the above formula, represents the output of the nth two-dimensional unit, represents the output of the n-1th two-dimensional unit, f conv2D (.) is a two-dimensional convolution operation, and σ(.) represents the ReLU activation function.

6. The hyperspectral image super-resolution reconstruction method according to claim 5, characterized in that: The ConvLSTM adds processing of forward and backward information.

7. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that: Introducing an additional cross-space residual connection F in the constructed model c , upsample the input low-resolution image block to the nearest neighbor interpolation of the high-resolution image block, convert the size of the high-resolution hyperspectral image reconstructed by the model to L*tW*tH, and obtain the final reconstruction result of the model I SR , where F c and I SR The expressions are: F c =F nearest (I LR '↑,t)∈R L*tH*tw I SR =F c +F up In the above formula, F nearest (I LR '↑,t) represents the input high-spectral low-resolution image block I LR 'Upsampling operation with a scaling factor of t is performed through nearest neighbor interpolation, R L*tH*tw represents the L*tH*tw-dimensional vector space, F up Represents the feature image output after upsampling and reconstruction of the model.

Images