An interactive hyperspectral image super-resolution method for simultaneously extracting spatial and spectral information

Abstract

The present invention discloses an interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information, comprising the steps of: 1, collecting a data set for hyperspectral image super-resolution, processing the data set, and generating a low-resolution hyperspectral image; 2, inputting the training set into an LSMSAformer module on a Transformer branch and a 3DDWTC module on a CNN branch, respectively extracting spatial and spectral features of the low-resolution hyperspectral image; 3, using a SWIU module to realize the interaction between the LSMSAformer module and the 3DDWTC module, and generating a feature map F n ; Step 4: transform the feature map F n Image reconstruction is performed to obtain a super-resolution hyperspectral image. The present invention can restore the global information and local information in the image space while restoring the high spatial resolution of the low-resolution hyperspectral image, thereby obtaining a better reconstructed image and enabling the hyperspectral image to achieve the expected super-resolution effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information. Background Art

[0002] Hyperspectral images refer to spectral images with a spectral resolution of 0.01λ. Compared with conventional RGB images, hyperspectral images have more bands to accurately and comprehensively describe the characteristics of the captured scene. They have been widely used in the fields of land mapping, resource exploration, agricultural remote sensing, environmental remote sensing, forestry monitoring, soil remote sensing, water color remote sensing, atmospheric science, etc. However, due to the lack of imaging instrument technology, hyperspectral imaging has always had the problem of low spatial resolution. At present, in the field of deep learning, the super-resolution model constructed based on CNN convolutional neural network has achieved great performance improvement and has been increasingly used in hyperspectral image extraction.

[0003] However, CNN convolutional neural network focuses on obtaining the spectral features of the image, while there is a large amount of long-distance information in the image. The limited perception of CNN convolutional neural network will hinder the acquisition of image spatial features, making it impossible for hyperspectral images to achieve the expected super-resolution effect.

[0004] Therefore, it is necessary to invent an interactive hyperspectral image super-resolution method that simultaneously extracts spatial information and spectral information to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide an interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information. The global information and local information of a low-resolution hyperspectral image in space are extracted by the LSMSAformer module, and the spectral information of the low-resolution hyperspectral image is extracted by the 3DDWTC module. The SWIU module is used to realize the interaction between the LSMSAformer module and the 3DDWTC module, so that the entire low-resolution hyperspectral image is restored to a high spatial resolution while also restoring the global information and local information in the image space, thereby obtaining a better reconstructed image to solve the above-mentioned deficiencies in the technology.

[0006] In order to achieve the above object, the present invention provides the following technical solution: an interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information, comprising the following steps:

[0007] Step 1: Collect a data set for hyperspectral image super-resolution, process the data set, and generate a low-resolution hyperspectral image;

[0008] Step 2: Input the training set into the LSMSAformer module on the Transformer branch and the 3DDWTC module on the CNN branch to extract the spatial and spectral features of the low-resolution hyperspectral image respectively;

[0009] Step 3: Use the SWIU module to implement the interaction between the LSMSAformer module and the 3DDWTC module to generate the feature map F n ;

[0010] Step 4: The feature map F n Image reconstruction is performed to obtain a super-resolution hyperspectral image.

[0011] In the aforementioned interactive hyperspectral image super-resolution method for simultaneously extracting spatial and spectral information, in step 1, a data set for hyperspectral image super-resolution is collected, and the data set is processed to generate a low-resolution hyperspectral image. The specific process is:

[0012] Dataset division: First, divide the dataset into training set and test set;

[0013] Apply Gaussian blur: Apply a 5×5 Gaussian blur to the data within the dataset with a standard deviation of 2 and a mean of 0 to generate a blurred image.

[0014] Downsampling: Downsample the blurred image by a factor of 4 to generate a low-resolution hyperspectral image.

[0015] In the aforementioned interactive hyperspectral image super-resolution method for simultaneously extracting spatial and spectral information, in step 2, the training set is input into the LSMSAformer module on the Transformer branch and the 3DDWTC module on the CNN branch to extract the spatial and spectral features of the low-resolution hyperspectral image, respectively. The specific steps are as follows:

[0016] 2.1. Let the low-resolution hyperspectral image be x in ,and

[0017] Where H is the height of the low-resolution hyperspectral image,

[0018] W is the width of the low-resolution hyperspectral image,

[0019] C is the number of channels of the low-resolution hyperspectral image;

[0020] 2.2. Use a 3D convolution with a kernel size of 3×3×3 from x in Extract the shallow feature F0, reshape the shallow feature F0 into 4D data, and then pass it into the LSMSAformer module as input to get T x0 , the specific formula is as follows:

[0021] F0=f 3Dconv (x in )

[0022] T x0 =f to4D (F0)

[0023] Among them, the function f to4D The function of (·) is to convert the size of N×D×C×H×W into (N×D)×C×H×W.

[0024] N represents the batch size,

[0025] D represents the embedding latitude;

[0026] T x0 x resized to fit the input dimensions of the LSMSAformer module branches in ;

[0027] f 3Dconv (x in ) is the shallow features of x extracted after passing through a 3×3×3 3D convolution layer in ;

[0028] 2.3. Using LSMSAformer module to analyze T x0 Processing is performed to obtain global and local information of low-resolution hyperspectral images in space;

[0029] 2.4. In the LSMSAformer module, in During processing, the shallow feature F0 will also be input into the 3DDWTC module for processing to obtain the spectral information of the low-resolution hyperspectral image.

[0030] The aforementioned interactive hyperspectral image super-resolution method for simultaneously extracting spatial and spectral information, in step 3, uses the SWIU module to implement the interaction between the LSMSAformer module and the 3DDWTC module to generate a feature map F n The specific steps are:

[0031] 3.1. Use the SWIU module to interactively fuse the outputs of each LSMSAformer module and the 3DDWTC module.

[0032] 3.2. Combine the outputs of the nth LSMSAformer module and the nth 3DDWTC module to form the feature map F n .

[0033] In the aforementioned interactive hyperspectral image super-resolution method for simultaneously extracting spatial and spectral information, in step 2.3: the specific steps of each LSMSAformer module in extracting the global information and local information of the low-resolution hyperspectral image in space are:

[0034] a. Enter the predefined zoom ratio s and x in Enter the LSMSAformer module;

[0035] And s is set to 4;

[0036] b. x in Projected into features through the projection layer;

[0037] c. Extract x using LSIB module in The local information W i , the specific steps are:

[0038] c1. Use two 2D convolutions to extract higher-level features from the input features in turn, and perform residual connections between the extracted features and the input features of the LSIB module to form a feature map P i ;

[0039] c2. The features extracted by the first 2D convolutional layer are further input into the SiLU activation function, which is a combination of the sigmoid function and the ReLU function. The low-resolution hyperspectral image extracted by the two convolutional layers is obtained through the sigmoid activation function symmetrical about the origin to obtain the attention map Q i ,

[0040] c3, feature map P i With the attention map Q i Perform multiplication operations between elements to obtain the final W i , and the calculation process of the i-th LSIB module can be expressed as:

[0041] P i =f 2dconv (f SiLU (f 2dconv (W i-1 )))+W i-1

[0042] Q i =f sigmoid (f 2dconv (f SiLU (f 2dconv (W i-1 ))))

[0043] W i =P i ⊙Q i

[0044] Among them, W i-1 Indicates the output result of the previous up / down sampling module of the LSIB module.

[0045] f 2dconv (·) indicates W i-1 The 2d convolutional layer used after the input,

[0046] f SiLU (·) represents the SiLU activation function, which is a combination of the sigmoid function and the ReLU function.

[0047] f sigmoid (·) represents the sigmoid activation function,

[0048] ⊙ represents the multiplication operation between elements;

[0049] d. W through LSIB module i Input to the multi-head self-attention MSA module to obtain global information x MSA ;

[0050] e. The obtained x MSA and W i Connect them in parallel to get the output of each LSMSAformer module.

[0051] In the aforementioned interactive hyperspectral image super-resolution method for simultaneously extracting spatial and spectral information, in step 2.4, each of the 3DDWTC modules obtains x in The specific process of obtaining the spectral information is as follows:

[0052] F out =f 1×1×1CNN (f 3×3×3CNN (f 1×1×1CNN (x in )))+f DWT-3dCRU (x in )+x in

[0053] Among them, f 1×1×1CNN is a 1×1×1 3d convolutional layer,

[0054] f 3×3×3CNN is a 3×3×3 3D convolutional layer,

[0055] F out is the spectral information,

[0056] f DWT-3dCRU It is a function of all the steps included in the DWT-3dCRU module;

[0057] Among them, the channel reconstruction unit DWT-3dCRU module constructed by wavelet transform is introduced to fully obtain spectral information, and the channel reconstruction unit DWT-3dCRU module constructed by wavelet transform is introduced. The specific steps are as follows:

[0058] A1. For x in Use the wavelet transform function to transform the original x in The input image of the C channel is converted into the high-frequency input of the 3C channel and the low-frequency input of the C channel, and then the number of channels of the low-frequency input and the high-frequency input is compressed to C;

[0059] A2. Control x with an extrusion ratio r in The number of input channels, and r is 1 / 2, after segmentation and squeezing, we get x up and x low ;

[0060] Among them, x up Refers to high-frequency information input, images with rich information,

[0061] x low Refers to low-frequency information input, images with sparse information;

[0062] The specific formula is as follows:

[0063] x up ,x low =f squeeze (f DWT (x in ))

[0064] Among them, f squeeze (·) represents the function of the extrusion channel,

[0065] f DWT (·) represents the wavelet transform function,

[0066] A3. x up Input to the upper stage, as a branch for extracting rich features, 3D group convolution 3D-GWC and 3D point-by-point convolution 3D-PWC are used instead of the ordinary k×K standard convolution on x up Process and obtain Y1;

[0067] A4, x low Input to the upper stage as a rare feature extraction branch, for the input feature x low Perform a simple 1×1×1 3D-PWC operation as x up The addition of x low The resulting feature map and x after the 3d-PWC operation low Perform parallel operation to obtain Y2. The specific formula is as follows:

[0068] Y1=f 3d-GWC (x up )+f 3d-PWC (x up )

[0069] Y2=f concat (f 3d-PWC (x up ),x low )

[0070] Among them, Y1 represents the feature map that extracts rich spectral information.

[0071] Y2 is the characteristic graph of sparse spectral information, which is used as a supplement to Y1.

[0072] f 3d-GWC (·) is the 3D group convolution operation,

[0073] f 3d-PWC (·) is the 3D point-by-point convolution operation,

[0074] f concat (·) is parallel operation;

[0075] A5. Perform an average pooling operation on Y1 and Y2, normalize them with the softmax function, and then multiply them with themselves to obtain the results out1 and out2. Use the inverse wavelet transform to restore the obtained out1 and out2, and finally output them as Y3. The specific formula is as follows:

[0076] out1=f softmax (f avgpool (Y1))·Y1

[0077] out2 = f softmax (f avgpool (Y2))·Y2

[0078] F out =f IWT (out1,out2)

[0079] Among them, f softmax is the softmax function,

[0080] f avgpool is the average pooling operation,

[0081] out1 is the result of multiplying Y1 and Y1 after the average pooling operation of the softmax function, which is used to represent the image with rich spectral information after improving the spectral information;

[0082] out1 represents the result of multiplying Y2 and Y2 after the average pooling operation of the softmax function, which is used to represent the image with sparse spectral information after improving the spectral information.

[0083] The above-mentioned interactive hyperspectral image super-resolution method for simultaneously extracting spatial and spectral information, in step 4, the feature map F n Perform image reconstruction to obtain a super-resolution hyperspectral image. The specific steps are as follows:

[0084] 4.1. Feature map F after fusing the LSMSAformer module and the 3DDWTC module n The 1×1×1 3D convolution is fed into the network and the residual connection is performed with the shallow feature F0. After a deconvolution and a 3×3×3 3D convolution, the super-resolution hyperspectral image is obtained after being magnified by the super-resolution factor. SR Indicates that, and I SR The specific formula for the entire acquisition process is as follows:

[0085] I SR =f 3×3×3conv (f deconv (f 1×1×1conv (F n )+F0))

[0086] Among them, the function f deconv (·) represents the deconvolution layer;

[0087] 4.2. Measuring I using L1 loss (mean absolute error) SR Optimize the pixel difference between the image and the ground truth SR ;

[0088] Among them, the specific formula of L1 loss is as follows:

[0089]

[0090] Among them, H TCC (·) is the network structure function,

[0091] N is the number of training image pairs,

[0092] represents the ground truth image.

[0093] Compared with the prior art, the present invention has the following beneficial effects:

[0094] The present invention extracts the global information and local information of the low-resolution hyperspectral image in space through the LSMSAformer module, and adopts the local spatial information extraction LSIB module and the long-distance spatial information extraction MSA module in the Transformer branch, and extracts the spectral information of the low-resolution hyperspectral image through the 3DDWTC module, and adopts the 3D convolution and local spectral attention extraction DWT-3dCRU module in the convolutional neural network branch. Subsequently, in the LSMSAformer module, the global spatial information and local spatial information at different scales are integrated, so that the entire low-resolution hyperspectral image restores the high spatial resolution while also restoring the detail information in the image space, that is, the global information and the local information; at the same time, in the DWT-3dCRU module of the convolutional neural network branch, the input image is divided into high-frequency input and low-frequency input by wavelet transform, and the image with rich features and the image with sparse features are processed separately by 3D group convolution and 3D point convolution, so that high efficiency and reduced parameter amount are achieved when extracting spectral information; and the shared weight SWIU module between parallel branches better integrates the global and local information in space and spectrum in an adaptive manner, so as to obtain a better reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0096] Figure 1 is a flow chart of the present invention;

[0097] Figure 2 is the super-resolution result image of the present invention;

[0098] Figure 3 It is the actual hyperspectral image of the present invention. DETAILED DESCRIPTION

[0099] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0100] The present invention provides Figure 1-3 An interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information includes the following steps:

[0101] Step 1: Collect a data set for hyperspectral image super-resolution, process the data set, and generate a low-resolution hyperspectral image;

[0102] Among them, in step 1, the specific process of collecting a data set for super-resolution of hyperspectral images, processing the data set, and generating a low-resolution hyperspectral image is as follows:

[0103] Dataset division: First, divide the dataset into training set and test set;

[0104] Apply Gaussian blur: Apply a 5×5 Gaussian blur to the data within the dataset with a standard deviation of 2 and a mean of 0 to generate a blurred image.

[0105] In step 1, a 5×5 Gaussian blur with a standard deviation of 2 and a mean of 0 is used to process the data within the dataset, which can smooth the image, blur details and noise, and make the generated low-resolution image closer to the low-resolution image in real life, suitable for various image processing scenarios, and reduce computational costs.

[0106] Step 2: Input the training set into the LSMSAformer module on the Transformer branch and the 3DDWTC module on the CNN branch to extract the spatial and spectral features of the low-resolution hyperspectral image, respectively. The specific steps are as follows:

[0107] 2.1. Let the low-resolution hyperspectral image be x in ,and

[0108] Where H is the height of the low-resolution hyperspectral image,

[0109] W is the width of the low-resolution hyperspectral image,

[0110] C is the number of channels of the low-resolution hyperspectral image;

[0111] 2.2. Use a 3D convolution with a kernel size of 3×3×3 from x in Extract the shallow feature F0, reshape the shallow feature F0 into 4D data, and then pass it into the LSMSAformer module as input to get T x0 , the specific formula is as follows:

[0112] F0=f 3Dconv (x in )

[0113] T x0 =f to4D (F0)

[0114] Among them, the function f to4D The function of (·) is to convert the size of N×D×C×H×W into (N×D)×C×H×W.

[0115] N represents the batch size,

[0116] D represents the embedding latitude;

[0117] T x0 x resized to fit the input dimensions of the LSMSAformer module branches in ;

[0118] f 3Dconv (x in ) is the shallow features of x extracted after passing through a 3×3×3 3D convolution layer in ;

[0119] 2.3. Using LSMSAformer module to analyze T x0 Processing is performed to obtain the global and local information of the low-resolution hyperspectral image in space. The specific steps are:

[0120] a. Enter the predefined zoom ratio s and x in Enter the LSMSAformer module;

[0121] And s is set to 4;

[0122] b. x in Projected into features through the projection layer;

[0123] c. Extract x using LSIB module in The local information W i , the specific steps are:

[0124] c1. Use two 2D convolutions to extract higher-level features from the input features in turn, and perform residual connections between the extracted features and the input features of the LSIB module to form a feature map P i ;

[0125] c2. The features extracted by the first 2D convolutional layer are further input into the SiLU activation function, which is a combination of the sigmoid function and the ReLU function. The low-resolution hyperspectral image extracted by the two convolutional layers is obtained through the sigmoid activation function symmetrical about the origin to obtain the attention map Q i ,

[0126] c3, feature map P i With the attention map Q i Perform multiplication operations between elements to obtain the final W i , and the calculation process of the i-th LSIB module can be expressed as:

[0127] P i =f 2dconv (f SiLU (f 2dconv (W i-1 )))+Wi-1

[0128] Q i =f sigmoid (f 2dconv (f SiLU (f 2dconv (W i-1 ))))

[0129] W i =P i ⊙Q i

[0130] Among them, W i-1 Indicates the output result of the previous up / down sampling module of the LSIB module.

[0131] f 2dconv (·) indicates W i-1 The 2d convolutional layer used after the input,

[0132] f SiLU (·) represents the SiLU activation function, which is a combination of the sigmoid function and the ReLU function.

[0133] f sigmoid (·) represents the sigmoid activation function,

[0134] ⊙ represents the multiplication operation between elements;

[0135] d. W through LSIB module i Input to the multi-head self-attention MSA module to obtain global information x MSA ;

[0136] e. The obtained x MSA and W i Connect them in parallel to get the output of each LSMSAformer module;

[0137] 2.4. In the LSMSAformer module, in During processing, the shallow feature F0 will also be input into the 3DDWTC module for processing to obtain the spectral information of the low-resolution hyperspectral image:

[0138] Among them, the 3DDWTC module is divided into two spectral attention modules, each spectral attention module is divided into two 3dCNN branches, and the upper 3dCNN branch consists of two 1×1×1 convolutions and one 3×3×3 convolution; and the 1×1×1 convolution is used for dimension increase and dimension reduction, and the 3×3×3 convolution is used for spatial-spectral feature extraction, while the lower branch consists of a DWT-3dCRU module for extracting local attention;

[0139] Each of the 3DDWTC modules obtains x in The specific process of obtaining the spectral information is as follows:

[0140] F out =f 1×1×1CNN (f 3×3×3CNN (f 1×1×1CNN (x in )))+f DWT-3dCRU (x in )+x in

[0141] Among them, f 1×1×1CNN is a 1×1×1 3d convolutional layer,

[0142] f 3×3×3CNN is a 3×3×3 3D convolutional layer,

[0143] F out It is the output of the entire 3DDWTC module, that is, the spectral information.

[0144] f DWT-3dCRU It is a function of all the steps included in the DWT-3dCRU module;

[0145] Among them, the channel reconstruction unit DWT-3dCRU module constructed by wavelet transform is introduced to fully obtain spectral information. Specifically, the channel reconstruction unit DWT-3dCRU module constructed by wavelet transform is introduced, and the specific steps are as follows:

[0146] A1. For x in Use the wavelet transform function to transform the original x in The input image of the C channel is converted into the high-frequency input of the 3C channel and the low-frequency input of the C channel, and then the number of channels of the low-frequency input and the high-frequency input is compressed to C;

[0147] A2. Control x with an extrusion ratio r in The number of input channels, and r is 1 / 2, after segmentation and squeezing, we get x up and x low ;

[0148] Among them, x up Refers to high-frequency information input, images with rich information,

[0149] x low Refers to low-frequency information input, images with sparse information;

[0150] The specific formula is as follows:

[0151] x up ,x low =f squeeze (f DWT(x in ))

[0152] Among them, f squeeze (·) represents the function of the extrusion channel,

[0153] f DWT (·) represents the wavelet transform function,

[0154] A3. x up Input to the upper stage, as a branch for extracting rich features, 3D group convolution 3D-GWC and 3D point-by-point convolution 3D-PWC are used instead of the ordinary k×K standard convolution on x up Process and obtain Y1;

[0155] A4, x low Input to the upper stage as a rare feature extraction branch, for the input feature x low Perform a simple 1×1×1 3D-PWC operation as x up The addition of x low The resulting feature map and x after the 3d-PWC operation low Perform parallel operation to obtain Y2. The specific formula is as follows:

[0156] Y1=f 3d-GWC (x up )+f 3d-PWC (x up )

[0157] Y2=f concat (f 3d-PWC (x up ),x low )

[0158] Among them, Y1 represents the feature map that extracts rich spectral information.

[0159] Y2 is the characteristic graph of sparse spectral information, which is used as a supplement to Y1.

[0160] f 3d-GWC (·) is the 3D group convolution operation,

[0161] f 3d-PWC (·) is the 3D point-by-point convolution operation,

[0162] f concat (·) is parallel operation;

[0163] A5. Perform an average pooling operation on Y1 and Y2, normalize them with the softmax function, and then multiply them with themselves to obtain the results out1 and out2. Use the inverse wavelet transform to restore the obtained out1 and out2, and finally output them as Y3. The specific formula is as follows:

[0164] out1=f softmax (f avgpool (Y1))·Y1

[0165] out2=f softmax (f avgpool (Y2))·Y2

[0166] F out =f IWT (out1,out2)

[0167] Among them, f softmax is the softmax function, which makes the gradient descent process smoother.

[0168] f avgpool is the average pooling operation,

[0169] out1 is the result of multiplying Y1 and Y1 after the average pooling operation of the softmax function, which is used to represent the image with rich spectral information after improving the spectral information;

[0170] out1 represents the result of multiplying Y2 and Y2 after the average pooling operation of the softmax function, which is used to represent the image with sparse spectral information after improving the spectral information.

[0171] In this step 2, when x up It is input to the upper stage as a feature-rich extraction branch. 3D grouped convolution 3D-GWC and 3D point-by-point convolution 3D-PWC are used instead of ordinary k×K standard convolution to process the information-rich feature map. Grouped convolution decomposes the traditional convolution kernel into multiple small convolution kernel groups, which can greatly reduce the number of model parameters. Each group is convolved through a small convolution kernel, which can better extract different aspects of the input features, thereby obtaining richer feature information. It greatly improves the efficiency and performance of extracting features of low-resolution hyperspectral images, and reduces complexity and computation. At the same time, grouped convolution can be applied to various types of network structures, improving applicability.

[0172] And because grouped convolution is also regarded as a sparse convolution, each output channel is only connected to a certain group of input channels, and there is no interaction between channels, cutting off the information flow between channel groups. Therefore, this step uses point-by-point convolution 3D-PWC to compensate for information loss and help information flow in feature channels. At the same time, dimensionality reduction is achieved by reducing the number of filters. Point-by-point convolution focuses on specific points, making the whole process more efficient and reducing the computational load. Point-by-point convolution does not require so many parameters, making the method of extracting features of low-resolution hyperspectral images simple and easy to manage.

[0173] In step 2.2, by performing 3D convolution extraction on the low-resolution hyperspectral image, the obtained shallow feature F0 will contain more position and detail information.

[0174] Step 3: Use the SWIU module to implement the interaction between the LSMSAformer module and the 3DDWTC module to generate the feature map F n :

[0175] 3.1. Use the SWIU module to interactively fuse the outputs of each LSMSAformer module and the 3DDWTC module.

[0176] 3.2. Combine the outputs of the nth LSMSAformer module and the nth 3DDWTC module to form the feature map F n .

[0177] In step 3, the SWIU module is used to realize the interactive fusion of Transformer and convolutional neural network. Transformer can capture the dependencies between each position in the input sequence through the self-attention mechanism, and can better model long-distance dependencies. However, the self-attention mechanism of the Transformer model requires a lot of computing resources and memory. For large-scale image data, huge models and computing power are required. In addition, for feature extraction based on the image pixel level, Transformer is not as effective as traditional convolutional neural network. Therefore, the interactive fusion of the two can capture the features of the image more comprehensively and achieve better super-resolution effect.

[0178] Step 4: The feature map F n Image reconstruction is performed to obtain a super-resolution hyperspectral image. The specific steps are as follows:

[0179] 4.1. Feature map F after fusing the LSMSAformer module and the 3DDWTC module n The 1×1×1 3D convolution is fed into the network and the residual connection is performed with the shallow feature F0. After a deconvolution and a 3×3×3 3D convolution, the super-resolution hyperspectral image is obtained after being magnified by the super-resolution multiple. SR Indicates that, and I SR The specific formula for the entire acquisition process is as follows:

[0180] I SR =f 3×3×3conv (f deconv (f 1×1×1conv (F n )+F0))

[0181] Among them, the function f deconv (·) represents the deconvolution layer;

[0182] 4.2. Since the L1 loss shows good convergence ability, we can use measurement I SR Optimize the pixel difference between the image and the ground truth SR ;

[0183] Among them, the specific formula of L1 loss is as follows:

[0184]

[0185] Among them, H TCC (·) is the network structure function,

[0186] N is the number of training image pairs,

[0187] Represents the ground truth image

[0188] In step 4, the loss function is a function that maps the value of a random event or its related random variable to a non-negative real number to characterize the risk or loss degree of the random event; the smaller the loss function, the closer the predicted value of the model is to the true value, and the better the robustness and accuracy of the model;

[0189] In the application of hyperspectral super-resolution in this application, the loss function is used to measure the difference between the model prediction value and the true value, that is, the model is solved and evaluated by minimizing the loss function. The loss function is used to guide the optimization of model parameters during the training process of this application to minimize the prediction error and improve the accuracy and generalization ability of this application.

[0190] Verification test

[0191] like Figure 2-3 As shown, Figure 2 To use the super-resolution result image processed by this application, Figure 3 , that is, the original image is compared and it is found that the two images are relatively close. Therefore, it can be considered that the image after the experiment is similar to the original image. The method of the present application can better reconstruct the blurred image.

[0192] In summary, the present invention proposes a novel network structure of the interaction of Transformer and convolutional neural network for hyperspectral image super-resolution, extracts the global information and local information of low-resolution hyperspectral images in space through the LSMSAformer module, and adopts the local spatial information extraction LSIB module and the long-distance spatial information extraction MSA module in the Transformer branch, extracts the spectral information of low-resolution hyperspectral images through the 3DDWTC module, and adopts the 3D convolution and local spectral attention extraction DWT-3dCRU module in the convolutional neural network branch, and integrates the global spatial information and local spatial information at different scales in the LSMSAformer module, so that the entire low-resolution hyperspectral image can restore the high spatial resolution while also restoring the detail information in the image space, that is, the global information and the local information;

[0193] At the same time, in the DWT-3dCRU module of the convolutional neural network branch, the input image is divided into high-frequency input and low-frequency input using wavelet transform, and 3D group convolution and 3D point convolution are used to process feature-rich images and feature-sparse images separately, achieving high efficiency and reducing the number of parameters when extracting spectral information;

[0194] The shared weight SWIU module between parallel branches better fuses the global and local information in space and spectrum in an adaptive manner, thereby obtaining a better reconstructed image.

[0195] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. An interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information, characterized in that: The following steps are involved: Step 1: Collect a data set for hyperspectral image super-resolution, process the data set, and generate a low-resolution hyperspectral image; Step 2: Input the training set into the LSMSAformer module on the Transformer branch and the 3DDWTC module on the CNN branch to extract the spatial and spectral features of the low-resolution hyperspectral image respectively; Step 3: Use the SWIU module to implement the interaction between the LSMSAformer module and the 3DDWTC module to generate the feature map F n ; Step 4: The feature map F n Perform image reconstruction to obtain a super-resolution hyperspectral image; In step 1, the data set for super-resolution of hyperspectral images is collected, and the data set is processed to generate low-resolution hyperspectral images. The specific process is as follows: Dataset division: First, divide the dataset into training set and test set; Apply Gaussian blur: Apply a 5×5 Gaussian blur to the data within the dataset with a standard deviation of 2 and a mean of 0 to generate a blurred image. Downsampling: Downsample the blurred image by a factor of 4 to generate a low-resolution hyperspectral image.

2. The interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information according to claim 1, characterized in that: In step 2, the training set is input into the LSMSAformer module on the Transformer branch and the 3DDWTC module on the CNN branch to extract the spatial and spectral features of the low-resolution hyperspectral image, respectively. The specific steps are as follows: 2.

1. Let the low-resolution hyperspectral image be x in ,and Where H is the height of the low-resolution hyperspectral image, W is the width of the low-resolution hyperspectral image, C is the number of channels of the low-resolution hyperspectral image; 2.

2. Use a 3D convolution with a kernel size of 3×3×3 from x in Extract the shallow feature F0, reshape the shallow feature F0 into 4D data, and then pass it into the LSMSAformer module as input to get T x0 , the specific formula is as follows: F0=f 3Dconv (x) in ) T x0 =f to4D (F0) Among them, the function f to4D The function of (·) is to convert the size of N×D×C×H×W into (N×D)×C×H×W. N represents the batch size, D represents the embedding latitude; T x0 x resized to fit the input dimensions of the LSMSAformer module branches in ; f 3Dconv (x in ) is the shallow features of x extracted after passing through a 3×3×3 3D convolution layer in ; 2.

3. Using LSMSAformer module to analyze T x0 Processing is performed to obtain global and local information of low-resolution hyperspectral images in space; 2.

4. In the LSMSAformer module, in During processing, the shallow feature F0 will also be input into the 3DDWTC module for processing to obtain the spectral information of the low-resolution hyperspectral image.

3. The interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information according to claim 2, characterized in that: In step 3, the SWIU module is used to implement the interaction between the LSMSAformer module and the 3DDWTC module to generate the feature map F n The specific steps are: 3.

1. Use the SWIU module to interactively fuse the outputs of each LSMSAformer module and the 3DDWTC module. 3.

2. Combine the outputs of the nth LSMSAformer module and the nth 3DDWTC module to form the feature map F n .

4. The interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information according to claim 2, characterized in that: In step 2.3: The specific steps of each LSMSAformer module in extracting the global information and local information of the low-resolution hyperspectral image in space are: a. Enter the predefined zoom ratio s and x in Enter the LSMSAformer module; And s is set to 4; b. x in Projected into features through the projection layer; c. Extract x using LSIB module in The local information W i , the specific steps are: c1. Use two 2D convolutions to extract higher-level features from the input features in turn, and perform residual connections between the extracted features and the input features of the LSIB module to form a feature map P i ; c2. The features extracted by the first 2D convolutional layer are further input into the SiLU activation function, which is a combination of the sigmoid function and the ReLU function. The low-resolution hyperspectral image extracted by the two convolutional layers is obtained through the sigmoid activation function symmetrical about the origin to obtain the attention map Q i , c3, feature map P i With the attention map Q i Perform multiplication operations between elements to obtain the final W i , and the calculation process of the i-th LSIB module can be expressed as: P i =f 2dconv (f SiLU (f 2dconv (W i-1 )))+W i-1 Q i =f sigmoid (f 2dconv (f SiLU (f 2dconv (W i-1 )))) W i =P i ⊙Q i Among them, W i-1 Indicates the output result of the previous up / down sampling module of the LSIB module. f 2dconv (·) indicates W i-1 The 2d convolutional layer used after the input, f SiLU (·) represents the SiLU activation function, which is a combination of the sigmoid function and the ReLU function. f sigmoid (·) represents the sigmoid activation function, ⊙ represents the multiplication operation between elements; d. W through LSIB module i Input to the multi-head self-attention MSA module to obtain global information x MSA ; e. The obtained x MSA and W i Connect them in parallel to get the output of each LSMSAformer module.

5. The interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information according to claim 2, characterized in that: In step 2.4: each of the 3DDWTC modules obtains x in The specific process of obtaining the spectral information is as follows: F out =f 1×1×1CNN (f 3×3×3CNN (f 1×1×1CNN (x in )))+f DWT-3dCRU (x in )+x in Among them, f 1×1×1CNN is a 1×1×1 3d convolutional layer, f 3×3×3CNN is a 3×3×3 3D convolutional layer, F out is the spectral information, f DWT-3dCRU It is a function of all the steps included in the DWT-3dCRU module; Among them, the channel reconstruction unit DWT-3dCRU module constructed by wavelet transform is introduced to fully obtain spectral information, and the channel reconstruction unit DWT-3dCRU module constructed by wavelet transform is introduced. The specific steps are as follows: A1. For x in Use the wavelet transform function to transform the original x in The input image of the C channel is converted into the high-frequency input of the 3C channel and the low-frequency input of the C channel, and then the number of channels of the low-frequency input and the high-frequency input is compressed to C; A2. Control x with an extrusion ratio r in The number of input channels, and r is 1 / 2, after segmentation and squeezing, we get x up and x low ; Among them, x up Refers to high-frequency information input, images with rich information, x low Refers to low-frequency information input, images with sparse information; The specific formula is as follows: x up ,x low =f squeeze (f DWT (x in )) Among them, f squeeze (·) represents the function of the extrusion channel, f DWT (·) represents the wavelet transform function, A3. x up Input to the upper stage, as a branch for extracting rich features, 3D group convolution 3D-GWC and 3D point-by-point convolution 3D-PWC are used instead of the ordinary k×K standard convolution on x up Process and obtain Y1; A4, x low Input to the upper stage as a rare feature extraction branch, for the input feature x low Perform a simple 1×1×1 3D-PWC operation as x up The addition of x low The resulting feature map and x after the 3d-PWC operation low Perform parallel operation to obtain Y2. The specific formula is as follows: Y1=f 3d-GWC (x up )+f 3d-PWC (x up ) Y2=f concat (f 3d-PWC (x) up ),x low ) Among them, Y1 represents the feature map that extracts rich spectral information. Y2 is the characteristic graph of sparse spectral information, which is used as a supplement to Y1. f 3d-GWC (·) is the 3D group convolution operation, f 3d-PWC (·) is the 3D point-by-point convolution operation, f concat (·) is parallel operation; A5. Perform an average pooling operation on Y1 and Y2, normalize them with the softmax function, and then multiply them with themselves to obtain the results out1 and out2. Use the inverse wavelet transform to restore the obtained out1 and out2, and finally output them as Y3. The specific formula is as follows: out1=f softmax (f avgpool (Y1))·Y1 out2=f softmax (f avgpool (Y2))·Y2 F out =f IWT (out1,out2) Among them, f softmax is the softmax function, f avgpool is the average pooling operation, out1 is the result of multiplying Y1 and Y1 after the average pooling operation of the softmax function, which is used to represent the image with rich spectral information after improving the spectral information; out1 represents the result of multiplying Y2 and Y2 after the average pooling operation of the softmax function, which is used to represent the image with sparse spectral information after improving the spectral information.

6. The interactive hyperspectral image super-resolution method for simultaneously extracting spatial information and spectral information according to claim 1, characterized in that: In step 4, the feature map F n Perform image reconstruction to obtain a super-resolution hyperspectral image. The specific steps are as follows: 4.

1. Feature map F after fusing the LSMSAformer module and the 3DDWTC module n The 1×1×1 3D convolution is fed into the network and the residual connection is performed with the shallow feature F0. After a deconvolution and a 3×3×3 3D convolution, the super-resolution hyperspectral image is obtained after being magnified by the super-resolution multiple. SR Indicates that, and I SR The specific formula for the entire acquisition process is as follows: I SR =f 3×3×3conv (f deconv (f 1×1×1conv (F n )+F0)) Among them, the function f deconv (·) represents the deconvolution layer; 4.

2. Measuring I using L1 loss (mean absolute error) SR Optimize the pixel difference between the image and the ground truth SR ; Among them, the specific formula of L1 loss is as follows: Among them, H TCC (·) is the network structure function, N is the number of training image pairs, represents the ground truth image.

Images