A method for infrared spectrum reconstruction based on convolutional neural network

Abstract

The present invention relates to a method for reconstructing infrared spectral images based on convolutional neural networks, establishing a SE-HSCNN+convolutional neural network model; inputting multi-channel infrared spectral data, preprocessing, and obtaining three-channel image data to be reconstructed; inputting the three-channel data into the network model, and obtaining shallow feature information through a feature extraction layer; inputting the obtained shallow feature information into an end-to-end mapping model composed of eight dense modules in sequence, and continuously performing feature channel fusion; recalibrating all feature channels to enhance the learned channel features; finally, outputting the infrared spectral image through a reconstruction layer to obtain a reconstructed image. The convolutional neural network model proposed in the present invention achieves better infrared spectral reconstruction results, with significant improvements in both subjective vision and objective evaluation indicators, significantly improved image clarity and brightness, faster convergence speed, and higher advantages in terms of precision.

Description

Technical Field

[0001] The invention relates to the field of hyperspectral imaging and digital image processing, and in particular to an infrared spectrum reconstruction method based on a convolutional neural network. Background Art

[0002] Infrared spectral image reconstruction is a classic problem in the field of hyperspectral imaging. Its purpose is to obtain a multi-band infrared image from a few-band image. Through signal processing and image processing methods, a hyperspectral infrared image with the most information can be reconstructed. For any given few-band infrared image, there are multiple solutions. This is a typical ill-posed inverse problem with serious ill-posedness. Solving this problem requires prior information.

[0003] Convolutional neural networks (CNNs) can obtain high-frequency features of images, enhance detail information, and are particularly good at obtaining correlation information between image channels. It is very necessary to study their application in image reconstruction. Convolutional neural networks are applied to infrared spectral image reconstruction and have achieved good results.

[0004] The algorithm HSCNN+ (Hyper-Spectral Convolutional Neural Network) establishes an end-to-end reconstruction network. However, the network structure of HSCNN+ cannot meet the requirements of small infrared image data volume and higher accuracy. According to the characteristics of convolutional neural networks, for networks with relatively small data volume, overfitting will occur during the reconstruction process and deeper information cannot be obtained. Only by reducing the number of convolutional network layers and recalibrating the feature channel information can more detailed features be obtained and multi-information infrared images can be reconstructed. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides an infrared image reconstruction method based on convolutional neural network, which overcomes the shortcomings of the current single image convolutional neural network technology in processing infrared spectrum reconstruction.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a convolutional neural network-based infrared spectrum reconstruction method, comprising the following steps:

[0007] Step 1: Establish a convolutional neural network model consisting of a feature extraction layer, a feature mapping layer, a feature channel enhancement layer, and a feature reconstruction layer;

[0008] Step 2: Preprocess the multi-channel infrared spectrum data to obtain three-channel image data;

[0009] Step 3: Input the three-channel data into the convolutional neural network model and obtain shallow feature information through the feature extraction layer;

[0010] Step 4: Input the shallow feature information obtained into the feature mapping layer to perform feature channel fusion;

[0011] Step 5: Recalibrate the obtained feature channels through the feature channel enhancement layer to enhance the learned channel features;

[0012] Step 6: Output the infrared spectrum image through the feature reconstruction layer to obtain the reconstructed image.

[0013] The establishment of the convolutional neural network model includes the following steps:

[0014] Establish feature extraction layer: construct parallel dual-path convolutional layers; the convolution kernel size of the first convolution layer of each path is 3x3, the number of convolution kernels is 16, the convolution kernel size of the second convolution layer is 1x1, the number of convolution kernels is 16; the output of the second convolution layer is used as the input of the feature mapping layer;

[0015] Establish a feature mapping layer, including eight dense modules connected sequentially;

[0016] Establish a feature channel enhancement layer, including the average pooling layer, the first fully connected layer, the ReLU layer, the second fully connected layer, the Sigmoid layer, and the Scale layer connected in sequence; the output of the feature mapping model is used as the input of the average pooling layer, and the Scale layer outputs to the feature reconstruction layer;

[0017] A feature reconstruction layer is established, and one convolution layer is used as the feature reconstruction layer. The convolution kernel size of the convolution layer is 1x1, and the number of convolution kernels is 29.

[0018] The dense module is constructed as follows:

[0019] The convolution kernel size of the fifth convolution layer is 1x1, and the number of convolution kernels is 64; the fifth convolution layer outputs to the convolution layers of two parallel paths, and then outputs to the tenth convolution layer; the convolution kernel size of the tenth convolution layer is 1x1, and the number of convolution kernels is 16;

[0020] Among them, the parallel path includes: the sixth convolution layer and the seventh convolution layer connected in sequence; the convolution kernel size of the sixth convolution layer is 3x3, and the number of convolution kernels is 16; the convolution kernel size of the seventh convolution layer is 1x1, and the number of convolution kernels is 8.

[0021] The pre-processing comprises:

[0022] According to the principle of wavelength averaging, the infrared spectrum image selects three channel image data as input, and the remaining channel images are used as corresponding label data.

[0023] In the feature extraction layer and feature mapping layer, the linear rectification unit ReLU is used instead of Sigmoid as the activation function.

[0024] The global parameters of the feature mapping model are set as follows: the learning rate decays exponentially from 0.001 at the beginning to 0.0001, the momentum factor momentum=0.9, and the batch size batch_size=6.

[0025] In the training of the convolutional neural network model, the weight parameters w and b are updated. First, they are initialized using a normal distribution with a mean of 0 and a variance of 0.01; secondly, the stochastic gradient descent method is used to complete the update.

[0026] In the process of training the feature mapping layer through the data set, the loss function is set, which uses the root mean square error:

[0027]

[0028] Where n represents the number of training samples, Y i is the label data, representing the original infrared spectrum image block, Y i ′ is the reconstructed infrared spectrum image block.

[0029] The present invention has the following beneficial effects and advantages:

[0030] The present invention can effectively handle the infrared spectrum reconstruction problem of few-band images. The network model can accurately learn the significant features of the image and finally give a reconstructed infrared image with multiple information.

[0031] The convolutional neural network model proposed in the present invention achieves better infrared spectrum reconstruction results, with obvious improvements in both subjective vision and objective evaluation indicators. The image clarity and brightness are significantly improved, the convergence speed is faster, and it has a higher advantage in terms of fineness. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a structural diagram of eight dense module network models based on feature channel fusion of the present invention; DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0034] A convolutional neural network-based infrared spectrum reconstruction method comprises the following steps:

[0035] Step 1: Establish SE-HSCNN+ convolutional neural network model;

[0036] Step 2: Input multi-channel infrared spectrum data and pre-process the image data to obtain three-channel image data;

[0037] Step 3: Input the three-channel data into the network model and obtain shallow feature information through the feature extraction layer;

[0038] Step 4: Input the shallow feature information obtained into the end-to-end mapping model composed of eight dense attention modules in sequence, and continuously perform feature channel fusion;

[0039] Step 5: Recalibrate all feature channels to enhance the learned channel features;

[0040] Step 6: Output the infrared spectrum image through the reconstruction layer to obtain a reconstructed image. The preprocessing includes:

[0041] According to the principle of wavelength equipartition, the infrared spectrum image is divided into three channels and the image data are selected as input, and the remaining channel images are used as the corresponding label data.

[0042] In the feature extraction and feature mapping parts, the linear rectified unit ReLU is used instead of Sigmoid as the activation function.

[0043] The data set includes a data set collected by oneself.

[0044] The end-to-end neural network model composed of eight dense modules based on the attention mechanism, each dense module includes a convolution layer, two convolution layers of dual paths, a feature fusion layer, a convolution layer and a feature fusion layer, and does not include a pooling layer and a batch normalization layer; the attention module includes an average pooling layer, a fully connected layer, a ReLU layer, a fully connected layer, a Sigmoid layer, and a Scale layer.

[0045] The global parameters of the end-to-end neural network model composed of eight dense modules based on the attention mechanism are set as follows: the learning rate decays exponentially from 0.001 at the beginning to 0.0001, momentum = 0.9, batch_size = 6;

[0046] The parameters of the feature extraction part are set as follows: the convolution kernel size of the convolution layer of the first parallel path of the dual path is 3x3, the number of convolution kernels is 16, the convolution kernel size of the second convolution layer is 1x1, and the number of convolution kernels is 16; the parameters of the second path are the same as those of the first path. The parameters of the feature mapping part are set as the same parameters of the eight dense modules, and the parameters of each module are set as follows: the convolution kernel size of the first convolution layer is 1x1, and the number of convolution kernels is 64; the convolution kernel size of the second convolution layer of the first path of the parallel path is 3x3, and the number of convolution kernels is 16; the convolution kernel size of the third convolution layer is 1x1, and the number of convolution kernels is 8; the convolution kernel parameters of the other path of the parallel path are the same as those of the first path, and the convolution kernel size of the fourth convolution layer is 1x1, and the number of convolution kernels is 16; the parameters of the attention module part are: the reduction factor a is 16; the parameters of the reconstruction layer part are set as follows: the convolution kernel size of the convolution layer is 1x1, and the number of convolution kernels is 29.

[0047] The three-channel image is input into the feature extraction module of the neural network model: after feature mapping, information after feature fusion is obtained, and then the extracted feature information is recalibrated for channel features, the learned features are added, and the reconstructed image channel information is obtained.

[0048] The weight parameter w and bias parameter b are updated. First, they are initialized and randomly selected from a normal distribution with a mean of 0 and a variance of 0.01. Then, the stochastic gradient descent method is used to complete the update.

[0049] In the process of training the end-to-end neural network model composed of eight dense modules based on the attention mechanism through the data set, the loss function is set, which uses the root mean square error:

[0050]

[0051] Where n represents the number of training samples, Y i is the original infrared spectrum image block, Y i ′ is the reconstructed infrared spectrum image block.

[0052] The present invention comprises the following process:

[0053] Firstly, according to the principle of wavelength equipartition, three channel image data are selected as input for the infrared spectrum image, and the remaining channel images are used as corresponding label data; then, a dense neural network structure with an attention module is designed, the size and number of convolution kernels in each layer are determined, and CNN is optimized, including the optimization of initialization parameters weights and bias, as well as the optimization of activation function; finally, shallow feature information is extracted through feature extraction operation and input into the dense network of the attention mechanism proposed in the present invention, the dense network fuses the feature information, the attention module strengthens the extracted feature channel information, and the feature reconstruction layer completes the final infrared image reconstruction.

[0054] like Figure 1 Shown is a structural diagram of eight dense module network models based on the attention mechanism of the present invention.

[0055] The reconstruction algorithm proposed in the present invention establishes an end-to-end model SE-HSCNN+ (Squeeze and Excitation Hyper-Spectral using Convolutional Neural Networks), the input is a few-band image, and the output is a multi-band image. Since image detail information is required, if there is downsampling, the extracted texture features will be reduced, so no pooling layer is introduced in feature extraction and feature mapping. The model is a fully convolutional structure, which is composed of a feature extraction layer, a feature mapping layer, a feature recalibration layer, and a channel reconstruction layer. Among them, 4 convolutional layers are used as feature extraction layers, 8 dense modules are used as feature mapping layers, each dense attention module has 6 convolutional layers, average pooling layers, fully connected layers, ReLU layers, fully connected layers, Sigmoid layers, Scale layers are used as feature enhancement layers, and 1 convolutional layer is used as a feature reconstruction layer.

[0056] An eight-density force module network model based on the attention mechanism was established. The first 4 layers of the network obtain shallow feature information through feature extraction, and the last 48 layers realize feature mapping. The feature enhancement layer is used to recalibrate the feature channels of the fused feature information, and the last layer is used to reconstruct the image to form an image with multi-channel information. Spectral reconstruction refers to improving the band information of the original image through hardware or software methods. The process of obtaining a multi-channel image through a series of images with fewer bands is spectral reconstruction.

[0057] The specific steps are as follows:

[0058] Usually, the spectral reconstruction of a single image involves a problem: for a given image X with few bands, how to reconstruct an image Y with more bands? The observed X is the image of Y collected under the sensor. This problem can be expressed by the following formula:

[0059] Y=QX+n

[0060] Where Q represents the response matrix of the system and n represents the random noise of the system.

[0061] Step 1. Image preprocessing.

[0062] When performing image preprocessing, the infrared spectrum data is first divided into three equal parts according to the principle of wavelength, and images of three channels are selected as input information, and the images of the remaining channels are used as label information.

[0063] Step 2. Input image data.

[0064] The three-band image data to be reconstructed is input into the spectral reconstruction system based on the attention mechanism.

[0065] Step 3. Optimize the convolutional neural network activation function.

[0066] ReLU (rectified linear unit) can solve the gradient vanishing problem by replacing Sigmoid as the activation function. The expression of ReLU is max(0, x+N(0,1)). The ReLU function has the characteristics of unilateral inhibition and sparse activation, which is closer to the activation state of brain neurons receiving signals. The use of the activation function ReLU introduces sparsity to the convolutional neural network, which is equivalent to introducing the preprocessing of unsupervised learning.

[0067] Let X be the first layer input, Y l is the output of the lth layer, then under the ReLU activation function, the relationship between the first input and output is:

[0068] Y l =max(0,w l *X+b l )

[0069] The subsequent input-output relationship of the l+1th layer is:

[0070] Y l+1 =max(0,w l *X+b l )

[0071] where w l With b l The input and output correspond to the weight and bias of the lth convolutional layer respectively, and “*” indicates the convolution operation.

[0072] Step 4. Model parameter setting.

[0073] The traditional method for initializing weights and biases is to randomly select values ​​from a normal distribution that obeys N(0,1). Inappropriate initial values ​​will cause the neural network to take too long to learn, seriously affecting the system's convergence speed and may also lead to falling into a local optimal value.

[0074] The improved algorithm is from a normal distribution with a mean of 0 and a variance of 0.01. Still using the hypothetical neural network, after the improved algorithm, the experiment verifies that the new initialization weight method converges significantly faster than the traditional method, and also has a certain improvement in accuracy. The network uses standard back propagation and stochastic gradient descent to complete the training process to minimize the loss. The algorithm uses the gradient descent method for convergence. In the gradient descent method, both the weight w and the bias b are updated; the stochastic gradient descent method reduces the value of the loss function and solves the entire neural network.

[0075] Step 5. Loss function setting.

[0076] In order to better train the network, the mean square error (MSE) is used as the loss function. MSE is closer to human visual perception, and objective experimental results show that the image quality obtained using MSE is also significantly improved. The mathematical form of MSE is:

[0077]

[0078] Here n represents the number of training samples. The physical meaning of MSE is to calculate the original infrared spectrum image block Y i The reconstructed spectral image patch Y output by the network i ′The difference of the center pixel.

[0079] Step 6. Design parameters of each layer of convolutional neural network.

[0080] During training, the global parameters were set as follows: the learning rate decayed exponentially from 0.001 at the beginning to 0.0001, momentum = 0.9, batch_size = 6. The convolution kernel size of the convolution layer of the first parallel path of the dual path is 3x3, the number of convolution kernels is 16, the convolution kernel size of the second convolution layer is 1x1, the number of convolution kernels is 16; the parameters of the second path are the same as those of the first path. The convolution kernel size of the fifth convolution layer is 1x1, and the number of convolution kernels is 64; the convolution kernel size of the sixth convolution layer of the first path of the parallel path is 3x3, the number of convolution kernels is 16; the convolution kernel size of the seventh convolution layer is 1x1, and the number of convolution kernels is 8; the convolution kernel parameters of the other path of the parallel path are the same as those of the first path, the convolution kernel size of the tenth convolution layer is 1x1, and the number of convolution kernels is 16; the reduction factor a is 16; the convolution kernel size of the eleventh convolution layer is 1x1, and the number of convolution kernels is 29.

[0081] Step 7. Feature extraction and fusion.

[0082] After extracting shallow feature information of the image through the first four layers of the SE-HSCNN+ network, feature mapping is performed, and the learned channel features are fused through the last 48 layers of convolution operations of SE-HSCNN+. The attention mechanism can enhance the image channel details and improve the reconstruction effect.

[0083] Step 8. Reconstruction of infrared spectrum.

[0084] After training the eight dense module network models based on the attention mechanism, the input is a three-channel infrared image and the output is a multi-channel infrared spectral image.

Claims

1. An infrared spectrum reconstruction method based on convolutional neural network, characterized in that: The following steps are involved: Step 1: Establish a convolutional neural network model consisting of a feature extraction layer, a feature mapping layer, a feature channel enhancement layer, and a feature reconstruction layer; Step 2: Preprocess the multi-channel infrared spectrum data to obtain three-channel image data; Step 3: Input the three-channel data into the convolutional neural network model and obtain shallow feature information through the feature extraction layer; Step 4: Input the shallow feature information obtained into the feature mapping layer to perform feature channel fusion; Step 5: Recalibrate the obtained feature channels through the feature channel enhancement layer to enhance the learned channel features; Step 6: Output the infrared spectrum image through the feature reconstruction layer to obtain a reconstructed image; The convolutional neural network model is established. The following steps are involved: Establish feature extraction layer: construct parallel dual-path convolutional layers; the convolution kernel size of the first convolution layer of each path is 3x3, the number of convolution kernels is 16, the convolution kernel size of the second convolution layer is 1x1, the number of convolution kernels is 16; the output of the second convolution layer is used as the input of the feature mapping layer; Establish a feature mapping layer, including eight dense modules connected sequentially; Establish a feature channel enhancement layer, including the average pooling layer, the first fully connected layer, the ReLU layer, the second fully connected layer, the Sigmoid layer, and the Scale layer connected in sequence; the output of the feature mapping model is used as the input of the average pooling layer, and the Scale layer outputs to the feature reconstruction layer; A feature reconstruction layer is established, and one convolution layer is used as the feature reconstruction layer. The convolution kernel size of the convolution layer is 1x1, and the number of convolution kernels is 29.

2. The infrared spectrum reconstruction method based on convolutional neural network according to claim 1, characterized in that: The dense module is constructed as follows: The convolution kernel size of the fifth convolution layer is 1x1, and the number of convolution kernels is 64; the fifth convolution layer outputs to the convolution layers of two parallel paths, and then outputs to the tenth convolution layer; the convolution kernel size of the tenth convolution layer is 1x1, and the number of convolution kernels is 16; Among them, the parallel path includes: the sixth convolution layer and the seventh convolution layer connected in sequence; the convolution kernel size of the sixth convolution layer is 3x3, and the number of convolution kernels is 16; the convolution kernel size of the seventh convolution layer is 1x1, and the number of convolution kernels is 8.

3. The infrared spectrum reconstruction method based on convolutional neural network according to claim 1, characterized in that: The pre-processing comprises: According to the principle of wavelength averaging, the infrared spectrum image selects three channel image data as input, and the remaining channel images are used as corresponding label data.

4. The infrared spectrum reconstruction method based on convolutional neural network according to claim 2, characterized in that: In the feature extraction layer and feature mapping layer, the linear rectification unit ReLU is used instead of Sigmoid as the activation function.

5. The infrared spectrum reconstruction method based on convolutional neural network according to claim 4, characterized in that: The global parameters of the feature mapping model are set as follows: the learning rate decays exponentially from 0.001 at the beginning to 0.0001, the momentum factor momentum=0.9, and the batch size batch_size=6.

6. The infrared spectrum reconstruction method based on convolutional neural network according to claim 1, characterized in that: In the training of the convolutional neural network model, the weight parameters w and b are updated. First, they are initialized using a normal distribution with a mean of 0 and a variance of 0.01; secondly, the stochastic gradient descent method is used to complete the update.

7. The infrared spectrum reconstruction method based on convolutional neural network according to claim 1, characterized in that: In the process of training the feature mapping layer through the data set, the loss function is set, which uses the root mean square error: Where n represents the number of training samples, Y i is the label data, representing the original infrared spectrum image block, Y i ′ is the reconstructed infrared spectrum image block.

Images