Multispectral reconstruction method based on RGB image and attention residual network

Abstract

The invention provides a multispectral reconstruction method based on an RGB image and an attention residual network, and the method comprises the steps: obtaining a natural color image corresponding to the red, green and blue wavebands of satellite multispectral data through a deep learning method; a natural color-to-multispectral network model capable of predicting red-green-blue, red-edge and near-infrared multispectral data is obtained from a natural color image corresponding to red, green and blue wavebands, so that a multispectral image is reconstructed by using a low-cost RGB image, and the problem of lack of a data set is solved; a channel attention mechanism is fused on the basis of the residual network model, so that the reconstruction precision of the multispectral image is improved; a data set with various ground feature types is used for training, and the problem that the generalization ability of the model is insufficient is solved; on the basis of preventing gradient disappearance caused by too deep network, different contribution degrees of different features of the image to multispectral reconstruction are emphasized, the channel is weighted, the error between a reconstruction value and a true value is reduced, and the multispectral reconstruction precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of spectral reconstruction, and in particular relates to a multi-spectral reconstruction method based on RGB images and an attention residual network. Background Art

[0002] With the rapid development of computer vision technology, multispectral images are widely used in agriculture, remote sensing, medical and other fields, such as monitoring crop health, identifying pests and diseases, implementing precision irrigation and variable fertilization, monitoring changes in land use and vegetation cover, and for medical diagnosis and identification of different cells and tissues. Multispectral imaging technology is widely used and has great development potential. Multispectral images perform well in areas such as image classification and target recognition, and outperform RGB images in many tasks. However, the cost of multispectral images is higher than that of RGB images. Compared with traditional RGB sensors, multispectral sensors are expensive and have lower spatial resolution.

[0003] In order to reduce costs, multispectral reconstruction methods are often used to obtain multispectral images from RGB images. Spectral reconstruction algorithms are mainly divided into two methods: based on prior knowledge and based on deep learning. Compared with traditional methods based on prior knowledge, the application of deep learning technology in multispectral image reconstruction has gradually become a research hotspot. However, multispectral reconstruction algorithms face the lack of training data sets, the types of objects in the data sets are single, and most multispectral reconstruction algorithms are applied to the visible light range, and the possibility of reconstructing bands outside the visible light has not been explored. Therefore, it is urgent to develop multispectral reconstruction methods for a wide range of object types and a wide wavelength coverage. Summary of the invention

[0004] In order to solve the problems of lack of natural color data sets corresponding to multispectral images and insufficient model generalization ability during model training, the present invention provides a multispectral reconstruction method based on RGB images and attention residual network. The channel attention mechanism is integrated on the basis of the residual network model, and a data set with various types of objects is used for training, which can improve the accuracy of multispectral reconstruction.

[0005] A multispectral reconstruction method based on RGB image and attention residual network, inputting the natural color image of the scene to be reconstructed into the trained natural color to multispectral network model to obtain the multispectral image of the scene to be reconstructed, wherein the training method of the natural color to multispectral network model is:

[0006] S1: Convert the hyperspectral images in the hyperspectral dataset into natural color images that match those taken by the camera according to the CIE color matching function;

[0007] S2: The red, green and blue bands in the hyperspectral dataset are used as true color images and as the input of the residual network. The natural color image obtained in step S1 is used as the theoretical output of the residual network, and the residual network is trained to obtain a true color to natural color network model based on the residual network.

[0008] S3: input the red, green and blue three bands in the satellite multispectral data set as the true color image to be tested into the true color to natural color network model obtained in step S2, and obtain the natural color prediction image of the red, green and blue three bands in the satellite multispectral data set;

[0009] S4: The natural color predicted image obtained in step S3 is used as the input of the residual network fused with the attention mechanism, and the red, green, blue, red edge, and near infrared in the satellite multispectral dataset are used as the theoretical outputs of the residual network fused with the attention mechanism. The residual network fused with the attention mechanism is trained to obtain a natural color to multispectral network model based on the residual network and the attention mechanism.

[0010] Furthermore, the natural color to multi-spectral network model includes a sequentially cascaded input layer, an input convolutional layer, a first residual block, a second residual block, a third residual block, an attention mechanism compression and excitation module, a connection layer, an activation layer, and an output convolutional layer, wherein the connection layer is used to fuse the output of the attention mechanism compression and excitation module and the input convolutional layer.

[0011] Furthermore, any residual block includes an input layer, a first convolutional layer, a first activation layer, a second convolutional layer, a connection layer and a second activation layer, wherein the connection layer is used to fuse the output of the input layer and the second convolutional layer.

[0012] Furthermore, the processing process of the attention mechanism compression and excitation module on the output feature map of the third residual block is as follows:

[0013] Pooling is performed on the output feature map of size h×w×c, compressing the information of each channel of the output feature map into a single value, and obtaining a global compressed feature of size 1×1×c corresponding to each channel of the output feature map;

[0014] Through the structure of fully connected layer-ReLU nonlinear activation layer-fully connected layer-Sigmoid layer, each global compression feature is first stimulated separately to obtain the weight of each channel in the output feature map; then the output feature map is weighted according to the weight corresponding to each channel to obtain the processed fusion feature map;

[0015] Among them, the first fully connected layer compresses c channels into c / r channels, then passes through a ReLU nonlinear activation layer, the second fully connected layer restores the number of channels back to c channels, and finally activates the Sigmoid layer to obtain the normalized weights, and r is the set compression ratio.

[0016] Furthermore, the hyperspectral image in the hyperspectral dataset is converted into a natural color image that conforms to the camera according to the CIE color matching function:

[0017] Calculate the color matching function and select the corresponding color matching function curve;

[0018] Perform weighted summation on each channel of the hyperspectral image according to the color matching function, and transform the result of the weighted summation from the XYZ color space to the RGB color space through the conversion matrix;

[0019] The image data in the RGB color space is gamma corrected to make the image consistent with the effect of shooting with an RGB camera, and a natural color image that conforms to the camera is obtained.

[0020] Beneficial effects:

[0021] The present invention provides a multispectral reconstruction method based on RGB image and attention residual network. Firstly, natural color RGB image is obtained from hyperspectral image by using CIE standard chromaticity theory. Then, a true color to natural color conversion model based on convolutional neural network is trained by using natural color RGB image and true color RGB image pair. The model is applied to true color image of multispectral data set to obtain corresponding natural color image. Finally, a natural color to multispectral image conversion model based on residual network and attention mechanism is trained by using the obtained natural color and multispectral image pair. It can be seen that the present invention obtains red, green, blue and blue of satellite multispectral data by deep learning method. The natural color images corresponding to the three bands are used to obtain the natural color to multispectral network model that can predict the full-band multispectral data from the natural color images corresponding to the red, green and blue bands, so that the multispectral images are reconstructed using low-cost RGB images, which solves the problem of lack of data sets; the channel attention mechanism is integrated on the basis of the residual network model to improve the reconstruction accuracy of the multispectral images; data sets with diverse types of objects are used for training to solve the problem of insufficient generalization ability of the model; on the basis of preventing the network from being too deep and causing the gradient to disappear, attention is paid to the different contributions of different image features to multispectral reconstruction, and the channels are weighted to reduce the error between the reconstructed value and the true value, thereby improving the multispectral reconstruction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall flow chart of the multi-spectral reconstruction algorithm based on RGB image and attention residual network.

[0023] Figure 2 It is a flow chart of generating natural color images from hyperspectral images.

[0024] Figure 3 It is a graph of CIE color matching functions.

[0025] Figure 4 It is a true color image of the hyperspectral image.

[0026] Figure 5 It is a natural color image generated by the hyperspectral image through the color matching function.

[0027] Figure 6 This is how the attention mechanism works.

[0028] Figure 7 It is the specific structure of the attention residual network.

[0029] Figure 8 It is the specific structure of the residual block.

[0030] Fig. 9 It is a training process based on the residual network model.

[0031] Fig.10 It is a training process based on the attention residual network.

[0032] Fig.11 It is a comparison diagram of the original multispectral image and the reconstructed multispectral image. The first row is the original multispectral image, the second row is the reconstructed multispectral image, and the third row is the error diagram of the two. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0034] A multispectral reconstruction method based on RGB image and attention residual network, inputting the natural color image of the scene to be reconstructed into the trained natural color to multispectral network model to obtain the multispectral image of the scene to be reconstructed, where Figure 1 As shown, the training method of the natural color to multispectral network model is:

[0035] S1: According to the CIE color matching function, the hyperspectral image in the hyperspectral dataset is converted into a natural color image that conforms to the one taken by the camera. The specific process is as follows:

[0036] like Figure 2As shown in , first calculate the color matching function and select the corresponding color matching function curve. Then, perform weighted summation on each channel of the hyperspectral image according to the color matching function. Then, transform the result obtained after weighted summation from XYZ color space to RGB color space through the conversion matrix. Finally, perform gamma correction on the image data in RGB color space to make the image conform to the effect of RGB camera shooting. CIE color matching function is as follows Figure 3 shown.

[0037] The formula for weighted summation is as follows:

[0038]

[0039]

[0040] Where λ is the wavelength, X, Y, and Z are the response values ​​of the CIE XYZ color space, S(λ) is the reflectance of the object being measured, and I(λ) is the spectral power distribution of the light source. is the tristimulus curve, N can normalize the response value, and K is a scaling factor, usually 100. True color images and natural color images of hyperspectral images are as follows Figure 4 , Figure 5 shown.

[0041] S2: The red, green and blue bands in the hyperspectral dataset are used as true color images and as the input of the residual network. The natural color image obtained in step S1 is used as the theoretical output of the residual network, and the residual network is trained to obtain a true color to natural color network model based on the residual network.

[0042] It should be noted that the true color to natural color network model is mainly composed of an input layer, a residual block, and an output layer. The residual block consists of a convolutional layer, an activation layer, a convolutional layer, and a short-circuit connection; the activation function layer introduces nonlinear transformations to increase the nonlinear expression ability of the model; the convolutional layer performs a sliding window operation on the input image by applying a series of convolution kernels, and performs a dot product operation on the input data in the window to extract the features of the image; the activation function layer can perform nonlinear mapping on the output of neurons, so that the network can learn more complex functional relationships.

[0043] S3: input the red, green and blue three bands in the satellite multispectral data set as the true color image to be tested into the true color to natural color network model obtained in step S2, and obtain the natural color prediction image of the red, green and blue three bands in the satellite multispectral data set;

[0044] S4: The natural color predicted image obtained in step S3 is used as the input of the residual network fused with the attention mechanism, and the red, green, blue, red edge, and near-infrared bands in the satellite multispectral dataset are used as the theoretical output of the residual network fused with the attention mechanism. The residual network fused with the attention mechanism is trained to obtain a natural color to multispectral network model based on the residual network and the attention mechanism.

[0045] like Figure 7 As shown, the natural color to multi-spectral network model includes a sequentially cascaded input layer, an input convolutional layer, a first residual block, a second residual block, a third residual block, an attention mechanism compression and excitation module, a connection layer, an activation layer, and an output convolutional layer, wherein the connection layer is used to fuse the output of the attention mechanism compression and excitation module and the input convolutional layer.

[0046] Among them, Figure 8 As shown in , any residual block includes an input layer, a first convolutional layer, a first activation layer, a second convolutional layer, a connection layer and a second activation layer, wherein the connection layer is used to fuse the output of the input layer and the second convolutional layer; it can be seen that the residual block passes through two convolutional layers and one activation layer, the input is x, the output is F(x), and the input x and the output F(x) are connected together through a short circuit. The output of the next layer is H(x) = F(x) + x. The formula for the forward propagation of the residual network is shown in formula (5).

[0047]

[0048] where x L is the input of the Lth convolutional layer of the residual network, x l is the identity map added by the short-circuit connection, is the original input, It is the result of multiple convolutions before the Lth layer.

[0049] The formula for the back propagation of the residual network is shown in formula (6):

[0050]

[0051] The gradient of the residual network back propagation consists of two items: the identity mapping to x and the gradient of the multi-layer traditional neural network mapping. The identity mapping to x enables the weight parameters of the shallow network layer to be effectively trained, avoiding the gradient vanishing problem.

[0052] The channel attention mechanism is implemented by adding a compression and excitation module (Squeeze and Excitation, SE) on the basis of the residual network. The working principle is as follows Figure 6As shown in the figure; first, compression is performed, and the feature map of size h×w×c is pooled to compress the information of each channel into a single value to obtain the global compressed feature of the current feature map, which is 1×1×c in size; then excitation is performed, and the global compressed feature obtained in the first step is excited through the structure of fully connected layer-ReLU activation layer-fully connected layer-Sigmoid layer to obtain the weight of each channel in the feature map, and the feature map is weighted to obtain the processed feature map; the first fully connected layer compresses c channels into c / r channels to reduce the amount of calculation, and then passes through a ReLU nonlinear activation layer, and the second fully connected layer restores the number of channels to c channels, and then activates the Sigmoid function to obtain the normalized weight; r refers to the compression ratio.

[0053] It should be noted that the dataset for training the natural color to multispectral network model uses satellite multispectral and generated natural color RGB image data pairs, of which 2 / 3 is the training set and 1 / 3 is the validation set.

[0054] The loss function in the training process of the natural color to multispectral network model adopts the mean relative absolute error, the batch size is set to 16, the optimizer is Adamax, the learning rate is set to 0.001, the exponential decay rate β1 = 0.9, β2 = 0.999, ε = 10 -8 , to avoid the divisor becoming 0. Multiply by 0.98 every 200 epochs. At the same time, stop training when the loss function no longer decreases within 200 epochs.

[0055] It should be noted that the present invention selects the mean relative absolute error as the training error, and the mean relative absolute error (MRAE) and the root mean square error (RMSE) as the evaluation error. The specific calculation formula is as follows:

[0056]

[0057] like Fig. 9 , 10 As shown in the figure, the error obtained in the experimental results based on the EuroSAT satellite multispectral dataset is within an acceptable range, which verifies the feasibility of this algorithm. And compared with the experimental results based only on the residual network model, the experimental effect of the multispectral reconstruction model integrating the attention mechanism is better. Fig.11As shown in the figure, the reconstructed multispectral image is compared with the original multispectral image and displayed using a true color RGB image. In order to more intuitively show the difference between the two, the root mean square error of each pixel of the two is mapped in the range of 0-255, and the color is mapped to cold and warm colors. The closer to 0, the bluer it is, and the closer to 255, the redder it is. The top is the original multispectral image, the middle is the reconstructed multispectral image, and the bottom is the error map.

[0058] In summary, the present invention provides a multispectral reconstruction method based on RGB images and attention residual networks, aiming to solve the problem of lack of data sets through deep learning methods. Specifically, the multispectral images acquired by satellite wide areas and the theory of natural light color generation solve the problem of lack of large-scale natural light and multispectral pairing data. The model is trained using a dataset with rich types of objects to enhance the generalization ability of the model and improve the adaptability of the model in complex and diverse scenarios. At the same time, on the basis of preventing the network from being too deep and causing the gradient to disappear, attention is paid to the different contributions of different image features to multispectral reconstruction, and the channels are weighted to improve the accuracy of multispectral reconstruction.

[0059] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multispectral reconstruction method based on RGB image and attention residual network, characterized in that: The natural color image of the scene to be reconstructed is input into the trained natural color to multispectral network model to obtain a multispectral image of the scene to be reconstructed, wherein the training method of the natural color to multispectral network model is: S1: Convert the hyperspectral images in the hyperspectral dataset into natural color images that match those taken by the camera according to the CIE color matching function; S2: The red, green and blue bands in the hyperspectral dataset are used as true color images and as the input of the residual network. The natural color image obtained in step S1 is used as the theoretical output of the residual network, and the residual network is trained to obtain a true color to natural color network model based on the residual network. S3: input the red, green and blue three bands in the satellite multispectral data set as the true color image to be tested into the true color to natural color network model obtained in step S2, and obtain the natural color prediction image of the red, green and blue three bands in the satellite multispectral data set; S4: The natural color predicted image obtained in step S3 is used as the input of the residual network fused with the attention mechanism, and the red, green, blue, red edge, and near infrared in the satellite multispectral dataset are used as the theoretical outputs of the residual network fused with the attention mechanism. The residual network fused with the attention mechanism is trained to obtain a natural color to multispectral network model based on the residual network and the attention mechanism.

2. A multispectral reconstruction method based on RGB image and attention residual network as claimed in claim 1, characterized in that: The natural color to multi-spectral network model includes a sequentially cascaded input layer, an input convolutional layer, a first residual block, a second residual block, a third residual block, an attention mechanism compression and excitation module, a connection layer, an activation layer, and an output convolutional layer, wherein the connection layer is used to fuse the output of the attention mechanism compression and excitation module and the input convolutional layer.

3. A multispectral reconstruction method based on RGB image and attention residual network as claimed in claim 2, characterized in that: Any residual block includes an input layer, a first convolutional layer, a first activation layer, a second convolutional layer, a connection layer and a second activation layer, wherein the connection layer is used to fuse the output of the input layer and the second convolutional layer.

4. The multispectral reconstruction method based on RGB image and attention residual network as claimed in claim 2, characterized in that: The processing process of the attention mechanism compression and excitation module on the output feature map of the third residual block is as follows: Pooling is performed on the output feature map of size h×w×c, compressing the information of each channel of the output feature map into a single value, and obtaining a global compressed feature of size 1×1×c corresponding to each channel of the output feature map; Through the structure of fully connected layer-ReLU nonlinear activation layer-fully connected layer-Sigmoid layer, each global compression feature is first stimulated separately to obtain the weight of each channel in the output feature map; then the output feature map is weighted according to the weight corresponding to each channel to obtain the processed fusion feature map; Among them, the first fully connected layer compresses c channels into c / r channels, then passes through a ReLU nonlinear activation layer, the second fully connected layer restores the number of channels back to c channels, and finally activates the Sigmoid layer to obtain the normalized weights, and r is the set compression ratio.

5. The multispectral reconstruction method based on RGB image and attention residual network as claimed in claim 1, characterized in that: According to the CIE color matching function, the hyperspectral image in the hyperspectral dataset is converted into a natural color image that matches the one taken by the camera: Calculate the color matching function and select the corresponding color matching function curve; Perform weighted summation on each channel of the hyperspectral image according to the color matching function, and transform the result of the weighted summation from the XYZ color space to the RGB color space through the conversion matrix; The image data in the RGB color space is gamma corrected to make the image consistent with the effect of shooting with an RGB camera, and a natural color image that conforms to the camera is obtained.

Images