A hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition

Through the method based on multi-layer perceptron and singular value decomposition, the accurate reconstruction of hyperspectral images is achieved using matrix transformation and SpectralMixer network, which solves the problem of insufficient real-time and cost control of hyperspectral image reconstruction algorithms in the prior art, and improves reconstruction Accuracy and inference speed.

CN114663283BActive Publication Date: 2024-12-03SOUTHEAST UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210313618.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-28
Publication Date
2024-12-03
Estimated Expiration
2042-03-28

AI Technical Summary

Technical Problem

The existing hyperspectral image reconstruction algorithms have shortcomings in real-time and cost control, and traditional methods have failed to effectively reconstruct the RGB domain image into a hyperspectral image, and the calculation cost is high.

Method used

Using a hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition, the zero-space coefficient is obtained through matrix transformation and singular value decomposition, and a high-resolution end-to-end SpectralMixer network is constructed to realize the accurate reconstruction of RGB domain images to hyperspectral images. .

Benefits of technology

The accuracy and inference speed of hyperspectral image reconstruction are improved, the average relative error and root mean square error of reconstruction are reduced, and the perfect transformation of hyperspectral images is achieved. It is suitable for tasks such as abnormal detection and target tracking that are required in real time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114663283B_ABST
    Figure CN114663283B_ABST
Patent Text Reader

Abstract

The present invention discloses a hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition, which can be used to realize high-resolution conversion from RGB images to hyperspectral images. The method comprises: according to the hyperspectral data, pre-processing obtains RGB domain images and true values ​​of zero space coefficients, and constructs a training data set; constructs an end-to-end model based on a deep high-resolution SpectralMixer network; supervised training is performed on the SpectralMixer network based on the constructed training set, thereby obtaining a converged optimized model SpectralMixer; and the training effect of the model SpectralMixer is evaluated on the test set. Compared with the traditional model HSCNN‑R p While reducing the computation time, lower reconstruction mean relative error (MRAE) and root mean square error (RMSE) are obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a hyperspectral image reconstruction method based on a multi-layer perceptron and singular value decomposition, and is applicable to the technical field of hyperspectral image reconstruction in computer vision. Background Art

[0002] Hyperspectral Imaging Systems (HISs) can record the spectral information of a scene through a set of narrow spectral bands. Compared with RGB images recorded by ordinary color cameras using three bands (red, green, and blue), hyperspectral images can obtain richer information. Hyperspectral imaging has a wide range of uses in remote sensing imaging, medical imaging, and anomaly detection. Among them, hyperspectral images can be used to investigate and analyze water resources, forestry resources, etc., to achieve the rational use and allocation of resources. For cancer patients, medical hyperspectral images are conducive to faster and more accurate diagnosis, and in anomaly detection, hyperspectral images can make objects that cannot be distinguished in the original feature space easier to distinguish.

[0003] Compared with the fast RGB imaging system, the complexity of the HISs system has been significantly improved, and it has a great lack of real-time performance. In order to meet the real-time requirements, the "snapshot" hyperspectral imaging system has emerged, but its recorded images use lower spatial and spectral resolutions, and no matter which hyperspectral imaging system, it is expensive. Therefore, researchers began to study how to use cheap RGB cameras to restore hyperspectral images, which can both meet real-time requirements and control costs. Traditional hyperspectral image reconstruction algorithms only consider reconstructing RGB domain images into hyperspectral images, and rarely consider whether the reconstructed hyperspectral images can be perfectly transformed into RGB domain images. Therefore, it is particularly important to find hyperspectral images that conform to physical meanings by establishing mathematical models.

[0004] Convolutional Neural Network (CNN) is a biophysical model inspired by the visual nervous system. It has the characteristics of translation invariance and local connection. In the past, it has been widely used in computer vision tasks in the development of deep learning, mainly used to extract features such as image structure and texture. Until the emergence of Transformer-related structures, it has achieved comparable results with traditional CNNs in computer vision tasks such as target detection and target tracking. However, due to its high computing power requirements and long computing time, the corresponding network structure based on multilayer perceptron (MLP) has achieved similar results to Transformer-related structures, with fewer parameters and faster inference speed. The hyperspectral image reconstruction problem is essentially a mapping problem from 3-channel RGB domain images to multi-channel hyperspectral images, which is a two-dimensional data. Therefore, when using a network structure based on a multilayer perceptron, it is necessary to input from two dimensions: width and height. Summary of the invention

[0005] Purpose of the invention: In view of the existing technology, a hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition is proposed to improve the reconstruction accuracy and speed up the inference time.

[0006] Technical solution: A hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition includes the following steps:

[0007] Step S1: Obtain the hyperspectral dataset and obtain the true value of the reconstructed hyperspectral image Transform it into RGB domain image I through matrix transformation RGB Obtain the true value α of the zero space coefficient through matrix operations and singular value decomposition g ;

[0008] Step S2: Build a high-resolution end-to-end deep SpectralMixer network;

[0009] Step S3: The RGB-Spectral Image data pair is input into the SpectralMixer network for forward operation, thereby obtaining an optimized SpectralMixer network, and then the optimized SpectralMixer network is used to operate the newly given RGB domain image to obtain the zero space coefficient α of the Spectral domain for subsequent reconstruction of the hyperspectral image;

[0010] Step S4: Use the zero spatial coefficient α obtained in step S3 to restore the 31-channel hyperspectral image

[0011] Furthermore, the step S1 specifically includes:

[0012] Step S101: read the hyperspectral image to RGB domain transformation matrix F, multiply it with the hyperspectral image, and obtain an RGB domain image;

[0013] Step S102: Get the vector space S expanded by F and calculate its projection matrix P S , the formula is as follows:

[0014] P S =F×(F T ×F) -1 ×F T (1)

[0015] Then the projection matrix of the orthogonal space of the vector space S is P N =1-P S , perform orthogonal decomposition to obtain the matrix N composed of orthogonal bases, and then reconstruct the true value of the hyperspectral image Together we get the true value of the zero space coefficient α g , the formula is as follows:

[0016]

[0017] Furthermore, the step S3 specifically includes:

[0018] Step S301: Divide the dataset ARAD HS into a training set and a test set. In the training of the SpectralMixer deep neural network, randomly cropped image pairs with a resolution of 256×256 are trained;

[0019] Step S302: supervise the SpectralMixer deep neural network by using the mean square error loss function. During the training process, the training is stopped after the loss function value converges, thereby obtaining an optimized and usable SpectralMixer deep neural network.

[0020] Step S303: Use the trained SpectralMixer deep neural network to predict the test set to obtain the null space coefficient α.

[0021] Furthermore, the step S4 specifically includes:

[0022] Step S401: Calculate the null space mapping image using the matrix N in step S102 and the null space coefficient α in step S303 The formula is as follows:

[0023]

[0024] Step S402: Utilize the transformation matrix F and the RGB domain image I in step S101 RGB Get the mapped image in vector space S The formula is as follows:

[0025]

[0026] Step S403: The step S401 Same as S402 Add to obtain the reconstructed hyperspectral image described in step S4

[0027] Finally, the reconstructed hyperspectral image is output

[0028] The beneficial effect of the present invention is that the present invention can realize accurate reconstruction of hyperspectral images. The null space coefficient α is obtained according to the obtained RGB domain image, and the orthogonal basis matrix N is obtained through the transformation matrix F, and the null space mapping image is obtained on this basis. and the mapping image in the vector space S Finally, the reconstructed hyperspectral image is obtained This hyperspectral image It can be perfectly transformed into the initial RGB domain image. Compared with the traditional model HSCNN-R p Compared with the traditional deep neural network, the SpectralMixer deep neural network reduces the average relative error and root mean square error of reconstruction, making the reconstruction of hyperspectral images through RGB domain images more accurate and faster inference. It has higher practical value in real-time scenarios (such as anomaly detection, target tracking and other tasks). In the field of deep learning, this paper further proves that the MLP structure can achieve similar or even better results than the convolutional neural network in some tasks. It has certain reference significance for the research of other researchers. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition in Example 1.

[0030] Figure 2 This is a flowchart of the SpectralMixer deep neural network training and prediction in Example 1.

[0031] Figure 3 This is a structural diagram of the SpectralMixer deep neural network model in Example 1.

[0032] Figure 4 The module structure diagram of Down-MLP and Up-MLP in Example 1

[0033] Figure 5 This is the MRAE effect diagram of reconstructing the hyperspectral image in Example 1.

[0034] Figure 6 are the MRAE and RMSE values ​​in Example 1. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Example 1

[0037] See also Figure 1-Figure 6 This embodiment provides a hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition. The method obtains the null space coefficient α according to the obtained RGB domain image, and obtains the matrix N composed of orthogonal bases through the transformation matrix F from the hyperspectral image to the RGB domain, and obtains the null space mapping image on this basis. and the mapping image in the vector space S Finally, the reconstructed hyperspectral image is obtained This hyperspectral image It can be perfectly transformed into the original RGB domain image.

[0038] Specifically, see Figure 1 , this method specifically includes:

[0039] Step S1: Obtain the hyperspectral dataset and obtain the true value of the reconstructed hyperspectral image Transform it into RGB domain image I through matrix transformation RGB Obtain the true value α of the zero space coefficient through matrix operations and singular value decomposition g ;

[0040] More specifically, step S1 includes: reading the transformation matrix F from the hyperspectral image to the RGB domain, multiplying it with the hyperspectral image to obtain the RGB domain image; calculating the projection matrix P of the vector space S expanded by F S ; Calculate the projection matrix P of the orthogonal space of the vector space S N , and decompose it orthogonally to obtain the matrix N composed of orthogonal bases; finally, reconstruct the true value of the hyperspectral image Get the true value of the zero space coefficient α g .

[0041] For a single 3-channel RGB pixel, the transformation matrix F has a dimension of 3×31, and the projection matrix P S and P N The dimension of is 31×31, the dimension of matrix N is 31×28, and the true value of the zero space coefficient is α g The number of channels is 28, which means that the SpectralMixer deep neural network finally transforms the 3-channel image into 28 channels. 31 represents the number of channels of the hyperspectral image. The transformation of 28 channels into 31 channels requires subsequent steps.

[0042] Step S2: Build a high-resolution end-to-end deep SpectralMixer network;

[0043] Step S3: input the RGB-Spectral Image data pair into the SpectralMixer network for forward operation, thereby obtaining an optimized SpectralMixer network, and then use the optimized SpectralMixer network to operate on the newly given RGB domain image to obtain the zero space coefficient α of the Spectral domain for subsequent reconstruction of the Spectral image;

[0044] More specifically, in this embodiment, step S3 includes: using the training set to train the SpectralMixer deep neural network based on the Pytorch framework, fixing the trained SpectralMixer deep neural network parameters, and reconstructing the hyperspectral image according to the test set features.

[0045] It should be noted that the hyperspectral reconstruction method provided in this embodiment is not limited to the Pytorch framework. As long as the dataset X can be trained and the loss function converges after several iterations (on the order of times) during the training process, and finally the hyperspectral image can be accurately predicted based on the RGB domain image, it can be used.

[0046] Step S4: Use the zero spatial coefficient α obtained in step S3 to restore the 31-channel hyperspectral image

[0047] More specifically, in this embodiment, the null space mapping image is calculated using the null space coefficient α obtained in step S3 and the orthogonal basis matrix N obtained in step S1. Using the transformation matrix F and RGB domain image I obtained in step S1 RGB Get the mapped image in vector space S Finally and Add to obtain the reconstructed hyperspectral image. The required formula is as follows:

[0048]

[0049] It should be noted that the indicators used in the present invention to measure the effect of reconstructing hyperspectral images are not limited to MRAE and RMSE. and reconstruct the true value of the hyperspectral image All the different indicators can be used as measurement indicators.

[0050] The process of learning a model from data is called training or learning. The data training set is input into the SpectralMixer deep neural network, which learns the texture, structure, and multi-scale features in the RGB domain image. After thousands of iterations of training, a trained SpectralMixer deep neural network can be obtained. This model can be used to reconstruct hyperspectral images based on singular value decomposition. Figure 2 As shown, the steps include:

[0051] Step S301: Divide the dataset ARAD HS into a training set and a test set, including 450 images and 10 images respectively. In the training of the SpectralMixer deep neural network, randomly cropped image pairs with a resolution of 256×256 are trained;

[0052] Step S302: supervise the SpectralMixer deep neural network by using the mean square error loss function. During the training process, the training is stopped after the loss function value converges, thereby obtaining an optimized and usable SpectralMixer deep neural network.

[0053] Step S303: using the trained SpectralMixer deep neural network to predict the test set, obtain the null space coefficient α, and calculate and reconstruct the hyperspectral image according to step S4;

[0054] like Figure 3 As shown in the figure, the model is the overall structure of the SpectralMixer deep neural network. The Down-MLP module and the Up-MLP module are connected in series. The five Up-MLP modules can respectively receive the shallow features provided by the corresponding Down-MLP modules, and combine them with the current features to output the 28-dimensional zero space coefficient α.

[0055] like Figure 4As shown in the figure, the model is the overall structure of the Down-MLP module and the Up-MLP module. The main difference between the Down-MLP module and the Up-MLP module is that the Down-MLP module needs to pass the MaxPooling layer after the input, while the Up-MLP module needs to perform an upsampling operation and then concatenate it with the corresponding Down-MLP module output. After that, both modules pass through the MLP-based structure. The MLP-based structure is mainly divided into three branches. The first two branches group the RGB domain images in the height and width dimensions respectively, and then pass through two fully connected layers and a GELU activation function layer. The third branch is a ResNet structure. Finally, a layer of convolutional neural network is used to transform the number of channels.

[0056] like Figure 5 The following are some of the effects of using the SpectralMixer deep neural network to reconstruct hyperspectral images. The pixel values ​​in the figure represent the MRAE values ​​of the reconstructed hyperspectral images at the pixel points. Obviously, from a qualitative point of view, it is better than the traditional model HSCNN-R p Compared with the previous example, the reconstruction effect is improved. Figure 6 As shown in the figure, from a qualitative point of view, the MRAE and RMSE of the SpectralMixer deep neural network are comparable to those of the traditional model HSCNN-R p Compared with the previous results, both of them have decreased, so the SpectralMixer deep neural network has a better effect.

[0057] The matters not described in detail in the present invention are all known technologies to those skilled in the art.

[0058] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition, characterized in that: The method comprises the following steps: Step S1: Obtain the hyperspectral dataset and obtain the true value of the reconstructed hyperspectral image Convert it into RGB domain image I through matrix transformation RGB , obtain the true value α of the zero space coefficient through matrix operation and singular value decomposition g ; Step S2: construct a high-resolution end-to-end deep SpectralMixer network, in which the Down-MLP module and the Up-MLP module are connected in series, and the five Up-MLP modules receive the shallow features provided by the corresponding Down-MLP modules respectively, and combine them with the current features to output a 28-dimensional null space coefficient α; Step S3: The RGB-Spectral Image data is input into the SpectralMixer network for forward operation, the loss function is calculated, and the Adam optimizer is used to optimize and update the SpectralMixer network, and it is iterated to obtain the optimized SpectralMixer network. The optimized SpectralMixer network is then used to operate the newly given RGB domain image to obtain the zero space coefficient α of the Spectral domain for subsequent reconstruction of the Spectral image. Step S4: Restore the 31-channel hyperspectral image using the zero spatial coefficient α obtained in step S3 The step S1 specifically includes: Step S101: read the transformation matrix F from the hyperspectral image to the RGB domain, and multiply it with the hyperspectral image to obtain an RGB domain image; Step S102: Get the vector space S expanded by F and calculate its projection matrix P S , the formula is as follows: P S =F×(F T ×F) -1 ×F T (1) Then the projection matrix of the orthogonal space of the vector space S is P N =1-P S , perform orthogonal decomposition to obtain the matrix N composed of orthogonal bases, and then reconstruct the true value of the hyperspectral image Together we get the true value of the zero space coefficient α g , the formula is as follows: The step S4 specifically includes: Step S401: Calculate the null space mapping image using the matrix N in step S102 and the null space coefficient α in step S303 The formula is as follows: Step S402: Utilize the transformation matrix F and the RGB domain image I in step S101 RGB Get the mapped image in vector space S The formula is as follows: Step S403: The step S401 Same as S402 Add to obtain the reconstructed hyperspectral image described in step S4 2. The hyperspectral image reconstruction method based on multi-layer perceptron and singular value decomposition according to claim 1, characterized in that: The step S3 specifically includes: Step S301: Divide the dataset ARAD HS into a training set and a test set, and train randomly selected image pairs with a resolution of 256×256 in the training of the SpectralMixer deep neural network; Step S302: supervise the SpectralMixer deep neural network by using the mean square error loss function. During the training process, the training is stopped after the loss function value converges, thereby obtaining an optimized and usable SpectralMixer deep neural network. Step S303: Use the SpectralMixer deep neural network that has converged in training to predict the test set to obtain the null space coefficient α.

Citation Information

Patent Citations

  • Hyperspectral imaging method for dedispersive ambiguity through deep learning

    CN113008371A

  • Single-view three-dimensional object reconstruction method and device based on RGB data

    CN113538662A

  • Latent low-rank projection learning based unsupervised feature extraction method for hyperspectral image

    WO2022001159A1