Hyperspectral image reconstruction method, device and system and storage medium

Abstract

The invention provides a hyperspectral image reconstruction method, device and system and a storage medium, and belongs to the technical field of image reconstruction, and the method comprises the steps: carrying out the image collection of a to-be-detected object through an image collection device, and obtaining a to-be-detected fuzzy light field image and a plurality of original fuzzy light field images; performing image acquisition on the to-be-measured object through the hyperspectral camera to obtain a hyperspectral image; preprocessing each fuzzy light field image to obtain a preprocessed light field image; performing Fourier transform processing on each preprocessed light field image to obtain a frequency domain light field image; and constructing a training model, and performing model analysis on the training model through all the original frequency domain light field images and all the hyperspectral images to obtain an image reconstruction model. According to the method, the quality of image reconstruction is improved, the blank in the field of image reconstruction is filled, rapid imaging is realized, the hyperspectral image is reconstructed, and the problem that frequency domain information is ignored in the prior art is solved.

Description

Technical Field

[0001] The present invention mainly relates to the field of image reconstruction technology, and in particular to a hyperspectral image reconstruction method, device, system and storage medium. Background Art

[0002] In the field of computational photography, the seven-dimensional plenoptic function is often used. Describe the light: at time t, from any position (x, y, z) in three-dimensional space, along the direction The observed wavelength is λ and the intensity is Light signals are the carriers of scene information. Different dimensions of light signals can carry different scene characteristics. Therefore, high-dimensional and high-resolution light signal imaging is the basis and key to obtaining more and more essential scene information.

[0003] Existing hyperspectral imaging devices are often bulky, expensive, and slow, limiting their widespread application, especially in the civilian market. Consequently, alternative methods are often employed to obtain hyperspectral images, such as deep learning-based methods that reconstruct hyperspectral images from RGB images. However, RGB images only contain information in three dimensions (x, y, λ), representing only three bands: red, green, and blue. Furthermore, existing deep learning methods often focus solely on information and connections in the spatial domain, while ignoring the frequency domain. This results in an inability to effectively separate various noise components from the raw data during hyperspectral imaging, leading to inaccurate image reconstruction results. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a hyperspectral image reconstruction method, device, system and storage medium in response to the deficiencies of the existing technology.

[0005] The present invention solves the above technical problems with the following technical solutions: A hyperspectral image reconstruction method comprises the following steps:

[0006] The image acquisition device is used to acquire an image of the object to be tested, thereby obtaining a blurred light field image to be tested and a plurality of original blurred light field images;

[0007] Capturing images of the object to be tested by a hyperspectral camera to obtain hyperspectral images corresponding to the original blurred light field images;

[0008] Preprocessing each of the blurred light field images to obtain a preprocessed light field image corresponding to each of the blurred light field images;

[0009] Performing Fourier transform processing on each of the pre-processed light field images to obtain a frequency domain light field image corresponding to each of the blurred light field images;

[0010] Constructing a training model, and performing model analysis on the training model using all the original frequency-domain light field images and all the hyperspectral images to obtain an image reconstruction model;

[0011] The image reconstruction model is used to reconstruct the blurred light field image to be measured to obtain a hyperspectral image reconstruction result.

[0012] Another technical solution of the present invention to solve the above technical problem is as follows: a hyperspectral image reconstruction device, comprising:

[0013] A fuzzy light field image acquisition module is used to acquire an image of the object to be tested through an image acquisition device to obtain a fuzzy light field image to be tested and a plurality of original fuzzy light field images;

[0014] A hyperspectral image acquisition module is used to acquire images of the object to be tested by a hyperspectral camera to obtain hyperspectral images corresponding to the original blurred light field images;

[0015] a preprocessing module, configured to preprocess each of the blurred light field images to obtain a preprocessed light field image corresponding to each of the blurred light field images;

[0016] A Fourier transform module, configured to perform Fourier transform processing on each of the pre-processed light field images to obtain a frequency domain light field image corresponding to each of the blurred light field images;

[0017] A model analysis module is used to construct a training model, and perform model analysis on the training model through all the original frequency domain light field images and all the hyperspectral images to obtain an image reconstruction model;

[0018] The image reconstruction result obtaining module is used to reconstruct the blurred light field image to be measured through the image reconstruction model to obtain a hyperspectral image reconstruction result.

[0019] Based on the above-mentioned hyperspectral image reconstruction method, the present invention also provides a hyperspectral image reconstruction system.

[0020] Another technical solution of the present invention to solve the above technical problem is as follows: A hyperspectral image reconstruction system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the hyperspectral image reconstruction method described above is implemented.

[0021] Based on the above hyperspectral image reconstruction method, the present invention also provides a computer-readable storage medium.

[0022] Another technical solution of the present invention to solve the above technical problem is as follows: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the hyperspectral image reconstruction method described above is implemented.

[0023] The beneficial effects of the present invention are as follows: an image acquisition device is used to acquire an image of the object to be tested and an original blurred light field image is obtained; a hyperspectral image is acquired by acquiring an image of the object to be tested through a hyperspectral camera; the blurred light field image is preprocessed to obtain a preprocessed light field image; the preprocessed light field image is subjected to Fourier transform processing to obtain a frequency domain light field image; an image reconstruction model is obtained by model analysis of a training model using the original frequency domain light field image and the hyperspectral image; and a hyperspectral image reconstruction result is obtained by reconstructing the image of the blurred light field image to be tested through the image reconstruction model, thereby improving the quality of image reconstruction, filling the gap in the field of image reconstruction, realizing rapid imaging and reconstruction into a hyperspectral image, and solving the problem of the prior art ignoring frequency domain information. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the process of a hyperspectral image reconstruction method provided by an embodiment of the present invention;

[0025] Figure 2 A schematic structural diagram of an image acquisition device for a hyperspectral image reconstruction method according to an embodiment of the present invention;

[0026] Figure 3 A network structure diagram of a frequency domain network of a hyperspectral image reconstruction method provided by an embodiment of the present invention;

[0027] Figure 4 This is a module block diagram of the hyperspectral image reconstruction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0029] Figure 1 A schematic flow chart of a hyperspectral image reconstruction method provided by an embodiment of the present invention.

[0030] like Figure 1 As shown, a hyperspectral image reconstruction method includes the following steps:

[0031] The image acquisition device is used to acquire an image of the object to be tested, thereby obtaining a blurred light field image to be tested and a plurality of original blurred light field images;

[0032] Capturing images of the object to be tested by a hyperspectral camera to obtain hyperspectral images corresponding to the original blurred light field images;

[0033] Preprocessing each of the blurred light field images to obtain a preprocessed light field image corresponding to each of the blurred light field images;

[0034] Performing Fourier transform processing on each of the pre-processed light field images to obtain a frequency domain light field image corresponding to each of the blurred light field images;

[0035] Constructing a training model, and performing model analysis on the training model using all the original frequency-domain light field images and all the hyperspectral images to obtain an image reconstruction model;

[0036] The image reconstruction model is used to reconstruct the blurred light field image to be measured to obtain a hyperspectral image reconstruction result.

[0037] In the above embodiment, the image acquisition device acquires the image of the object to be tested and obtains the blurred light field image to be tested and the original blurred light field image, the hyperspectral camera acquires the image of the object to be tested and obtains the hyperspectral image, the blurred light field image is preprocessed to obtain the preprocessed light field image, the preprocessed light field image is Fourier transformed to obtain the frequency domain light field image, the training model is analyzed through the original frequency domain light field image and the hyperspectral image to obtain the image reconstruction model, and the image reconstruction model is used to reconstruct the blurred light field image to be tested to obtain the hyperspectral image reconstruction result, which improves the quality of image reconstruction, fills the gap in the field of image reconstruction, realizes rapid imaging and reconstruction of the hyperspectral image, and solves the problem of the prior art ignoring frequency domain information.

[0038] Optionally, as an embodiment of the present invention, Figure 2 As shown, the image acquisition device includes a main lens, a liquid crystal microlens array, a PDLC film and a CCD camera;

[0039] The main lens, the liquid crystal microlens array, the PDLC film and the CCD camera are sequentially arranged on one side of the object to be tested.

[0040] It should be understood that a light field image dataset (i.e., multiple original blurred light field images and their corresponding hyperspectral images) is acquired through the constructed optical imaging system based on the liquid crystal microlens array (i.e., image acquisition device) and the hyperspectral camera.

[0041] Specifically, the object to be tested is aligned with the main lens, liquid crystal microlens array, PDLC film and CCD camera on the same optical path. By applying voltage to the liquid crystal microlens array and PDLC film and adjusting the voltage values ​​of both sides, the required light field image (i.e., multiple original blurred light field images) is obtained on the CCD camera.

[0042] It should be understood that applying voltage to the liquid crystal microlens array controls the focal length, and applying voltage to the PDLC film controls the light transmittance.

[0043] It should be understood that the data set is composed of multiple sets of two-dimensional light field images acquired by the optical imaging system and hyperspectral images taken by the hyperspectral camera at the same position.

[0044] Specifically, the object to be tested is aligned with the main lens, liquid crystal microlens array, PDLC film, and CCD camera on the same optical path, and the voltages of the liquid crystal microlens array and PDLC film are adjusted. By adjusting the liquid crystal microlens array to focus the image on the CCD camera, and by adjusting the PDLC film to change the light transmittance, the image clarity is adjusted, which also means that different imaging environments can be simulated. A hyperspectral camera is used to capture hyperspectral images at the same location, and multiple sets of images are obtained as a data set (i.e., multiple original blurred light field images and their corresponding hyperspectral images).

[0045] Specifically, the PDLC film is a variable device. By changing this device, light field images with different degrees of blur can be generated. Then, by inputting light field images with different degrees of blur and their corresponding hyperspectral images to observe the effect of the model, the optimal voltage parameters of the PDLC film can also be obtained.

[0046] In the above embodiment, clearer images can also be captured through the main lens, liquid crystal microlens array, PDLC film and CCD camera, laying the foundation for subsequent data processing.

[0047] Optionally, as an embodiment of the present invention, the process of preprocessing each of the original blurred light field images to obtain a preprocessed light field image corresponding to each of the original blurred light field images includes:

[0048] performing pixel normalization processing on each of the original blurred light field images to obtain a normalized light field image corresponding to each of the original blurred light field images;

[0049] Performing image cropping processing on each of the normalized light field images to obtain a cropped light field image corresponding to each of the original blurred light field images;

[0050] performing image enhancement processing on each of the cropped light field images to obtain an enhanced light field image corresponding to each of the original blurred light field images;

[0051] Registration processing is performed on each of the enhanced light field images to obtain a pre-processed light field image corresponding to each of the original blurred light field images.

[0052] It should be understood that before feature extraction is performed on the dataset (ie, multiple original blurred light field images), pixel normalization, image cropping and enhancement, and registration are performed on the dataset (ie, multiple original blurred light field images).

[0053] Specifically, the formula used for pixel normalization of the original image is as follows:

[0054]

[0055] Here, original_pixel refers to the value of each pixel in the original image (between 0 and 255), while normalized_pixel is the normalized pixel value, in the interval [0, 1]. min and max are the minimum and maximum values ​​of all pixel values ​​in a band, respectively. Cropping (i.e., image cropping) is achieved by calculating the number of sub-tiles that each image can be divided into. The crop size and step size can be customized based on the image size, and the sub-tiles are then extracted using a given index. Data augmentation (i.e., image augmentation) is implemented using the argument function, including random rotation, vertical flipping, and horizontal flipping.

[0056] In the above embodiment, the original blurred light field image is preprocessed to obtain a preprocessed light field image, thereby obtaining a more accurate image, laying the foundation for subsequent data processing, improving the quality of image reconstruction, and filling the gap in the field of image reconstruction.

[0057] Optionally, as an embodiment of the present invention, the preprocessed light field image includes multiple preprocessed light field sub-images, a first x-axis spatial frequency index corresponding to each of the preprocessed light field sub-images, a first y-axis spatial frequency index corresponding to each of the preprocessed light field sub-images, a first apex angle corresponding to each of the preprocessed light field sub-images, a first azimuth angle corresponding to each of the preprocessed light field sub-images, a first x-axis spatial coordinate corresponding to each of the preprocessed light field sub-images, and a first y-axis spatial coordinate corresponding to each of the preprocessed light field sub-images;

[0058] The process of performing Fourier transform processing on each of the pre-processed light field images to obtain an original frequency domain light field image corresponding to each of the original blurred light field images includes:

[0059] By using the first formula, Fourier transform processing is performed on each of the preprocessed light field sub-images, the first x-axis spatial frequency index corresponding to each of the preprocessed light field sub-images, the first y-axis spatial frequency index corresponding to each of the preprocessed light field sub-images, the first apex angle corresponding to each of the preprocessed light field sub-images, the first azimuth angle corresponding to each of the preprocessed light field sub-images, the first x-axis spatial coordinate corresponding to each of the preprocessed light field sub-images, and the first y-axis spatial coordinate corresponding to each of the preprocessed light field sub-images, to obtain a plurality of original frequency-domain light field sub-images corresponding to each of the original blurred light field images. The first formula is:

[0060]

[0061] Among them, F1 x,y () is the original frequency domain light field sub-image, k x1 is the first x-axis spatial frequency index, k y1 is the first y-axis spatial frequency index, θ1 is the first apex angle, is the first azimuth, N x1 is the total number of coordinates in the first x-axis space, N y1 is the total number of spatial coordinates of the first y-axis, f1() is the pre-processed light field sub-image, j is the imaginary unit, n x1 is the first x-axis space coordinate, n y1 is the first y-axis space coordinate;

[0062] An original frequency domain light field image corresponding to each of the original blurred light field images is obtained through a plurality of original frequency domain light field sub-images corresponding to each of the original blurred light field images.

[0063] It should be understood that for a space coordinate (x, y), the angular coordinate is Light field image (i.e., light field sub-image after preprocessing) Apply discrete Fourier transform and inverse transform in the spatial dimension. The discrete Fourier transform formula in the spatial dimension is as follows:

[0064]

[0065] Where N x and N y are the number of sampling points in the spatial dimensions x and y, respectively, and k x and k y is the spatial frequency index and j is the imaginary unit.

[0066] Specifically, the discrete Fourier transform is introduced to convert the light field image (i.e., the pre-processed light field image) from the spatial domain to the frequency domain. For a spatial coordinate (x, y), the angular coordinate is Light field image (i.e., light field sub-image after preprocessing) The discrete Fourier transform formula in the spatial dimension is as follows:

[0067]

[0068] Where N x and N y are the number of sampling points of spatial dimensions x and y (i.e., the total number of spatial coordinates of the first x-axis and the total number of spatial coordinates of the first y-axis), k x and k y is the spatial frequency index (ie, the first x-axis spatial frequency index and the first y-axis spatial frequency index), and j is an imaginary unit.

[0069] In the above embodiment, the pre-processed light field image is subjected to Fourier transform processing to obtain the original frequency domain light field image, which realizes rapid imaging and reconstruction into a hyperspectral image, and solves the problem of the prior art ignoring frequency domain information.

[0070] Optionally, as an embodiment of the present invention, the training model includes an encoder and a decoder;

[0071] The process of performing model analysis on the training model through all the original frequency domain light field images and all the hyperspectral images to obtain an image reconstruction model includes:

[0072] Encoding each of the original frequency domain light field images by the encoder to obtain encoded light field features corresponding to each of the original frequency domain light field images;

[0073] Decoding each of the encoded light field features by the decoder, respectively, to obtain a target frequency domain light field image corresponding to each of the original frequency domain light field images, the target frequency domain light field image comprising a plurality of target frequency domain light field sub-images, a second x-axis spatial frequency index corresponding to each of the target frequency domain light field sub-images, a second y-axis spatial frequency index corresponding to each of the target frequency domain light field sub-images, a second second apex angle corresponding to each of the target frequency domain light field sub-images, a second azimuth angle corresponding to each of the target frequency domain light field sub-images, a second x-axis spatial coordinate corresponding to each of the target frequency domain light field sub-images, and a second y-axis spatial coordinate corresponding to each of the target frequency domain light field sub-images;

[0074] Through the second formula, each of the target frequency domain light field sub-images, the second x-axis spatial frequency index corresponding to each of the target frequency domain light field sub-images, the second y-axis spatial frequency index corresponding to each of the target frequency domain light field sub-images, the second second apex angle corresponding to each of the target frequency domain light field sub-images, the second azimuth angle corresponding to each of the target frequency domain light field sub-images, the second x-axis spatial coordinate corresponding to each of the target frequency domain light field sub-images, and the second y-axis spatial coordinate corresponding to each of the target frequency domain light field sub-images are inverse Fourier transform-processed to obtain a plurality of hyperspectral sub-images corresponding to each of the original frequency domain light field images. The second formula is:

[0075]

[0076] Among them, f2() is the target frequency domain light field sub-image, n x2 is the second x-axis space coordinate, n y2 is the second y-axis spatial coordinate, θ2 is the second apex angle, is the second azimuth, k x2 is the second x-axis spatial frequency index, k y2 is the second y-axis spatial frequency index, N x2 is the total number of coordinates in the second x-axis space, N y2 is the total number of spatial coordinates of the second y-axis, f() is the pre-processed light field sub-image, and j is the imaginary unit;

[0077] Obtaining a hyperspectral image corresponding to each of the original frequency domain light field images through a plurality of hyperspectral sub-images corresponding to each of the original frequency domain light field images;

[0078] The parameters of the encoder and the decoder are updated together through a pre-built optimizer, all the hyperspectral images, and all the hyperspectral images, and each of the original frequency domain light field images is re-encoded by the updated encoder until the number of iterations is reached, and an image reconstruction model is constructed by the updated encoder and the updated decoder.

[0079] It should be understood that the training model is divided into an encoder module (i.e., encoder) and a decoder module (i.e., decoder) through a U-shaped structure, and the encoder part (i.e., encoder) and the decoder part (i.e., decoder) adopt jump connections.

[0080] It should be understood that the number of groups of encoders and decoders can be customized. A skip connection is used between each encoder and its corresponding decoder.

[0081] It should be understood that the training batch, hyperparameters, number of iterations, and optimizer are set, and the performance of the model is evaluated using evaluation metrics such as MARE, RMSE, and PSNR. The model with the best performance is saved and put into the validation set to evaluate the feasibility and generalization of the model.

[0082] Specifically, after the last decoder module (i.e., decoder), an inverse two-dimensional discrete Fourier transform is performed to convert the image (i.e., the target frequency domain light field image) from the frequency domain back to the spatial domain, and the formula is as follows:

[0083]

[0084] In the above embodiment, an image reconstruction model is obtained by performing model analysis on the training model through all the original frequency domain light field images, thereby improving the quality of image reconstruction, filling the gap in the field of image reconstruction, realizing rapid imaging and reconstruction into a hyperspectral image, and solving the problem of the existing technology ignoring frequency domain information.

[0085] Optionally, as an embodiment of the present invention, the encoder includes a first feature analysis network and a downsampling layer;

[0086] The process of encoding each of the original frequency domain light field images by the encoder to obtain encoded light field features corresponding to each of the original frequency domain light field images includes:

[0087] Performing feature analysis on each of the original frequency domain light field images through the first feature analysis network to obtain first light field features to be sampled corresponding to each of the original frequency domain light field images;

[0088] Downsampling is performed on each of the first light field features to be sampled through the downsampling layer to obtain encoded light field features corresponding to each of the original frequency domain light field images.

[0089] It should be understood that the encoder module includes a downsampling module (ie, a downsampling layer) in addition to the feature extraction module (ie, the first feature analysis network).

[0090] In the above embodiment, each original frequency domain light field image is encoded by an encoder to obtain encoded light field features, thereby achieving rapid imaging and reconstruction into a hyperspectral image, and solving the problem of ignoring frequency domain information in the prior art.

[0091] Optionally, as an embodiment of the present invention, the decoder includes a second feature analysis network and an upsampling layer;

[0092] The process of decoding each of the encoded light field features by the decoder to obtain a target frequency domain light field image corresponding to each of the original frequency domain light field images includes:

[0093] Performing feature analysis on each of the encoded light field features through the second feature analysis network to obtain a second light field feature to be sampled corresponding to each of the original frequency domain light field images;

[0094] The upsampling layer performs upsampling processing on each of the second light field features to be sampled, so as to obtain a target frequency domain light field image corresponding to each of the original frequency domain light field images.

[0095] It should be understood that the decoder module (ie, decoder) includes an upsampling module (ie, upsampling layer).

[0096] It should be understood that the data processing process of the second feature analysis network is the same as that of the first feature analysis network, and only the processed objects are different.

[0097] In the above embodiment, the decoder decodes each encoded light field feature to obtain a target frequency domain light field image, thereby achieving rapid imaging and reconstruction into a hyperspectral image, and solving the problem of ignoring frequency domain information in the prior art.

[0098] Optionally, as an embodiment of the present invention, the first feature analysis network includes an attention layer, a first normalization layer, a convolution layer, and a second normalization layer;

[0099] The process of performing feature analysis on each of the original frequency domain light field images through the first feature analysis network to obtain first light field features to be sampled corresponding to each of the original frequency domain light field images includes:

[0100] Performing feature extraction on each of the original frequency domain light field images through the attention layer to obtain first feature-extracted light field features corresponding to each of the original frequency domain light field images;

[0101] performing normalization processing on each of the light field features extracted by the first feature through the first normalization layer to obtain a normalized light field feature corresponding to each of the original frequency domain light field images;

[0102] Performing feature extraction on each of the normalized light field features through the convolution layer to obtain a second feature-extracted light field feature corresponding to each of the original frequency-domain light field images;

[0103] The second normalization layer performs normalization processing on each of the light field features extracted after the second feature extraction, so as to obtain a first light field feature to be sampled corresponding to each of the original frequency domain light field images.

[0104] It should be understood that an attention mechanism (i.e., an attention layer) is introduced, based on a convolutional layer and a normalization layer (i.e., a first normalization layer and a second normalization layer), and a residual connection is used.

[0105] Specifically, it consists of an attention mechanism layer (i.e., attention layer) plus a normalization layer combination (i.e., the first normalization layer) and a convolutional layer plus a normalization layer combination (i.e., the second normalization layer), and a residual connection is performed between the two combinations.

[0106] In the above embodiment, the first feature analysis network is used to perform feature analysis on each original frequency domain light field image to obtain the first light field feature to be sampled, thereby achieving rapid imaging and reconstruction into a hyperspectral image, and solving the problem of the prior art ignoring frequency domain information.

[0107] Alternatively, as another embodiment of the present invention, the present invention includes: 1. acquiring a light field image dataset using a constructed optical imaging system based on a liquid crystal microlens array and a hyperspectral camera; 2. constructing a hyperspectral reconstruction network based on Fourier transform; 3. preprocessing the dataset and dividing it into a training set, a test set, and a validation set in proportion; 4. training the frequency domain hyperspectral reconstruction network and testing it using evaluation indicators; 5. inputting the obtained light field image into the reconstruction network to obtain a hyperspectral image. The present invention combines optical devices to reconstruct the light field image into a hyperspectral image using a frequency domain optoelectronic hybrid neural network, thereby improving the quality of the reconstruction and filling a gap in the corresponding research field.

[0108] Alternatively, as another embodiment of the present invention, Figure 3 As shown, the present invention includes the following steps:

[0109] S1. Obtain a blurred light field image dataset through the constructed optical imaging system based on liquid crystal microlens array and hyperspectral camera;

[0110] S2. Construct a hyperspectral reconstruction network based on Fourier transform. Specifically, two-dimensional discrete Fourier transform and inverse two-dimensional discrete Fourier transform are introduced into the transformer network based on the attention mechanism. For a spatial coordinate (x, y) and an angular coordinate of Light field image Apply discrete Fourier transform and inverse transform in the spatial dimension. The discrete Fourier transform formula in the spatial dimension is as follows:

[0111]

[0112] Where N x and N y are the number of sampling points in the spatial dimensions x and y, respectively, and k x and k y is the spatial frequency index, and j is the imaginary unit:

[0113] The formula for inverse transformation in spatial dimension is as follows:

[0114]

[0115] S3. Preprocess the data set and divide it into training set, test set and validation set according to the proportion;

[0116] S4, training the frequency domain hyperspectral reconstruction network and testing it using evaluation metrics;

[0117] S5. The obtained light field image is input into the reconstruction network to obtain a hyperspectral image.

[0118] Alternatively, as another embodiment of the present invention, the dataset is partitioned into a training set, a validation set, and a test set with a size ratio of 8:1:1. The training set is used for training, and the training batch, hyperparameters, number of iterations, and optimizer are set. Model performance is evaluated using metrics such as MARE, RMSE, and PSNR. The model with the best performance is saved and used for the validation set to evaluate its feasibility and generalization.

[0119] Optionally, as another embodiment of the present invention, the present invention inputs a light field image obtained by using the constructed optical imaging system based on the liquid crystal microlens array into the evaluated model to generate a reconstructed hyperspectral image.

[0120] Optionally, as another embodiment of the present invention, during the hyperspectral imaging process, the raw data usually contains various noises due to factors such as sensor noise and atmospheric interference. The frequency domain method can effectively separate the noise components and useful information in the signal. Frequency domain transformation can also help extract spectral features in hyperspectral images, which are very useful for target detection and classification. For example, the specific absorption peaks or reflection characteristics of certain substances may be more obvious in the frequency domain, which is convenient for identifying and distinguishing different ground materials. In order to solve the problems existing in the field of hyperspectral imaging mentioned above, the present invention combines an optical imaging system and a deep learning network model, which can achieve rapid imaging and reconstruction into a hyperspectral image and is small-scale and easy to use. It can make full use of frequency domain information and has excellent reconstruction effect.

[0121] Optionally, as another embodiment of the present invention, the present invention can solve the problem that the existing deep learning reconstruction method ignores the frequency domain information, and designs a frequency domain-based network model to achieve hyperspectral reconstruction. The reconstruction quality is excellent and the imaging reconstruction integration can be realized. The optical imaging system used is small in size and has a fast imaging speed.

[0122] Optionally, as another embodiment of the present invention, the present invention includes the following steps:

[0123] 1. Acquire light field images through the constructed optical imaging system based on liquid crystal microlens array. The object to be tested is aligned with the main lens, liquid crystal microlens array, PDLC film and CCD camera on the same optical path, and the voltage of the liquid crystal microlens array and PDLC film is adjusted. By adjusting the liquid crystal microlens array to focus the image on the CCD camera, and by adjusting the PDLC film to change the transmittance of light, the clarity of the image is adjusted. This also means that different imaging environments can be simulated. Use a hyperspectral camera to capture hyperspectral images at the same location. Acquire multiple sets of images as a data set;

[0124] 2. Construct a hyperspectral reconstruction network based on Fourier transform;

[0125] 3. Preprocess the dataset, including normalization, image cropping and enhancement, and divide it into training set, test set and validation set in a size ratio of 8:1:1;

[0126] Fourth, train the frequency domain hyperspectral reconstruction network and test it using evaluation metrics such as PSNR, SSIM, and MARE. The training batch size is 100, the number of iterations is 300, and the Adam optimizer is used with an initial learning rate of 0.0001.

[0127] 5. The obtained light field image is input into the reconstruction network to obtain a hyperspectral image, that is, a .mat file data with 31 bands from 400nm to 700nm at intervals of 10nm is obtained.

[0128] Figure 4 This is a module block diagram of a hyperspectral image reconstruction device provided by an embodiment of the present invention.

[0129] Alternatively, as another embodiment of the present invention, Figure 4 As shown, a hyperspectral image reconstruction device includes:

[0130] A fuzzy light field image acquisition module is used to acquire an image of the object to be tested through an image acquisition device to obtain a fuzzy light field image to be tested and a plurality of original fuzzy light field images;

[0131] A hyperspectral image acquisition module is used to acquire images of the object to be tested by a hyperspectral camera to obtain hyperspectral images corresponding to the original blurred light field images;

[0132] a preprocessing module, configured to preprocess each of the blurred light field images to obtain a preprocessed light field image corresponding to each of the blurred light field images;

[0133] A Fourier transform module, configured to perform Fourier transform processing on each of the pre-processed light field images to obtain a frequency domain light field image corresponding to each of the blurred light field images;

[0134] A model analysis module is used to construct a training model, and perform model analysis on the training model through all the original frequency domain light field images and all the hyperspectral images to obtain an image reconstruction model;

[0135] The image reconstruction result obtaining module is used to reconstruct the blurred light field image to be measured through the image reconstruction model to obtain a hyperspectral image reconstruction result.

[0136] Alternatively, another embodiment of the present invention provides a hyperspectral image reconstruction system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the hyperspectral image reconstruction method described above is implemented. The system may be a computer or other system.

[0137] Optionally, another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the hyperspectral image reconstruction method described above is implemented.

[0138] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0139] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0141] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of the embodiments of the present invention.

[0142] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hyperspectral image reconstruction method, characterized in that: The steps include: The image acquisition device is used to acquire an image of the object to be tested, thereby obtaining a blurred light field image to be tested and a plurality of original blurred light field images; Capturing images of the object to be tested by a hyperspectral camera to obtain hyperspectral images corresponding to the original blurred light field images; Preprocessing each of the blurred light field images to obtain a preprocessed light field image corresponding to each of the blurred light field images; Performing Fourier transform processing on each of the pre-processed light field images to obtain a frequency domain light field image corresponding to each of the blurred light field images; Constructing a training model, and performing model analysis on the training model using all the original frequency-domain light field images and all the hyperspectral images to obtain an image reconstruction model; The image reconstruction model is used to reconstruct the blurred light field image to be measured to obtain a hyperspectral image reconstruction result.

2. The hyperspectral image reconstruction method according to claim 1, characterized in that: The image acquisition device includes a main lens, a liquid crystal microlens array, a PDLC film and a CCD camera; The main lens, the liquid crystal microlens array, the PDLC film and the CCD camera are sequentially arranged on one side of the object to be tested.

3. The hyperspectral image reconstruction method according to claim 1, characterized in that: The process of preprocessing each of the original blurred light field images to obtain a preprocessed light field image corresponding to each of the original blurred light field images includes: performing pixel normalization processing on each of the original blurred light field images to obtain a normalized light field image corresponding to each of the original blurred light field images; Performing image cropping processing on each of the normalized light field images to obtain a cropped light field image corresponding to each of the original blurred light field images; performing image enhancement processing on each of the cropped light field images to obtain an enhanced light field image corresponding to each of the original blurred light field images; Registration processing is performed on each of the enhanced light field images to obtain a pre-processed light field image corresponding to each of the original blurred light field images.

4. The hyperspectral image reconstruction method according to claim 1, characterized in that: The preprocessed light field image includes a plurality of preprocessed light field sub-images, a first x-axis spatial frequency index corresponding to each of the preprocessed light field sub-images, a first y-axis spatial frequency index corresponding to each of the preprocessed light field sub-images, a first apex angle corresponding to each of the preprocessed light field sub-images, a first azimuth angle corresponding to each of the preprocessed light field sub-images, a first x-axis spatial coordinate corresponding to each of the preprocessed light field sub-images, and a first y-axis spatial coordinate corresponding to each of the preprocessed light field sub-images; The process of performing Fourier transform processing on each of the pre-processed light field images to obtain an original frequency domain light field image corresponding to each of the original blurred light field images includes: By using the first formula, Fourier transform processing is performed on each of the preprocessed light field sub-images, the first x-axis spatial frequency index corresponding to each of the preprocessed light field sub-images, the first y-axis spatial frequency index corresponding to each of the preprocessed light field sub-images, the first apex angle corresponding to each of the preprocessed light field sub-images, the first azimuth angle corresponding to each of the preprocessed light field sub-images, the first x-axis spatial coordinate corresponding to each of the preprocessed light field sub-images, and the first y-axis spatial coordinate corresponding to each of the preprocessed light field sub-images, to obtain a plurality of original frequency-domain light field sub-images corresponding to each of the original blurred light field images. The first formula is: Among them, F1 x,y () is the original frequency domain light field sub-image, k x1 is the first x-axis spatial frequency index, k y1 is the first y-axis spatial frequency index, θ1 is the first apex angle, is the first azimuth, N x1 is the total number of coordinates in the first x-axis space, N y1 is the total number of spatial coordinates of the first y-axis, f1() is the pre-processed light field sub-image, j is the imaginary unit, n x1 is the first x-axis space coordinate, n y1 is the first y-axis space coordinate; An original frequency domain light field image corresponding to each of the original blurred light field images is obtained through a plurality of original frequency domain light field sub-images corresponding to each of the original blurred light field images.

5. The hyperspectral image reconstruction method according to claim 1, characterized in that: The training model includes an encoder and a decoder; The process of performing model analysis on the training model through all the original frequency domain light field images and all the hyperspectral images to obtain an image reconstruction model includes: Encoding each of the original frequency domain light field images by the encoder to obtain encoded light field features corresponding to each of the original frequency domain light field images; Decoding each of the encoded light field features by the decoder, respectively, to obtain a target frequency domain light field image corresponding to each of the original frequency domain light field images, the target frequency domain light field image comprising a plurality of target frequency domain light field sub-images, a second x-axis spatial frequency index corresponding to each of the target frequency domain light field sub-images, a second y-axis spatial frequency index corresponding to each of the target frequency domain light field sub-images, a second second apex angle corresponding to each of the target frequency domain light field sub-images, a second azimuth angle corresponding to each of the target frequency domain light field sub-images, a second x-axis spatial coordinate corresponding to each of the target frequency domain light field sub-images, and a second y-axis spatial coordinate corresponding to each of the target frequency domain light field sub-images; Through the second formula, each of the target frequency domain light field sub-images, the second x-axis spatial frequency index corresponding to each of the target frequency domain light field sub-images, the second y-axis spatial frequency index corresponding to each of the target frequency domain light field sub-images, the second second apex angle corresponding to each of the target frequency domain light field sub-images, the second azimuth angle corresponding to each of the target frequency domain light field sub-images, the second x-axis spatial coordinate corresponding to each of the target frequency domain light field sub-images, and the second y-axis spatial coordinate corresponding to each of the target frequency domain light field sub-images are inverse Fourier transform-processed to obtain a plurality of hyperspectral sub-images corresponding to each of the original frequency domain light field images. The second formula is: Among them, f2() is the target frequency domain light field sub-image, n x2 is the second x-axis space coordinate, n y2 is the second y-axis spatial coordinate, θ2 is the second apex angle, is the second azimuth, k x2 is the second x-axis spatial frequency index, k y2 is the second y-axis spatial frequency index, N x2 is the total number of coordinates in the second x-axis space, N y2 is the total number of spatial coordinates of the second y-axis, f() is the pre-processed light field sub-image, and j is the imaginary unit; Obtaining a hyperspectral image corresponding to each of the original frequency domain light field images through a plurality of hyperspectral sub-images corresponding to each of the original frequency domain light field images; The parameters of the encoder and the decoder are updated together through a pre-built optimizer, all the hyperspectral images, and all the hyperspectral images, and each of the original frequency domain light field images is re-encoded by the updated encoder until the number of iterations is reached, and an image reconstruction model is constructed by the updated encoder and the updated decoder.

6. The hyperspectral image reconstruction method according to claim 5, characterized in that: The encoder includes a first feature analysis network and a downsampling layer; The process of encoding each of the original frequency domain light field images by the encoder to obtain encoded light field features corresponding to each of the original frequency domain light field images includes: Performing feature analysis on each of the original frequency domain light field images through the first feature analysis network to obtain first light field features to be sampled corresponding to each of the original frequency domain light field images; Downsampling is performed on each of the first light field features to be sampled through the downsampling layer to obtain encoded light field features corresponding to each of the original frequency domain light field images.

7. The hyperspectral image reconstruction method according to claim 5, characterized in that: The decoder includes a second feature analysis network and an upsampling layer; The process of decoding each of the encoded light field features by the decoder to obtain a target frequency domain light field image corresponding to each of the original frequency domain light field images includes: Performing feature analysis on each of the encoded light field features through the second feature analysis network to obtain a second light field feature to be sampled corresponding to each of the original frequency domain light field images; The upsampling layer performs upsampling processing on each of the second light field features to be sampled, so as to obtain a target frequency domain light field image corresponding to each of the original frequency domain light field images.

8. The hyperspectral image reconstruction method according to claim 6, characterized in that: The first feature analysis network includes an attention layer, a first normalization layer, a convolutional layer, and a second normalization layer; The process of performing feature analysis on each of the original frequency domain light field images through the first feature analysis network to obtain first light field features to be sampled corresponding to each of the original frequency domain light field images includes: Performing feature extraction on each of the original frequency domain light field images through the attention layer to obtain first feature-extracted light field features corresponding to each of the original frequency domain light field images; performing normalization processing on each of the light field features extracted by the first feature through the first normalization layer to obtain a normalized light field feature corresponding to each of the original frequency domain light field images; Performing feature extraction on each of the normalized light field features through the convolution layer to obtain a second feature-extracted light field feature corresponding to each of the original frequency-domain light field images; The second normalization layer performs normalization processing on each of the light field features extracted after the second feature extraction, so as to obtain a first light field feature to be sampled corresponding to each of the original frequency domain light field images.

9. A hyperspectral image reconstruction device, characterized in that: include: A fuzzy light field image acquisition module is used to acquire an image of the object to be tested through an image acquisition device to obtain a fuzzy light field image to be tested and a plurality of original fuzzy light field images; A hyperspectral image acquisition module is used to acquire images of the object to be tested by a hyperspectral camera to obtain hyperspectral images corresponding to the original blurred light field images; a preprocessing module, configured to preprocess each of the blurred light field images to obtain a preprocessed light field image corresponding to each of the blurred light field images; A Fourier transform module, configured to perform Fourier transform processing on each of the pre-processed light field images to obtain a frequency domain light field image corresponding to each of the blurred light field images; A model analysis module is used to construct a training model, and perform model analysis on the training model through all the original frequency domain light field images and all the hyperspectral images to obtain an image reconstruction model; The image reconstruction result obtaining module is used to reconstruct the blurred light field image to be measured through the image reconstruction model to obtain a hyperspectral image reconstruction result.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the hyperspectral image reconstruction method according to any one of claims 1 to 8 is implemented.

Images