Hyperspectral image reconstruction method based on metasurface

Through the hyperspectral image reconstruction method based on metasurface, the problems of large hyperspectral camera size, long imaging time and poor reconstruction accuracy of metasurface cameras are solved, and compact and efficient spectral imaging is achieved, which improves imaging speed and accuracy.

CN119068072APending Publication Date: 2024-12-03HARBIN INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411209246.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing hyperspectral cameras are too large in size, long imaging time, and the reconstruction accuracy of metasurface hyperspectral cameras is poor and the design process is complex.

Method used

Using a hyperspectral image reconstruction method based on metasurface, the optimal metasurface response matrix arrangement is obtained by training the metasurface response layer, real physical dimension parameters are derived, metasurface lenses are made and covered on CMOS, and fast and efficient spectral imaging.

Benefits of technology

The size of the hyperspectral camera is significantly reduced, the imaging speed is improved, the design complexity is reduced, the reconstruction accuracy is improved, and the imaging speed at the millisecond level and the metasurface lens below 1 square centimeter are achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068072A_ABST
    Figure CN119068072A_ABST
Patent Text Reader

Abstract

The invention discloses a hyperspectral image reconstruction method based on a metasurface, relates to a hyperspectral image reconstruction method, and belongs to the technical field of hyperspectral cameras. The objective of the invention is to solve the problems of overlarge size and overlong imaging time of an existing hyperspectral camera, and to solve the problems of poor reconstruction precision and complex design process of an existing super-surface hyperspectral camera. The method comprises the following steps: 1, training a metasurface response layer, and obtaining a response matrix of a metasurface unit based on the trained metasurface response layer; 2, obtaining an optimal metasurface response matrix arrangement mode; 3, deducing a real physical size parameter of an optimal metasurface response matrix arrangement mode based on a reverse design algorithm; manufacturing a metasurface lens according to the real physical dimension parameters; 4, directly covering the metasurface lens obtained in the step 3 on a CMOS (Complementary Metal Oxide Semiconductor) of a camera, and receiving a continuous spectrum of a shot scene by the camera to obtain a multispectral image; and obtaining a hyperspectral image based on the multispectral image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a hyperspectral image reconstruction method and belongs to the technical field of hyperspectral cameras. Background Art

[0002] Hyperspectral camera is a key device for realizing hyperspectral imaging. It consists of an optical system, a spectrum separation element, a detector, and a data processing unit. The spectrum separation element (such as a prism, a grating, or a filter) decomposes the light signal into multiple bands, and then the detector acquires the spectral data point by point. With the development of computer technology, the resolution, sensitivity, and speed of hyperspectral cameras have been continuously improved, making them more powerful and have broader prospects in various application fields.

[0003] Metasurface technology is an advanced method for manipulating electromagnetic waves (including light waves) by precisely controlling the structure and properties of materials at the nanoscale. Metasurfaces are composed of a series of subwavelength-scale nanostructures that can regulate the phase, amplitude, and polarization state of light, thereby achieving functions that are difficult to achieve with traditional optical components. Compact hyperspectral cameras based on metasurfaces combine the advantages of metasurfaces and hyperspectral imaging technologies. Traditional hyperspectral cameras usually require bulky optical components (such as prisms or gratings) to separate spectra, which limits their application scenarios. The introduction of metasurface technology can significantly reduce the size and weight of hyperspectral cameras while improving spectral resolution and sensitivity. This new camera uses the special optical properties of metasurfaces to achieve efficient spectral separation and regulation of incident light by designing different nanostructures. Metasurfaces can be directly integrated on detectors, thereby simplifying the optical path design and reducing system complexity. This not only makes hyperspectral cameras more compact, but also enables high-precision spectral imaging over a wider spectral range. Compact hyperspectral cameras based on metasurfaces have broad application prospects in environmental monitoring, agriculture, medical imaging, national defense security, and other fields.

[0004] However, since the design process of metasurface cameras is very complicated and there is a problem of not being able to make good use of spatial resolution, obtaining an efficient and accurate metasurface design and imaging process is an urgent challenge. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of existing hyperspectral cameras being too large in size and having too long imaging time, as well as the problems of existing metasurface hyperspectral cameras having poor reconstruction accuracy and complex design process, and to propose a hyperspectral image reconstruction method based on a metasurface.

[0006] The specific process of the hyperspectral image reconstruction method based on super surface is as follows:

[0007] Step 1: training the metasurface response layer, and obtaining the response matrix of the metasurface unit based on the trained metasurface response layer;

[0008] Step 2: Based on the response matrix of the metasurface unit obtained in step 1, obtain the optimal metasurface response matrix arrangement;

[0009] Step 3: Derivation of the real physical size parameters of an optimal metasurface response matrix arrangement based on the inverse design algorithm;

[0010] The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens;

[0011] Step 4: directly cover the metasurface lens obtained in step 3 on the CMOS of the camera, and the camera receives the continuous spectrum of the photographed scene to obtain a multispectral image; and obtain a hyperspectral image based on the multispectral image.

[0012] The beneficial effects of the present invention are:

[0013] This invention solves the problem of hyperspectral camera being too large and taking too long to image:

[0014] Previous hyperspectral cameras were relatively large, with length, width and height all at the decimeter level. At the same time, the imaging principle of hyperspectral cameras is relatively complex, and often requires a longer acquisition time in exchange for a higher spectral resolution. The present invention utilizes the different light absorption rates of transmissive metasurfaces in different bands, and greatly improves the imaging speed by means of computational imaging, which can reach the millisecond level. At the same time, since the preparation of the metasurface is relatively simple and does not require complex assembly, the metasurface lens can be less than 1 square centimeter, which greatly reduces the size of the hyperspectral camera;

[0015] This invention solves the problems of poor reconstruction accuracy and complicated design process of existing metasurface hyperspectral cameras:

[0016] Most of the existing metasurface hyperspectral cameras are made using the forward design method, that is, the metasurface is first prepared, and then the appropriate metasurface is selected through its response matrix. This method is costly, wasteful, and has a long construction period. It is also difficult to meet the fine requirements of the camera. Since the response matrix of the metasurface and the reconstruction algorithm cannot completely match, the existing metasurface hyperspectral cameras have poor reconstruction accuracy. Steps one, two, and three of the present invention propose a process for reverse designing a metasurface based on a reconstruction network, which can meet the algorithm's requirements for the characteristics of the metasurface response matrix. Step four of the present invention proposes a reconstruction algorithm update process based on the metasurface response matrix, which can make the reconstruction algorithm further adapt to the metasurface response matrix. Therefore, the present invention solves the problems of poor reconstruction accuracy and complex design process of existing hypersurface hyperspectral cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the present invention;

[0018] Figure 2 It is a general schematic diagram of the response curves of nine metasurfaces;

[0019] Figure 3 Schematic diagrams of nine kinds of metasurface response curves, a is a schematic diagram of metasurface response 1 curve, b is a schematic diagram of metasurface response 2 curve, c is a schematic diagram of metasurface response 3 curve, d is a schematic diagram of metasurface response 4 curve, e is a schematic diagram of metasurface response 5 curve, f is a schematic diagram of metasurface response 6 curve, g is a schematic diagram of metasurface response 7 curve, h is a schematic diagram of metasurface response 8 curve, and i is a schematic diagram of metasurface response 9 curve;

[0020] Figure 4 Schematic diagram of the super Bayer array structure;

[0021] Figure 5 Schematic diagram of the metasurface microlens structure. DETAILED DESCRIPTION

[0022] Specific implementation method 1: The specific process of the hyperspectral image reconstruction method based on the super surface in this implementation method is as follows:

[0023] Step 1: training the metasurface response layer, and obtaining the response matrix of the metasurface unit based on the trained metasurface response layer;

[0024] Step 2: Based on the response matrix of the metasurface unit obtained in step 1, obtain the optimal metasurface response matrix arrangement;

[0025] Step 3: Derivation of the real physical size parameters of an optimal metasurface response matrix arrangement based on the inverse design algorithm;

[0026] The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens;

[0027] Step 4: directly cover the metasurface lens obtained in step 3 on the CMOS of the camera, and the camera receives the continuous spectrum of the photographed scene to obtain a multispectral image; and obtain a hyperspectral image based on the multispectral image.

[0028] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that in step 1, the super surface response layer is trained to obtain the trained super surface response layer, and the response matrix of the super surface unit is obtained based on the trained super surface response layer; the specific process is:

[0029] Step 1: Obtain the original hyperspectral image H;

[0030] Step 1 and 2: the original hyperspectral image H is passed through the super surface response layer to reduce the dimension into a multispectral image, and the multispectral image is passed through a reconstruction module, and the reconstruction module outputs a reconstructed hyperspectral image H′;

[0031] Step 13, repeating steps 1 and 2, the parameters of the reconstruction module remain unchanged, and adjusting the parameters of the metasurface response layer until the root mean square error MSE between the original hyperspectral image H and the reconstructed hyperspectral image H′ meets the requirements, and a trained metasurface response layer is obtained, that is, the optimal metasurface response matrix is ​​obtained; during training, the parameters of the reconstruction module remain unchanged, and only the parameters of the metasurface response layer are changed;

[0032] The metasurface response layer is a self-attention layer (Self-Attention);

[0033] The reconstruction module is an image reconstruction model MST;

[0034] The original hyperspectral image H is input into the trained metasurface response layer, and the trained metasurface response layer outputs an n×n×9 metasurface response matrix;

[0035] Where n is the number of spectra of the original hyperspectral image H, and 9 represents the response matrices of 9 different metasurface units.

[0036] The other steps and parameters are the same as those in the first embodiment.

[0037] Specific implementation method three: This implementation method is different from specific implementation method one or two in that, in step two, based on the response matrix of the metasurface unit obtained in step one, the optimal metasurface response matrix arrangement is obtained; the specific process is:

[0038] Step 21: Design a 4×4 sixteen-square metasurface microlens array, with rows arranged from top to bottom and columns arranged from left to right;

[0039] Step 22: Based on the response matrices of the 9 different metasurface units in step 1, nine metasurface response matrix arrangements are obtained;

[0040] Step 2 and 3: Each metasurface response matrix arrangement is used as a new metasurface response layer, and the original hyperspectral image H is passed through the new metasurface response layer to reduce the dimension into a multispectral image. The multispectral image is passed through a reconstruction module, and the reconstruction module outputs a reconstructed hyperspectral image H′;

[0041] Nine metasurface response matrix arrangements correspond to nine reconstructed hyperspectral images H′;

[0042] Step 24: Compare the root mean square error (MSE) between the original hyperspectral image H and the nine reconstructed hyperspectral images H′, and take the metasurface response matrix arrangement corresponding to the minimum value of the root mean square error (MSE) as the optimal metasurface response matrix arrangement.

[0043] The other steps and parameters are the same as those in the first or second embodiment.

[0044] Specific implementation method 4: This implementation method is different from the specific implementation methods 1 to 3 in that, in step 22, nine metasurface response matrix arrangements are obtained based on the response matrices of the 9 different metasurface units in step 1; the specific process is:

[0045] Place the response matrix of the first type of metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight types of metasurface units at each of the remaining positions to obtain a first metasurface response matrix arrangement mode;

[0046] Place the response matrix of the second type of metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight types of metasurface units at each of the remaining positions to obtain a second metasurface response matrix arrangement;

[0047] Place the response matrix of the third type of metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight types of metasurface units at each of the remaining positions to obtain a third metasurface response matrix arrangement;

[0048] Place the response matrix of the fourth type of metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight types of metasurface units at each of the remaining positions to obtain a fourth metasurface response matrix arrangement;

[0049] Place the response matrix of the fifth metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain a fifth metasurface response matrix arrangement;

[0050] Place the response matrix of the sixth metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain a sixth metasurface response matrix arrangement;

[0051] Place the response matrix of the seventh metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain the seventh metasurface response matrix arrangement;

[0052] Place the response matrix of the eighth metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain an eighth metasurface response matrix arrangement;

[0053] Place the response matrix of the ninth metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain a ninth metasurface response matrix arrangement;

[0054] The main component was placed eight times, and the remaining eight auxiliary components were placed once each randomly;

[0055] This arrangement of the super-surface microlens array is subsequently referred to as a super-Bayer array. The specific implementation method is as follows: Figure 4 shown.

[0056] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0057] Through the above-mentioned design method of the metasurface response matrix, the optimal response matrix in the sense of specific scene data has been obtained. Taking the Nx9-dimensional metasurface response matrix as an example, where N is the number of hyperspectral spectra, nine metasurface unit responses can be obtained. When actually making a camera, since it is impossible to obtain nine metasurface responses of a pixel at the same time and position, a certain degree of design is required in the spatial or temporal dimension. A super Bayer array structure is now proposed, and the specific design is as follows: Figure 4 shown.

[0058] Different numbers represent different metasurfaces. The largest number represents the most important and main component in the image spectrum, similar to the green filter in a normal Bayer array, which is used as a reference for the subsequent reconstruction demosaicing algorithm. Finally, using the same demosaicing algorithm, all arrangements of the nine metasurface filters are calculated, and the arrangement of the super Bayer array with the best demosaicing effect is selected.

[0059] A metasurface unit arrangement method is proposed, whose basic structure is a sixteen-grid rectangular array structure, which includes nine or more different metasurface units. The arrangement of the metasurface units is similar to the Bayer array structure, in which different types of metasurface structures are alternately arranged to form a mosaic array. It is worth mentioning that the key to this arrangement is to select one metasurface unit as the main component of the array, and the remaining eight as auxiliary components. Taking a sixteen-grid array as an example, the rows are sorted from top to bottom, and the columns are sorted from left to right, using odd rows and odd columns such as one row and one column, one row and three columns, two rows and two columns, two rows and four columns, three rows and one column, three rows and three columns, four rows and two columns, and four rows and four columns. The same main component metasurface units are placed in even rows and even columns, and different types of auxiliary component metasurface structures are placed in each of the remaining positions. This arrangement of the metasurface array is subsequently referred to as a super-Bayer array, and the specific implementation method is as follows: Figure 4 shown.

[0060] like Figure 4As shown in the figure, different numbers represent different metasurface structures, which are composed of a main structure and eight auxiliary structures. The smallest number represents the most important and main component in the image spectrum, similar to the green filter in the ordinary Bayer array. It retains most of the most important information in the image and can be used as a reference for the subsequent reconstruction demosaicing algorithm. The demosaicing algorithm here is not the mosaic removal technology in the current broad sense, but an algorithm dedicated to the Bayer array that converts grayscale images into color images and improves the resolution. Finally, using the same demosaicing algorithm, the information on the light intensity received by the subsequent photosensitive element, that is, the grayscale image, is demosaiced and converted into a metasurface response multispectral image. After calculating all the arrangements of the nine metasurface filters, the arrangement of the super Bayer array with the best demosaicing effect is selected.

[0061] Specific implementation method five: This implementation method is different from any one of specific implementation methods one to four in that in step three, the real physical size parameters of an optimal metasurface response matrix arrangement are derived based on the inverse design algorithm;

[0062] The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens;

[0063] The specific process is:

[0064] Step 31: Obtain basic data;

[0065] Step 32: Obtain a trained neural network model;

[0066] Step 3. Reversely derive the real physical size parameters of an optimal metasurface response matrix arrangement;

[0067] The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens.

[0068] It can then be prepared into a physical object and used for subsequent image acquisition tasks.

[0069] Deriving the real metasurface physical size parameter information from the metasurface response curve is a key step that affects the accuracy of subsequent image reconstruction. The traditional design process usually involves preparing the metasurface first, measuring its physical properties through experiments, and then using these physical properties to make related instruments. This process is called forward design. With the development of simulation technology and the improvement of metasurface preparation technology, the uncertainty and operational difficulty in forward design are increasingly unable to meet people's demand for metasurface performance, thus evolving into a reverse design method.

[0070] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0071] Specific implementation method 6: This implementation method is different from specific implementation methods 1 to 5 in that basic data is obtained in step 31; the specific process is:

[0072] Use numerical simulation methods such as finite difference time domain method (FDTD) or finite element analysis (FEA) to obtain a large amount of basic data of metasurface structures;

[0073] The basic data of the metasurface structure include the physical structural parameters of each metasurface unit and the corresponding metasurface response matrix (metasurface response matrix n×n);

[0074] Physical structural parameters such as the size, spacing, and repetition period of micro- and nano-pillars;

[0075] During the simulation process, different parameter combinations need to be set to cover various possible structural designs to ensure the comprehensiveness and diversity of the data.

[0076] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0077] Specific implementation method 7: This implementation method is different from the specific implementation methods 1 to 6 in that the trained neural network model is obtained in step 32; the specific process is:

[0078] The neural network model is convolutional neural network (CNN), Transformer, generative adversarial network (GAN) or graph neural network (GNN);

[0079] Step 3: the super surface response matrix in the preprocessed basic data is used as the input of the neural network model, the physical structure parameters of each super surface unit in the preprocessed basic data is used as the output of the neural network model, and the neural network model is trained to obtain a trained neural network model;

[0080] Design a deep neural network with a multi-layer structure. The input layer of the network is the physical structure parameters, and the output layer is the metasurface response matrix.

[0081] The feature extraction models used include, but are not limited to, convolutional neural networks (CNN), Transformer, generative adversarial networks (GAN), and graph neural networks (GNN). The specific choice depends on the specific task and data.

[0082] The neural network is trained using supervised learning methods; the preprocessed training data is input into the model, and the network weights are continuously adjusted through the back-propagation algorithm to minimize the prediction error.

[0083] Use the validation set to verify the model in real time to prevent overfitting.

[0084] After training is completed, the test set is used to evaluate the performance of the model to ensure the generalization ability and prediction accuracy of the model.

[0085] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0086] Specific implementation eight: This implementation differs from specific implementations one to seven in that in step three, reverse deduction is performed in step three to obtain the real physical size parameters of an optimal metasurface response matrix arrangement mode;

[0087] The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens; the specific process is:

[0088] The response matrices of the nine different metasurface units in step one are respectively used as inputs of the neural network model trained in step three and two, and the trained neural network model outputs the physical structure parameters of the nine metasurface units;

[0089] Arrange the physical structural parameters of the nine metasurface units output by the trained neural network model according to the optimal metasurface response matrix arrangement in step 2 to obtain the real physical size parameters of an optimal metasurface response matrix arrangement;

[0090] The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens.

[0091] It can then be prepared into a physical object and used for subsequent image acquisition tasks.

[0092] Deriving the real metasurface physical size parameter information from the metasurface response curve is a key step that affects the accuracy of subsequent image reconstruction. The traditional design process usually involves preparing the metasurface first, measuring its physical properties through experiments, and then using these physical properties to make related instruments. This process is called forward design. With the development of simulation technology and the improvement of metasurface preparation technology, the uncertainty and operational difficulty in forward design are increasingly unable to meet people's demand for metasurface performance, thus evolving into a reverse design method.

[0093] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0094] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that, in step 4, the metasurface lens obtained in step 3 is directly covered on the CMOS of the camera, and the camera receives the continuous spectrum of the photographed scene to obtain a multispectral image; a hyperspectral image is obtained based on the multispectral image;

[0095] The specific process is:

[0096] Step 41: directly cover the metasurface lens obtained in step 3 on the CMOS of the camera, and the camera receives continuous hyperspectral images of the captured scene to obtain multispectral images;

[0097] Step 42: extract the gradient, brightness and texture information of the image at the positions of odd rows and odd columns, and even rows and even columns of the multispectral image, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column; the brightness and darkness are directly determined by the grayscale value; the gradient is to find the derivative, and the image is Fourier transformed to obtain the texture;

[0098] Step 43, extract all pixel points at the positions of odd rows and odd columns, and even rows and even columns such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of the multispectral image;

[0099] For the pixel points in the first row and the first column of the extracted pixel points, find the pixel points similar to the pixel points in the first row and the first column, take the pixel points in the first row and the first column, the four pixel points above, below, left, and right of the pixel points in the first row and the first column, the pixel points similar to the pixel points in the first row and the first column, and the four pixel points above, below, left, and right of the pixel points similar to the pixel points in the first row and the first column as the first group of similar pixel points, solve the equations for the first group of similar pixel points simultaneously, obtain the multispectral value M and the super surface response matrix F, and obtain the hyperspectral value H of the pixel points in the first row and the first column of the hyperspectral based on M and F; the hyperspectral value H multiplied by the super surface response matrix F=multispectral value M;

[0100] For the pixel point in the first row and second column of the extracted pixel points, find the pixel point similar to the pixel point in the first row and second column, take the pixel point in the first row and second column, the four pixel points above, below, left, and right of the pixel point in the first row and second column, the pixel point similar to the pixel point in the first row and second column, and the four pixel points above, below, left, and right of the pixel point similar to the pixel point in the first row and second column as the second group of similar pixel points, solve the equations for the second group of similar pixel points simultaneously, obtain the multispectral value M and the super surface response matrix F, and obtain the hyperspectral value H of the pixel point in the first row and second column of the hyperspectral based on M and F; the hyperspectral value H multiplied by the super surface response matrix F=multispectral value M;

[0101] For the pixel point in the first row and the third column of the extracted pixel points, find the pixel point similar to the pixel point in the first row and the third column, take the pixel point in the first row and the third column, the four pixel points above, below, left, and right of the pixel point in the first row and the third column, the pixel point similar to the pixel point in the first row and the third column, and the four pixel points above, below, left, and right of the pixel point similar to the pixel point in the first row and the third column as the third group of similar pixel points, solve the equations for the third group of similar pixel points simultaneously, obtain the multispectral value M and the super surface response matrix F, and obtain the hyperspectral value H of the pixel point in the first row and the third column of the hyperspectral based on M and F; the hyperspectral value H multiplied by the super surface response matrix F=multispectral value M;

[0102] Until the hyperspectral value H corresponding to all pixels is obtained.

[0103] The metasurface response matrix corresponding to the pixel at each position is fixed, that is, it is calculated in step one of the entire patent and is known.

[0104] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0105] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 1 to 9 in that the similar pixel point determination process is as follows:

[0106] Calculate the similarity C of the gradients of pixel A and pixel B;

[0107] Calculate the similarity D between the brightness of pixel A and pixel B;

[0108] Calculate the similarity E of the texture information of pixel A and pixel B;

[0109] The similarity between pixel A and pixel B is obtained by taking a weighted average of the similarity C of the gradient, the similarity D of the brightness and the similarity E of the texture information;

[0110]

[0111] Among them, G is the similarity between pixel A and pixel B;

[0112] λ1 is the weight coefficient of the similarity C of the gradients of pixel A and pixel B, λ2 is the weight coefficient of the similarity D of the gradients of pixel A and pixel B, and λ3 is the weight coefficient of the similarity E of the gradients of pixel A and pixel B; λ1+λ2+λ3=1;

[0113] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0114] The hyperspectral data set is sampled using the super-Bayer array to obtain multi-spectral observation results of the metasurface. Image reconstruction is essentially a process of solving an underdetermined set of equations. By combining more equations and obtaining the responses of the same spectral pixels under different metasurfaces, the underdetermined equations can be better constrained to a certain extent, significantly improving the accuracy of the solution. In the super-Bayer array design, the main components have the highest arrangement density on the metasurface, and the reconstruction algorithm needs to rely on the main component pixels as a framework for reconstruction, ensuring the consistency and stability of the reconstruction process.

[0115] Taking the super-Bayer array with 9 different super-surface units as an example, since the main components need to be reused, a transformation structure with a maximum of 4x4 and a minimum of 2x2 will be formed, which is highly adaptive. In the design of the image reconstruction algorithm, a principal component skeleton reference module is added, and the basic features of the image, such as gradient, brightness and texture, are obtained more accurately through the dense structure of the No. 1 super-surface. The principal component skeleton reference module uses a non-depth method to obtain the basic information of these images, which can better pre-extract similar spectral information pixels to constrain the underdetermined equations. Through the scaling of the super-Bayer array, a small transformation structure can be used to adapt to the boundaries of highly changing frequencies and reduce the sawtooth effect during reconstruction. It can also adapt to larger areas of pixels and use large transformation structures to further improve the accuracy of reconstruction. Using this method can improve the speed and quality of reconstruction to a certain extent.

[0116] This module achieves optimization through the following steps:

[0117] Dense structural information extraction: Using the dense arrangement of the No. 1 hypersurface, the gradient, brightness and texture information of the image are extracted to form the main component skeleton. This step not only improves the extraction accuracy of basic features, but also provides a solid foundation for the subsequent spectral information selection.

[0118] Spectral similarity selection: Based on the extracted basic features of the image, pixels with similar spectral information are adaptively selected to ensure the consistency of spectral data during the reconstruction process. By comparing similar spectral information, errors can be effectively reduced and the accuracy of spectral reconstruction can be improved.

[0119] Multi-equation solution: Combined with the various response structures of the Super Bayer array, more equations are solved simultaneously to improve the accuracy of the solution. Optimization algorithms such as least squares and non-negative matrix decomposition are used to solve underdetermined equations to ensure the stability and accuracy of the solution.

[0120] Adaptive adjustment: Adaptively adjust the reconstruction algorithm according to the response characteristics of different metasurface structures to improve the stability and adaptability of the algorithm. By continuously adjusting and optimizing parameters, ensure that the algorithm can perform well in different application scenarios.

[0121] Multi-scale analysis: Combine multi-scale analysis methods to reconstruct images at different scales to ensure the unity of details and overall structure and improve the quality of reconstructed images.

[0122] The introduction of the principal component skeleton reference module significantly enhances the accuracy and efficiency of image reconstruction. While ensuring consistency and stability in the image reconstruction process, it also enables the reconstruction algorithm to perform well when processing complex hyperspectral data.

[0123] The process of restoring a multispectral image to a hyperspectral image by the reconstruction network can be regarded as a process of solving an underdetermined set of equations. For each pixel point in the hyperspectral image, it is necessary to select multiple pixels with similar spectra near the same pixel position in the multispectral image to solve the set of equations simultaneously to complete the process of restoring a multispectral image to a hyperspectral image.

[0124] In step 2, since the metasurface structure is divided into one main component and eight auxiliary components according to the arrangement, and a 4x4 sixteen-square grid is formed, the restoration network from multispectral images to hyperspectral images is now completed according to the following steps.

[0125] First, dense structural information extraction: using the No. 1 hypersurface, that is, the dense arrangement of the main components, to extract the gradient, brightness and texture information of the image, which is for the subsequent more accurate selection of spectral similarities. This step not only improves the extraction accuracy of basic features, but also provides a solid foundation for the subsequent selection of spectral information.

[0126] Second, spectral similarity selection: traverse each pixel in the multispectral image and search for pixels similar to the selected pixel among other pixels in the image. Based on the basic image features extracted in the first step, the similarity between pixels is calculated under different evaluation indicators (such as gradient, brightness, texture similarity, etc.), and pixels with similar spectral information are adaptively selected to ensure the consistency of spectral data during the reconstruction process. By comparing similar spectral information, errors can be effectively reduced and the accuracy of spectral reconstruction can be improved.

[0127] Third, solve multiple equations simultaneously: In the previous step, multiple pixels with similar spectra are dynamically selected. The spectral observation matrix obtained by degrading the hypersurface structure on the pixels is combined, and the optimization algorithm (such as least squares, convex optimization, deep learning, etc.) is used to complete the final spectrum restoration. Finally, each pixel in the multispectral image is restored and the results are combined into a hyperspectral image.

[0128] The introduction of the principal component skeleton reference module significantly enhances the accuracy and efficiency of image reconstruction. While ensuring consistency and stability in the image reconstruction process, it also enables the reconstruction algorithm to perform well when processing complex hyperspectral data.

[0129] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A hyperspectral image reconstruction method based on a metasurface, characterized in that: The specific process of the method is: Step 1: training the metasurface response layer, and obtaining the response matrix of the metasurface unit based on the trained metasurface response layer; Step 2: Based on the response matrix of the metasurface unit obtained in step 1, obtain the optimal metasurface response matrix arrangement; Step 3: Derivation of the real physical size parameters of an optimal metasurface response matrix arrangement based on the inverse design algorithm; The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens; Step 4: directly cover the metasurface lens obtained in step 3 on the CMOS of the camera, and the camera receives the continuous spectrum of the photographed scene to obtain a multispectral image; and obtain a hyperspectral image based on the multispectral image.

2. The hyperspectral image reconstruction method based on a supersurface according to claim 1, characterized in that: In the step 1, the metasurface response layer is trained to obtain the trained metasurface response layer, and a response matrix of the metasurface unit is obtained based on the trained metasurface response layer; The specific process is: Step 1: Obtain the original hyperspectral image H; Step 1 and 2: the original hyperspectral image H is passed through the super surface response layer to reduce the dimension into a multispectral image, and the multispectral image is passed through a reconstruction module, and the reconstruction module outputs a reconstructed hyperspectral image H′; Step 13: Repeat steps 1 and 2, keep the parameters of the reconstruction module unchanged, and adjust the parameters of the metasurface response layer until the root mean square error (MSE) between the original hyperspectral image H and the reconstructed hyperspectral image H′ meets the requirements, and obtain the trained metasurface response layer, that is, obtain the optimal metasurface response matrix; The metasurface response layer is a self-attention layer; The reconstruction module is an image reconstruction model MST; The original hyperspectral image H is input into the trained metasurface response layer, and the trained metasurface response layer outputs an n×n×9 metasurface response matrix; Where n is the number of spectra of the original hyperspectral image H, and 9 represents the response matrices of 9 different metasurface units.

3. The hyperspectral image reconstruction method based on supersurface according to claim 2, characterized in that: In the step 2, based on the response matrix of the metasurface unit obtained in step 1, the optimal metasurface response matrix arrangement is obtained; the specific process is: Step 21: Design a 4×4 sixteen-square metasurface microlens array, with rows arranged from top to bottom and columns arranged from left to right; Step 22: Based on the response matrices of the 9 different metasurface units in step 1, nine metasurface response matrix arrangements are obtained; Step 2 and 3: Each metasurface response matrix arrangement is used as a new metasurface response layer, and the original hyperspectral image H is passed through the new metasurface response layer to reduce the dimension into a multispectral image. The multispectral image is passed through a reconstruction module, and the reconstruction module outputs a reconstructed hyperspectral image H′; Nine metasurface response matrix arrangements correspond to nine reconstructed hyperspectral images H′; Step 24: Compare the root mean square error (MSE) between the original hyperspectral image H and the nine reconstructed hyperspectral images H′, and take the metasurface response matrix arrangement corresponding to the minimum value of the root mean square error (MSE) as the optimal metasurface response matrix arrangement.

4. The hyperspectral image reconstruction method based on a supersurface according to claim 3, characterized in that: In step 22, based on the response matrices of the 9 different metasurface units in step 1, nine metasurface response matrix arrangements are obtained; the specific process is: Place the response matrix of the first type of metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight types of metasurface units at each of the remaining positions to obtain a first metasurface response matrix arrangement mode; Place the response matrix of the second type of metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight types of metasurface units at each of the remaining positions to obtain a second metasurface response matrix arrangement; Place the response matrix of the third type of metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight types of metasurface units at each of the remaining positions to obtain a third metasurface response matrix arrangement; Place the response matrix of the fourth type of metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight types of metasurface units at each of the remaining positions to obtain a fourth metasurface response matrix arrangement; Place the response matrix of the fifth metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain a fifth metasurface response matrix arrangement; Place the response matrix of the sixth metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain a sixth metasurface response matrix arrangement; Place the response matrix of the seventh metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain the seventh metasurface response matrix arrangement; Place the response matrix of the eighth metasurface unit at the positions of odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and fourth column of the metasurface microlens array, and place the response matrices of the remaining eight metasurface units at each of the remaining positions to obtain an eighth metasurface response matrix arrangement; The response matrix of the ninth type of metasurface unit is placed in the odd rows and odd columns, and even rows and even columns, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column of the metasurface microlens array, and the response matrices of the remaining eight types of metasurface units are placed at each of the remaining positions to obtain the ninth metasurface response matrix arrangement.

5. The hyperspectral image reconstruction method based on supersurface according to claim 4, characterized in that: In the step 3, based on the inverse design algorithm, the real physical size parameters of an optimal metasurface response matrix arrangement are derived; The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens; The specific process is: Step 31: Obtain basic data; Step 32: Obtain a trained neural network model; Step 3. Reversely derive the real physical size parameters of an optimal metasurface response matrix arrangement; The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens.

6. The hyperspectral image reconstruction method based on the metasurface according to claim 5, characterized in that: In step 31, basic data is obtained; the specific process is: Use finite difference time domain method or finite element analysis numerical simulation method to obtain basic data of metasurface structure; The basic data of the metasurface structure include the physical structure parameters of each metasurface unit and the corresponding metasurface response matrix.

7. The hyperspectral image reconstruction method based on a supersurface according to claim 6, characterized in that: The trained neural network model is obtained in step 3.2; the specific process is: The neural network model is a convolutional neural network, Transformer, generative adversarial network, or graph neural network; The super surface response matrix in the basic data after preprocessing in step 3 is used as the input of the neural network model, and the physical structure parameters of each super surface unit in the basic data after preprocessing in step 3 are used as the output of the neural network model. The neural network model is trained to obtain a trained neural network model.

8. The hyperspectral image reconstruction method based on a supersurface according to claim 7, characterized in that: In step 33, reverse deduction is performed to obtain the real physical size parameters of an optimal metasurface response matrix arrangement mode; The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens; the specific process is: The response matrices of the nine different metasurface units in step one are respectively used as inputs of the neural network model trained in step three and two, and the trained neural network model outputs the physical structure parameters of the nine metasurface units; The physical structural parameters of the nine metasurface units output by the trained neural network model are arranged according to the optimal metasurface response matrix arrangement of step 2 to obtain the real physical size parameters of an optimal metasurface response matrix arrangement; The real physical size parameters of the optimal metasurface response matrix arrangement are made into a metasurface lens.

9. The hyperspectral image reconstruction method based on a supersurface according to claim 8, characterized in that: In the step 4, the metasurface lens obtained in the step 3 is directly covered on the CMOS of the camera, and the camera receives the continuous spectrum of the captured scene to obtain a multispectral image; a hyperspectral image is obtained based on the multispectral image; the specific process is: Step 41: directly cover the metasurface lens obtained in step 3 on the CMOS of the camera, and the camera receives continuous hyperspectral images of the captured scene to obtain multispectral images; Step 42: extract the gradient, brightness and texture information of the image at the positions of odd rows and odd columns, and even rows and even columns of the multispectral image, such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column; Step 43, extract all pixel points at the positions of odd rows and odd columns, and even rows and even columns such as the first row and first column, the first row and third column, the second row and second column, the second row and fourth column, the third row and first column, the third row and third column, the fourth row and second column, and the fourth row and fourth column of the multispectral image; For the pixel points in the first row and first column of the extracted pixel points, find the pixel points similar to the pixel points in the first row and first column, take the pixel points in the first row and first column, the four pixel points above, below, left, and right of the pixel points in the first row and first column, the pixel points similar to the pixel points in the first row and first column, and the four pixel points above, below, left, and right of the pixel points similar to the pixel points in the first row and first column as the first group of similar pixel points, solve the equations for the first group of similar pixel points simultaneously, obtain the multispectral value M and the super surface response matrix F, and obtain the hyperspectral value H of the pixel points in the first row and first column of the hyperspectral based on M and F; For the pixel point in the first row and second column of the extracted pixel points, find the pixel point similar to the pixel point in the first row and second column, take the pixel point in the first row and second column, the four pixel points above, below, left, and right of the pixel point in the first row and second column, the pixel point similar to the pixel point in the first row and second column, and the four pixel points above, below, left, and right of the pixel point similar to the pixel point in the first row and second column as the second group of similar pixel points, solve the equations for the second group of similar pixel points simultaneously, obtain the multispectral value M and the super surface response matrix F, and obtain the hyperspectral value H of the pixel point in the first row and second column of the hyperspectral based on M and F; For the pixel point in the first row and third column of the extracted pixel points, find a pixel point similar to the pixel point in the first row and third column, take the pixel point in the first row and third column, the four pixel points above, below, left, and right of the pixel point in the first row and third column, the pixel point similar to the pixel point in the first row and third column, and the four pixel points above, below, left, and right of the pixel point similar to the pixel point in the first row and third column as a third group of similar pixel points, solve the equations for the third group of similar pixel points simultaneously, obtain the multispectral value M and the super surface response matrix F, and obtain the hyperspectral value H of the pixel point in the first row and third column of the hyperspectral based on M and F; Until the hyperspectral value H corresponding to all pixels is obtained.

10. The hyperspectral image reconstruction method based on a supersurface according to claim 9, characterized in that: The similar pixel point determination process is as follows: Calculate the similarity C of the gradients of pixel A and pixel B; Calculate the similarity D between the brightness of pixel A and pixel B; Calculate the similarity E of the texture information of pixel A and pixel B; The similarity between pixel A and pixel B is obtained by taking a weighted average of the similarity C of the gradient, the similarity D of the brightness and the similarity E of the texture information; Among them, G is the similarity between pixel A and pixel B; λ1 is the weight coefficient of the similarity C of the gradients of pixel A and pixel B, λ2 is the weight coefficient of the similarity D of the gradients of pixel A and pixel B, and λ3 is the weight coefficient of the similarity E of the gradients of pixel A and pixel B; λ1+λ2+λ3=1.