Liquid crystal calculation modulator, calculation type liquid crystal spectrum imager and data reconstruction method of calculation type liquid crystal spectrum imager

Abstract

The invention provides a liquid crystal calculation modulator, a calculation type liquid crystal spectrum imager and a data reconstruction method of the calculation type liquid crystal spectrum imager. The liquid crystal calculation modulator is used for replacing an LCTF in a traditional liquid crystal spectrum imager, and a calculation type liquid crystal spectrum imager with the characteristics of wide spectrum band and high transmittance is formed. In order to solve the problem of low spectral resolution of the obtained modulated hyperspectral data, the invention also provides a reconstruction method for calculating the transmittance of the central point of the modulator by using the known liquid crystal and solving an underdetermined problem. The computational liquid crystal spectral imager gives consideration to spectral resolution, transmittance and free spectral range, and the data reconstruction method can realize scene reconstruction with low cost and high spectral resolution based on data acquired by the computational liquid crystal spectral imager and a trained model.

Description

Technical Field

[0001] The present invention relates to the technical field of spectral imaging, and in particular, to a liquid crystal computing modulator, a computing liquid crystal spectral imager, and a data reconstruction method thereof. Background Art

[0002] A liquid crystal spectral imager is an instrument that combines liquid crystal technology and spectral imaging technology. Liquid crystal spectral imaging technology is a new type of spectral imaging technology based on the electro-optic birefringence effect of liquid crystals. It adopts a planar array staring imaging mode and realizes continuous spectral tuning through electro-control. This technology can obtain spectral images of targets with high temporal resolution, high spatial resolution, and high spectral resolution, and has the advantages of small volume, light weight, and easy integration. A liquid crystal spectral imager modulates incident light through a liquid crystal tunable filter (LCTF), thereby realizing selective transmission of light of different wavelengths, and then obtaining the spectral information of the target. Liquid crystal spectral imagers have been widely used in the fields of remote sensing, medical imaging, materials science, etc. In the field of remote sensing, liquid crystal spectral imagers can achieve high-precision monitoring of targets such as surface vegetation, water bodies, and soil; in the field of medical imaging, liquid crystal spectral imagers can achieve spectral imaging and diagnosis of biological tissues; in the field of materials science, liquid crystal spectral imagers can achieve analysis of material composition, structure, and properties.

[0003] Liquid crystal tunable filters are the core spectral splitting devices of liquid crystal spectrometers. They are mainly composed of Lyot-type or Solc-type filters. The former is composed of a series of twisted nematic liquid crystal crystals and parallel polarizers stacked together in a sandwich manner; the latter is composed of exactly the same twisted nematic liquid crystal crystals placed between two polarizers in a cascaded manner with a certain orientation. The spectral resolution ability of the Lyot-type filter has a geometric series relationship with the number of crystals, while the spectral resolution ability of the Solc-type filter has a linear relationship with the number of crystals. In order to achieve a high spectral resolution ability, a large number of crystals are often required to be stacked. Therefore, compared with the Solc-type filter, the Lyot-type filter usually only requires fewer birefringent crystals to achieve a comparable spectral resolution to the former. In terms of peak transmittance, since the Solc-type filter does not use polarizers inside, it can reduce the loss of light energy. Therefore, at the same spectral resolution, the light energy utilization rate of the Lyot-type is lower. Generally speaking, in the design and manufacturing process of liquid crystal tunable filters, the spectral resolution and the full width at half maximum are inversely proportional to each other, and their performances restrict each other; although more liquid crystal cell stages can improve the spectral resolution, it will continuously decrease the light transmittance and also cause more sidelobe effects. Among the commercially available liquid crystal tunable filters, relatively mature products include the Varispec series liquid crystal tunable filters of CRI in the United States, the liquid crystal tunable filter components for fluorescence imaging manufactured by PerkinElmer in the United States, the Kurios series liquid crystal tunable filters of Thorlabs in the United States, and the LCTF series of Zhongda Ruihe in China, etc. However, the performance parameters demonstrated by these products are also made by compromising among the spectral resolution, peak transmittance, and free spectral range according to the requirements of the usage scenarios.

[0004] To balance spectral resolution, transmittance, and free spectral range, researchers have made many attempts. In 2006, Zhang Jian et al. from Qufu Normal University studied a three-stage Lyot-type liquid crystal tunable filter, but due to the small number of stages, its performance was poor. In 2008, Yang Guowei et al. from Zhejiang University fabricated a filter combining Lyot and Sloc structures. Although the number of stages was reduced in the liquid crystal tunable filter made with this filter, the number of liquid crystal cells used did not decrease or even increased. In 2010, Wuxi Aoda Optoelectronics Company developed a four-stage liquid crystal tunable filter combining Lyot and Solc structures, which could achieve a peak transmittance of nearly 20% (@550nm) while achieving a full width at half maximum of 7.5nm in the spectral range of 420 - 700nm. In 2016, Xia Gang et al. from Shanghai University of Engineering Science studied the possibility of using a two-piece Solc-type or Lyot-type to replace the Solc-type or Lyot-type to solve the problem of sidelobes near the main peak of its transmittance. In 2023, Chen Lixin et al. from Shantou University proposed to optimize a five-stage Lyot-type liquid crystal tunable filter using an artificial immune algorithm, improving the sidelobe suppression effect. Although these structural improvements based on existing liquid crystal spectrometers or new structures of liquid crystal tunable filters can improve some performance, the inherent contradiction between their spectral resolution and transmittance remains unresolved.

[0005] In addition, there is also a contradiction in liquid crystal spectral imagers that the acquired image is actually a broadband modulated spectral image, while the actual user demand is a narrowband modulated image. The common solution is data reconstruction. In the early research on reconstruction algorithms, domestic and foreign scholars proposed methods such as weighted linear regression, nonlinear dimensionality reduction, kernel partial least squares, Wiener estimation, camera response value expansion, and sparse coding to reconstruct hyperspectral data. These methods mostly utilize various prior knowledge such as camera spectral sensitivity, spatio-spectral sparsity of hyperspectral data, exposure time, global context information, and spatial features to establish the mapping relationship between a single RGB image or a single RGB pixel value in it and hyperspectral data. However, this easily ignores the common spectral features among multiple images, resulting in low reconstruction accuracy and poor model generalization performance. To further improve the result of reconstructing hyperspectral data from RGB images and enhance the generalization accuracy of the model, data-driven algorithms, especially deep learning-based algorithms, have been gradually and widely used for this task. Currently, most CNN-based spectral reconstruction methods are dedicated to designing deeper or wider network architectures to obtain high-level feature representations, and the networks designed based on these ideas exhibit strong reconstruction performance. However, due to the limited receptive field of CNN convolution operations, there are inherent limitations in capturing rich context information, non-local self-similarity, and long-range interdependencies, restricting the continuous improvement of reconstruction accuracy.

[0006] In 2022, Cai et al. first proposed the MST++ method based on the Transformer model, and the reconstruction effect of this method is significantly improved compared with the AWAN model. Based on the hyperspectral spatial sparsity and spectral similarity, the MST++ method proposed a spectral-wise attention unit SAB (Spectral-wise Attention Block), and used this unit to construct a single-stage spectral-wise SST (Single-stage Spectral-wise Transformer) for extracting multi-resolution context information. Finally, SSTs were stacked to form MST+, and the hyperspectral reconstruction quality was continuously improved from coarse to fine. However, the MST++ method is not applicable to computational liquid crystal spectrometers, that is, the broadband modulation data obtained by computational liquid crystal spectrometers still cannot directly complete spectral reconstruction relying on existing models. Summary of the Invention

[0007] The present invention aims to solve at least one of the above technical problems existing in the prior art.

[0008] To this end, a first aspect of the present invention provides a liquid crystal computational modulator.

[0009] A second aspect of the present invention provides a computational liquid crystal spectrometer.

[0010] A third aspect of the present invention provides a data reconstruction method for a computational liquid crystal spectrometer.

[0011] The present invention provides a computational liquid crystal spectrometer based on broadband modulation, including:

[0012] A first-stage light-transmitting unit, including a plurality of first liquid crystal sheets with the same thickness and a plurality of first quartz sheets with the same thickness. The plurality of first liquid crystal sheets and the plurality of first quartz sheets are arranged alternately in sequence, and the number of the first liquid crystal sheets is equal to the number of the first quartz sheets;

[0013] A first polarizer, arranged on one side of the first-stage light-transmitting unit close to the incident direction of light;

[0014] A second polarizer, arranged on one side of the first-stage light-transmitting unit close to the exit direction of light, and the polarization direction of the second polarizer is perpendicular to the polarization direction of the first polarizer;

[0015] A second-stage light-transmitting unit, including a plurality of second liquid crystal sheets with the same thickness and a plurality of second quartz sheets with the same thickness. The plurality of second liquid crystal sheets and the plurality of second quartz sheets are arranged alternately in sequence, and the number of the second liquid crystal sheets is equal to the number of the second quartz sheets; the thickness of the second liquid crystal sheet is greater than the thickness of the first liquid crystal sheet, and the thickness of the second quartz sheet is greater than the thickness of the second liquid crystal sheet; the second polarizer is located between the first-stage light-transmitting unit and the second-stage light-transmitting unit;

[0016] A third polarizer, wherein the second-stage light-transmitting unit is disposed between the second polarizer and the third polarizer; the polarization direction of the third polarizer is the same as the polarization direction of the first polarizer;

[0017] The first polarizer, the first-stage light-transmitting unit, the second polarizer, the second-stage light-transmitting unit, and the third polarizer are arranged in sequence in the light propagation direction.

[0018] The liquid crystal computing modulator according to the above technical solution of the present invention may further have the following additional technical features:

[0019] In the above technical solution, the optical axis directions of all the liquid crystal sheets in the liquid crystal computing modulator alternately maintain an angle of α and -α with the first direction in sequence;

[0020] Wherein, the light propagation direction is defined as the y-axis direction, and the first direction is the x-axis direction perpendicular to the light propagation direction.

[0021] In the above technical solution, the number of the first liquid crystal sheets and the second liquid crystal sheets is 3;

[0022] The thickness of the second liquid crystal sheet is twice the thickness of the first liquid crystal sheet;

[0023] The thickness of the second quartz sheet is twice the thickness of the first quartz sheet.

[0024] In the above technical solution, the transmittance of the first-stage light-transmitting unit is:

[0025]

[0026] The transmittance of the second-stage light-transmitting unit is:

[0027]

[0028] The total transmittance of the liquid crystal computing modulator is:

[0029] T LCMD (λ)=T LCMD_1 T LCMD_2

[0030] Wherein, T LCMD_1 represents the transmittance of the first-stage light-transmitting unit; λ represents the wavelength; δ1 and α1 are both coefficients related to the driving voltage; Δn=n e -n0 represents the birefringence of the crystal in the liquid crystal sheet, n e represents the extraordinary light refractive index, n0 represents the ordinary light refractive index; d Q1 represents the thickness of the first quartz sheet; d LC1 represents the thickness of the first liquid crystal sheet; TLCMD_2 represents the transmittance of the second - level light - transmitting unit; T LCMD represents the total transmittance of the liquid - crystal computational modulator.

[0031] A computational liquid - crystal spectral imager based on broadband modulation provided by the present invention includes:

[0032] An imaging lens that collects light reflected or emitted by a target object;

[0033] The liquid - crystal computational modulator as described in any one of the above - mentioned technical solutions, which realizes spectral addressing or multispectral scanning by adjusting the driving voltage to complete continuous spectral analysis within a preset spectral band; wherein, the liquid - crystal computational modulator realizes selective transmission of light with different wavelengths by changing the position of the spectral passband;

[0034] A detector that receives the light after passing through the imaging lens and spectral modulation and converts it into an electrical signal; the converted electrical signal is processed and analyzed to generate a spectral image of the target scene.

[0035] A data reconstruction method for a computational liquid - crystal spectral imager based on broadband modulation provided by the present invention performs data reconstruction based on a computational liquid - crystal spectral imager based on broadband modulation as described in the above - mentioned technical solutions. The data reconstruction method includes:

[0036] Establish a guiding database of the transmittance of the liquid - crystal computational modulator;

[0037] Use the computational liquid - crystal spectral imager and a calibrated spectral imager to collect spectral data of several scenes and several ground objects. Among them, the spectral data collected by the computational liquid - crystal spectral imager corresponds and matches the spectral data collected by the calibrated spectral imager, and an acquisition database is formed according to the collected spectral data;

[0038] Establish a spectral - dimension deep - learning model guided by the transmittance of the liquid - crystal computational modulator. The deep - learning model is based on the MST++ network as the basic framework, and a continuous - spectral - band selection method is added to the input end of the network to pre - process the input data; and the spectral multi - head self - attention mechanism in the MST++ network is replaced by a transmittance - guided spectral - dimension multi - head self - attention mechanism. In the transmittance - guided spectral - dimension multi - head self - attention mechanism, when calculating the attention value, the feature vectors of the image blocks are re - weighted based on the transmittance data in the guiding database;

[0039] Taking the spectral data of the computational liquid crystal spectral imager in the acquisition database as the network input of the deep learning model, and taking the spectral data of the calibrated spectral imager in the acquisition database as the network output of the deep learning model, training the deep learning model to make the reconstruction result of the spectral data of the computational liquid crystal spectral imager gradually approach the spectral data of the calibrated spectral imager, and obtaining the trained deep learning model;

[0040] Inputting the spectral data newly collected by the computational liquid crystal spectral imager into the trained deep learning model to reconstruct the spectral information of the corresponding scene or the corresponding ground object.

[0041] According to the data reconstruction method of the computational liquid crystal spectral imager based on broadband modulation of the above technical solution of the present invention, the following additional technical features may also be included:

[0042] In the above technical solution, adding a continuous spectral band selection method at the input end of the network to preprocess the input data includes:

[0043] Dividing the spectral data H of the computational liquid crystal spectral imager input into the network into C parts, where the first parts contain continuous spectral bands, and the last parts contain continuous bands, and n represents the number of sampling bands;

[0044] Summing the spectral data within each part and performing normalization processing to obtain the final input data.

[0045] In the above technical solution, replacing the spectral multi-head self-attention mechanism in the MST++ network with a transmittance-guided spectral-dimensional multi-head self-attention mechanism includes:

[0046] Establishing a transmittance guidance matrix according to the transmittance of the liquid crystal computational modulator in the guidance database:

[0047] LT=(W1T L )⊙(1+δ(f dw (W2W1T L )))

[0048] where LT represents the transmittance guidance matrix; both W1 and W2 represent 1×1 convolutional layers of learnable parameters; T L represents the transmittance data of the liquid crystal computational modulator in the guidance database; f dw (·) represents a 7×7 convolutional layer in the depth direction; δ(·) represents the sigmoid activation function;

[0049] Dividing the transmittance guidance matrix into N heads in the spectral dimension LT=[LT1,LT2,...LTN ;

[0050] For any multi-head self-attention b j , by using LT j to re-weight the feature vectors of image patches to guide the spectral attention change:

[0051]

[0052] where b j represents the attention value of the j-th image patch; σ j represents a learnable parameter; represents the attention score between the i-th image patch and the j-th image patch; v j represents the feature vector of the j-th image patch.

[0053] In the above technical solution, a single-layer spectral neural network is constructed according to the transmittance-guided spectral multi-head self-attention mechanism, and the single-layer spectral neural network adopts a U-shaped structure to extract multi-resolution spectral context information;

[0054] The spectral deep learning model guided by the transmittance of the liquid crystal computational modulator is formed by stacking the single-layer spectral neural networks.

[0055] In the above technical solution, when using a computational liquid crystal spectral imager to collect spectral data, the number of sampling bands is equal to the number of modulation times of the liquid crystal computational modulator in the guidance database.

[0056] In summary, due to the adoption of the above technical features, the beneficial effects of the present invention are:

[0057] The peak transmittance of traditional narrowband filtering devices such as LCTF is usually about 15%-20%. The peak transmittance of the liquid crystal computational modulator (abbreviated as LCMD) developed in the present invention can reach 30%-35%, and at the same time has a wider passband range. This means that within the same exposure time, the LCMD device provided by the present invention can pass more spectral information.

[0058] Since the LCMD adopts a two-stage structure with fewer stages, it has a lower cost and a thinner thickness. It is not only convenient for later optical and mechanical design, but also easy to combine with the existing optical camera structure, reducing the cost of design and component procurement. At the same time, it can replace the position of the LCTF in the spectral imager with a liquid crystal device as the core, reducing the volume and cost of the original instrument.

[0059] The spectral resolution of a liquid crystal spectral modulator is usually not uniform within the spectral range. Generally speaking, the longer the wavelength, the lower the spectral resolution. Spectral imaging instruments based on liquid crystal spectral modulators also mostly have the same characteristics. The spectral resolution of the computational liquid crystal spectral imager (abbreviated as CLSI) with the LCMD proposed in the present invention as the core comes from the learning of the spectral resolution of the scene data collected by a standard instrument by a deep learning model, so it is not restricted by the technical characteristics of the liquid crystal spectral modulator, and the spectral data obtained can still maintain a high spectral resolution accuracy in the part with a longer wavelength.

[0060] The existing RGB reconstruction hyperspectral data model is to solve the problem of reconstructing a multi-band RGB image into a hyperspectral image with dozens of bands; when solving the reconstruction of the wide spectral band data collected by CLSI, it is to solve the problem of reconstructing several modulated spectral images into a hyperspectral image with dozens of bands. The two are not interoperable, so a dedicated reconstruction algorithm needs to be designed for this. The present invention proposes a spectral dimension deep learning model guided by the LCMD transmittance based on the MST++ model, which solves the problem that the wide spectral band modulated data obtained by CLSI cannot directly complete spectral reconstruction relying on the existing model. Based on the data collected by CLSI and the trained model, the low-cost and high-spectral-resolution reconstruction of the scene can be realized.

[0061] The additional aspects and advantages of the present invention will become obvious in the following description part, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0063] Figure 1 is a schematic structural diagram of a liquid crystal computational modulator according to an embodiment of the present invention;

[0064] Figure 2 is a schematic diagram of the transmittance simulation curve of a liquid crystal computational modulator according to an embodiment of the present invention;

[0065] Figure 3 is a schematic structural diagram of a computational liquid crystal spectral imager based on wide spectral modulation according to an embodiment of the present invention;

[0066] Figure 4 is a schematic structural diagram of the spectral dimension deep learning model LTST guided by the LCMD transmittance according to an embodiment of the present invention;

[0067] Figure 5 is a schematic diagram of the spectral response curves of the merged bands formed by grouping the input information by the E-SSB according to bands and after grouping according to an embodiment of the present invention;

[0068] Figure 6 It is a schematic structural diagram of a spectral - dimensional multi - head self - attention mechanism LTS - MSA guided by transmittance in an embodiment of the present invention;

[0069] Figure 7 It is a schematic structural diagram of a spectral - type attention unit LTSAB guided by transmittance in an embodiment of the present invention;

[0070] Figure 8 It is a schematic structural diagram of a feed - forward neural network in an embodiment of the present invention;

[0071] Figure 9 It is a schematic structural diagram of a single - layer spectral - dimensional neural network SLTST in an embodiment of the present invention;

[0072] Figure 10 It is a standard hyperspectral image captured by a calibrated spectral imager in a specific embodiment of the present invention;

[0073] Figure 11 It is a reconstruction using 6 algorithms in a specific embodiment of the present invention Figure 10 Schematic diagram of the comparison result of RGB sub - images in the reconstruction comparison region;

[0074] Figure 12 It is a reconstruction using 6 algorithms in a specific embodiment of the present invention Figure 10 Schematic diagram of the comparison of DN value curves in the DN value curve comparison region of ; Detailed implementation manners

[0075] In order to more clearly understand the above - mentioned objects, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0076] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0077] Next, refer to Figures 1 to 12 to describe a liquid - crystal computing modulator, a computational liquid - crystal spectral imager and its data reconstruction method provided according to some embodiments of the present invention.

[0078] Some embodiments of the present application provide a liquid - crystal computing modulator.

[0079] As Figure 1As shown in the figure, the first embodiment of the present invention proposes a liquid crystal computing modulator, which includes a first polarizer, a first-stage light-transmitting unit, a second polarizer, a second-stage light-transmitting unit, and a third polarizer. The above-mentioned first polarizer, first-stage light-transmitting unit, second polarizer, second-stage light-transmitting unit, and third polarizer are arranged in sequence along the light propagation direction.

[0080] The first polarizer is arranged on one side of the first-stage light-transmitting unit close to the light incident direction, and the polarization direction of the first polarizer can be any direction.

[0081] The first-stage light-transmitting unit includes a plurality of first liquid crystal sheets with the same thickness and a plurality of first quartz sheets with the same thickness. The plurality of first liquid crystal sheets and the plurality of first quartz sheets are arranged alternately in sequence, and the number of the first liquid crystal sheets and the first quartz sheets is equal; Figure 1 In the shown embodiment, the number of both the first liquid crystal sheets and the first quartz sheets is 3.

[0082] The second polarizer is arranged on one side of the first-stage light-transmitting unit close to the light exit direction, and the polarization direction of the second polarizer is perpendicular to the polarization direction of the first polarizer; Figure 1 In the shown embodiment, the direction of the lines on the polarizer represents its polarization direction.

[0083] The second-stage light-transmitting unit includes a plurality of second liquid crystal sheets with the same thickness and a plurality of second quartz sheets with the same thickness. The plurality of second liquid crystal sheets and the plurality of second quartz sheets are arranged alternately in sequence, and the number of the second liquid crystal sheets and the second quartz sheets is equal; the thickness of the second liquid crystal sheets is greater than the thickness of the first liquid crystal sheets, and the thickness of the second quartz sheets is greater than the thickness of the second liquid crystal sheets; the second polarizer is located between the first-stage light-transmitting unit and the second-stage light-transmitting unit; Figure 1 In the shown embodiment, the number of both the second liquid crystal sheets and the second quartz sheets is 3. In a specific embodiment, the thickness of the second liquid crystal sheets is twice the thickness of the first liquid crystal sheets; the thickness of the second quartz sheets is twice the thickness of the first quartz sheets.

[0084] The second-stage light-transmitting unit is arranged between the second polarizer and the third polarizer; the polarization direction of the third polarizer is the same as the polarization direction of the first polarizer.

[0085] Specifically, in the above liquid crystal computing modulator, the first polarizer is arranged on one side of the first-stage light-transmitting unit close to the light incident direction, and is used to convert the incident light into light with a specific polarization direction. The second polarizer is arranged between the first-stage light-transmitting unit and the second-stage light-transmitting unit, and its polarization direction is perpendicular to the polarization direction of the first polarizer, and is used to further modulate the polarization state of the light. The third polarizer is arranged on one side of the second-stage light-transmitting unit close to the light exit direction, and its polarization direction is the same as the polarization direction of the first polarizer, and is used to restore the polarization state of the light.

[0086] The liquid crystal sheet controls the alignment of its molecules through an electric field, thereby modulating the phase and amplitude of light; the quartz sheet provides stable optical properties. In the second-level light-transmitting unit, the thickness of the second liquid crystal sheet and the second quartz sheet is greater than that of the first liquid crystal sheet and the first quartz sheet, thereby providing a stronger light modulation ability.

[0087] The light modulation process is as follows: In the first-level modulation, the incident light becomes light with a specific polarization direction after passing through the first polarizer. This light enters the first-level light-transmitting unit, and through the alternating modulation of the first liquid crystal sheet and the first quartz sheet, the phase and amplitude of the light change. The light after the first-level modulation enters the second polarizer, whose polarization direction is perpendicular to that of the first polarizer, further modulating the polarization state of the light. Subsequently, the light enters the second-level light-transmitting unit, and through the alternating modulation of the second liquid crystal sheet and the second quartz sheet, the phase and amplitude of the light change again. The light after the second-level modulation enters the third polarizer, whose polarization direction is the same as that of the first polarizer, restoring the polarization state of the light, and finally outputting the modulated light.

[0088] Through the multi-level modulation of the first-level and second-level light-transmitting units, the phase and amplitude of light are finely adjusted, reducing light loss and increasing the transmittance. The combined design of the first-level and second-level light-transmitting units enables the modulator to cover a wider spectral range. Liquid crystal sheets and quartz sheets with different thicknesses can effectively modulate light of different wavelengths, thereby expanding the passband range. The polarization directions of the first polarizer and the third polarizer are the same, and the polarization direction of the second polarizer is perpendicular to that of the first polarizer. This design enables the modulator to effectively modulate light of different wavelengths, further expanding the passband range. Through the multi-level modulation of the first-level and second-level light-transmitting units, the phase and amplitude of light are finely adjusted, and more spectral information can be passed through within the same exposure time. In addition, liquid crystal sheets and quartz sheets with different thicknesses can effectively modulate light of different wavelengths, thereby obtaining more spectral information within the same exposure time.

[0089] The liquid crystal computing modulator of this embodiment realizes good transmittance, a wider passband range, and passing through more spectral information within the same exposure time through the combined design of multi-level light-transmitting units and polarizers. This design has important application value in fields such as spectral analysis and imaging.

[0090] In some embodiments, the optical axis directions of all liquid crystal sheets in the liquid crystal computing modulator alternately maintain an angle of α and -α with the first direction in sequence; wherein, the propagation direction of light is defined as the y-axis direction, and the first direction is the x-axis direction perpendicular to the propagation direction of light.

[0091] Figure 2The transmittance simulation curves of the above liquid crystal computational modulator at different central wavelengths with an interval of 20 nm in the 450 - 720 nm band are shown. It can be seen that the peak transmittance of the liquid crystal computational modulator developed by the present invention can reach 30% - 35%.

[0092] In the above liquid crystal computational modulator, the transmittance of the first - stage light - transmitting unit is:

[0093]

[0094] The transmittance of the second - stage light - transmitting unit is:

[0095]

[0096] The total transmittance of the liquid crystal computational modulator is:

[0097] T LCMD (λ) = T LCMD_1 T LCMD_2

[0098] where, T LCMD_1 represents the transmittance of the first - stage light - transmitting unit; λ represents the wavelength; δ1 and α1 are both coefficients related to the driving voltage; Δn = n e −n0 represents the birefringence of the crystal in the liquid crystal sheet, n e represents the extraordinary - light refractive index, n0 represents the ordinary - light refractive index; d Q1 represents the thickness of the first quartz sheet; d LC1 represents the thickness of the first liquid crystal sheet; T LCMD_2 represents the transmittance of the second - stage light - transmitting unit; T LCMD represents the total transmittance of the liquid crystal computational modulator.

[0099] As Figure 3 shown, a computational liquid - crystal spectral imager based on broadband modulation provided by the second embodiment of the present invention includes: an imaging lens, a fuselage, and a host computer.

[0100] The imaging lens is used to collect the light reflected or emitted by the target object.

[0101] The fuselage mainly includes a control circuit, the LCMD described in the above embodiments, a detector, and a temperature control module; among them, the control circuit is used to control the operation of each component in the imager, including functions such as power management, signal processing, and data transmission; it ensures that each part of the imager can work in coordination to achieve stable imaging and spectral analysis. The LCMD realizes spectral addressing or multispectral scanning by adjusting the driving voltage, and completes continuous spectral analysis within the preset spectral band; by changing the position of the spectral passband, it can selectively transmit light of different wavelengths. The detector receives the light rays after passing through the imaging lens and spectral modulation, and converts them into electrical signals; the converted electrical signals are processed and analyzed to generate a spectral image of the target scene. The temperature control module is used to maintain the temperature stability inside the imager, preventing the influence of temperature changes on the imaging quality and the accuracy of spectral analysis; the temperature control module can ensure that the detector and other sensitive components work at an appropriate temperature, thereby improving the performance and reliability of the imager.

[0102] The host computer is used to send control commands to the control circuit and receive detector data, etc.

[0103] The third embodiment of the present invention provides a data reconstruction method for a computational liquid crystal spectral imager based on broadband modulation. Based on a computational liquid crystal spectral imager based on broadband modulation as described in the above embodiments, the data reconstruction method includes the following steps S1 - S5.

[0104] S1. Establish a guiding database for the transmittance of the liquid crystal computational modulator; it should be noted that the transmittance data recorded in the guiding database is the transmittance of the LCMD measured and collected in the laboratory, denoted as T L (m,n). Where m represents the spectral sampling accuracy and n represents the spectral modulation times. That is to say, S1 can be regarded as the calibration process of the transmittance of the liquid crystal computational modulator.

[0105] S2. Use the computational liquid crystal spectral imager and the calibrated spectral imager to collect spectral data of several scenes and several ground objects. Among them, the spectral data collected by the computational liquid crystal spectral imager corresponds and matches the spectral data collected by the calibrated spectral imager, and an acquisition database is formed according to the collected spectral data.

[0106] Specifically, the spectral range of the calibrated spectral imager is larger than that of the computational liquid crystal spectral imager, and the viewing angle is also larger than that of the computational liquid crystal spectral imager. The specific acquisition process is as follows:

[0107] Use the two instruments to collect the original data of the same scene respectively;

[0108] Perform alignment operation and cropping on the data collected by the two instruments in the spectral dimension to ensure that the captured data is accurately matched in the spectral range;

[0109] Align and crop the data collected by the two instruments in terms of spatial dimensions;

[0110] Through feature matching, match the data captured by the computational liquid crystal spectral imager to the data captured by the calibrated spectral imager, i.e., they have the same spatial resolution; form an acquisition database for subsequent deep learning training;

[0111] Among them, when the computational liquid crystal spectral imager collects data, use the same number of sampling band numbers \(n\) as the maximum spectral modulation times during the calibration of the guiding database of the LCMD. The number of spectral bands collected by the calibrated spectral imager is \(N\). In the matched data, the data collected by the computational liquid crystal spectral imager is represented as The data collected by the calibrated spectral imager is represented as where \(x\) and \(y\) represent the horizontal and vertical spatial coordinates respectively. Divide the data in the above acquisition database into a training set, a test set, and a validation dataset to carry out subsequent network training.

[0112] S3. Establish a spectral - dimension deep learning model guided by the transmittance of the liquid crystal computational modulator. The deep learning model uses the MST++ network as the basic framework, adds a continuous spectral band selection method at the input end of the network to pre - process the input data; and replaces the spectral multi - head self - attention mechanism in the MST++ network with a transmittance - guided spectral - dimension multi - head self - attention mechanism. In the transmittance - guided spectral - dimension multi - head self - attention mechanism, when calculating the attention value, re - weight the feature vectors of the image patches based on the transmittance data in the guiding database.

[0113] The network architecture of the above spectral - dimension deep learning model is as Figure 4 shown. The continuous spectral band selection method is denoted as E - SSB; Conv3×3 represents a 3×3 convolutional layer; SLTST represents a single - layer spectral - dimension neural network, and \(N_s\) represents the stacking number of single - layer spectral - dimension neural networks.

[0114] In a specific embodiment, adding a continuous spectral band selection method at the input end of the network to pre - process the input data includes:

[0115] Divide the spectral data \(H\) of the computational liquid crystal spectral imager input into the network into \(C\) parts. Among them, the first parts contain continuous spectral bands, and the last parts contain continuous bands, where \(n\) represents the number of sampling bands;

[0116] Sum the spectral data within each part and perform normalization processing to obtain the final input data \(H_{out}\). Figure 5It shows a schematic diagram of the spectral response curve of the E-SSB grouping the input information by wavelength band and the combined wavelength bands formed after grouping.

[0117] This disclosure is based on the prior information T of the spectral dimension of known instruments L It proposes a transmittance-guided spectral multi-head self-attention mechanism (abbreviated as LTS-MSA) to replace the original spectral multi-head self-attention mechanism (Spectral-wise Multi-head Self-Attention, abbreviated as S-MSA) in the MST++ network. Except for the above differences, the original content in the MST++ network can be used. As Figure 6 shown, in the transmittance-guided spectral multi-head self-attention mechanism:

[0118] According to the transmittance of the liquid crystal computational modulator in the guidance database, a transmittance guidance matrix is established:

[0119] LT = (W1T L ) ⊙ (1 + δ(f dw (W2W1T L )))

[0120] where LT represents the transmittance guidance matrix; both W1 and W2 represent 1×1 convolutional layers of learnable parameters; T L represents the transmittance data of the liquid crystal computational modulator in the guidance database; f dw () represents a 7×7 convolutional layer in the depth direction; δ(·) represents the sigmoid activation function;

[0121] On the spectral dimension, the transmittance guidance matrix is divided into N heads LT = [LT1, LT2,... LT N ;

[0122] For any multi-head self-attention b j , by using LT j to re-weight the feature vectors of the image patches to guide the spectral attention change:

[0123]

[0124] where b j represents the attention value of the j-th image patch; σ j represents a learnable parameter; represents the attention score between the i-th image patch and the j-th image patch; v j represents the feature vector of the j-th image patch.

[0125] It can be understood that in the present disclosure, a transmittance-guided spectral attention unit LTSAB is created based on the LTS-MSA structure, and the LTSAB can be directly used to replace the original spectral attention unit SAB in the MST++ network, while the remaining structures remain the same. The structure of the transmittance-guided spectral attention unit LTSAB proposed in the present disclosure is as Figure 7 shown, which includes two normalization layers (shown as Layer Norm in the figure), an LTS-MSA structure, and a feed-forward neural network FFN; the structure of the feed-forward neural network is as Figure 8 shown, which includes two 1×1 convolutional layers (shown as conv1×1 in the figure), two layers of GELU activation functions, and a depthwise separable convolution (shown as DWconv3×3 in the figure).

[0126] A single-layer spectral dimension neural network SLTST is constructed using the transmittance-guided spectral attention unit LTSAB. In some embodiments, SLTST adopts a U-shaped structure to extract multi-resolution spectral context information crucial for hyperspectral image reconstruction; the structure of SLTST is as Figure 9 shown. The working process of this network (SLTST) is as follows: The input data first undergoes feature extraction through an embedding layer to generate an initial feature map H0. In the encoder, the feature map sequentially passes through multiple LTSAB units and downsampling operations to gradually extract multi-scale features, further compressing the feature dimension to generate feature maps H1 and H2. The generated feature maps enter the bottleneck layer. In the decoder, the feature maps gradually recover the resolution through upsampling operations and LTSAB modules, and at the same time are fused with the feature maps in the encoder through skip connections (C) to generate feature maps H0′, H1′, and H2′. Finally, the feature maps generate the final output result through a mapping layer.

[0127] SLTSTs are stacked to form a spectral dimension deep learning model LTST based on LCMD transmittance guidance. Through this multi-stage learning strategy, the reconstruction quality of H can be gradually improved from coarse to fine, continuously approaching G, thereby significantly improving the performance.

[0128] S4. Use the spectral data of the computational liquid crystal spectral imager in the acquisition database as the network input of the deep learning model, and use the spectral data of the calibrated spectral imager in the acquisition database as the network output of the deep learning model to train the deep learning model, so that the reconstruction result of the spectral data of the computational liquid crystal spectral imager gradually approaches the spectral data of the calibrated spectral imager, and obtain the trained deep learning model.

[0129] S5. Input the spectral data recollected by the computational liquid crystal spectral imager into the trained deep learning model, and the hyperspectral information of the corresponding unknown scene or ground object can be reconstructed.

[0130] In a specific embodiment, taking a certain hyperspectral imager as an example, it uses LCMD as the core component. During the calibration process, it uses HySpex Baldur V-1024 as the standard calibration instrument. When calibrating the guidance database, take n = 50 and m = 55. The spectral range of this spectral imager is 450 - 720 nm, and the number of spectral bands within this range is n = 50. The spectral range of HySpex Baldur V-1024 is 400 - 1000 nm, and the number of spectral bands collected within this range is 108. After spectral matching, the number of spectral bands within the range of 450 - 720 nm is N = 55. Based on the acquisition database containing 77 groups of data made by acquisition, start the training of the reconstruction model.

[0131] During network training, the input is divided into C = 5 parts, and the number of spectral bands merged for each part in the E-SSB stage is 11 consecutive spectral bands. To show the effect of the trained network for reconstructing hyperspectral ground object images, as Figure 10 shown, it shows the standard hyperspectral image (DN) taken by HySpex Baldur V-1024. The blue dashed box represents the sub-image used for reconstruction comparison, and the yellow solid box represents the selected DN value curve area. In Figure 11 it shows the reconstruction results of 6 different networks participating in the comparison, where GT represents the ground truth, EDSR, HDNet, Restormer, MIRNet, MST++ are typical general algorithms participating in the comparison (to reduce variables, E-SSB is used as the preprocessing method in the above methods). The red image takes the typical band as R = 685.30 nm, the yellow image takes the typical band as Y = 582.56 nm, the green image takes the typical band as G = 539.30 nm, and the blue image takes the typical band as B = 479.82 nm. Here we take the spectral angle distance (SAM) as the evaluation index and compare the spectral angle distances of the same reconstructed area. As Figure 12 shown, the LTST algorithm proposed by the present invention has achieved the best effect. Through Figures 10 - 12 experiments, it can be proved that the computational liquid crystal spectral imager based on broadband modulation and its data reconstruction method proposed in this disclosure can be effectively implemented in general scenes of hyperspectral imaging.

[0132] In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0133] Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A liquid crystal computing modulator, characterized in that: include: A first-stage light-transmitting unit includes a plurality of first liquid crystal sheets having the same thickness and a plurality of first quartz sheets having the same thickness, wherein the plurality of first liquid crystal sheets and the plurality of first quartz sheets are arranged alternately in sequence, and the number of the first liquid crystal sheets and the number of the first quartz sheets are equal; A first polarizer is arranged on a side of the first light-transmitting unit close to the incident direction of light; A second polarizer is arranged on a side of the first light-transmitting unit close to the light emission direction, and the polarization direction of the second polarizer is perpendicular to the polarization direction of the first polarizer; The second light-transmitting unit comprises a plurality of second liquid crystal sheets having the same thickness and a plurality of second quartz sheets having the same thickness, the plurality of second liquid crystal sheets and the plurality of second quartz sheets being arranged alternately in sequence, and the number of the second liquid crystal sheets and the second quartz sheets being equal; the thickness of the second liquid crystal sheet is greater than the thickness of the first liquid crystal sheet, and the thickness of the second quartz sheet is greater than the thickness of the second liquid crystal sheet; the second polarizing sheet is located between the first light-transmitting unit and the second light-transmitting unit; a third polarizer, wherein the second light-transmitting unit is disposed between the second polarizer and the third polarizer; and the polarization direction of the third polarizer is the same as that of the first polarizer; The first polarizer, the first light-transmitting unit, the second polarizer, the second light-transmitting unit and the third polarizer are sequentially arranged in the propagation direction of light.

2. The liquid crystal computing modulator according to claim 1, characterized in that: The optical axis directions of all the liquid crystal sheets in the liquid crystal computing modulator alternately maintain angles of α and -α with the first direction in turn; The propagation direction of light is defined as the y-axis direction, and the first direction is the x-axis direction perpendicular to the propagation direction of light.

3. The liquid crystal computing modulator according to claim 2, characterized in that: The number of the first liquid crystal panels and the number of the second liquid crystal panels are both 3; The thickness of the second liquid crystal sheet is twice the thickness of the first liquid crystal sheet; The thickness of the second quartz plate is twice the thickness of the first quartz plate.

4. The liquid crystal computing modulator according to claim 3, characterized in that: The transmittance of the first-stage light-transmitting unit is: The transmittance of the second-level light-transmitting unit is: The total transmittance of the liquid crystal modulator is: T LCMD (λ)=T LCMD_1 T LCMD_2 Among them, T LCMD_1 represents the transmittance of the first-level light-transmitting unit; λ represents the wavelength; δ1 and α1 are coefficients related to the driving voltage; Δn = n e -n0 represents the birefringence of the crystal in the liquid crystal cell, n e represents the extraordinary refractive index, n0 represents the ordinary refractive index; d Q1 represents the thickness of the first quartz plate; d LC1 represents the thickness of the first liquid crystal panel; T LCMD_2 Indicates the transmittance of the second-level light-transmitting unit; T LCMD Represents the total transmittance of the liquid crystal modulator.

5. A computational liquid crystal spectral imager based on wide spectrum modulation, characterized in that: include: Imaging lens, which collects light reflected or emitted by the target object; The liquid crystal computational modulator according to any one of claims 1 to 4, wherein spectral addressing or multi-spectral scanning is achieved by adjusting the driving voltage to complete continuous spectral analysis within a preset spectral band; wherein the liquid crystal computational modulator achieves selective transmission of light of different wavelengths by changing the position of the spectral passband; The detector receives the light after passing through the imaging lens and spectrally modulated and converts it into electrical signals; the converted electrical signals are processed and analyzed to generate a spectral image of the target scene.

6. A data reconstruction method for a computational liquid crystal spectral imager based on wide spectrum modulation, characterized in that: Data reconstruction is performed based on a computational liquid crystal spectral imager based on wide spectrum modulation as claimed in claim 5, wherein the data reconstruction method comprises: Establish a guide database for calculating the transmittance of liquid crystal modulators; Using the computational liquid crystal spectral imager and the calibrated spectral imager to collect spectral data of several scenes and several ground objects, wherein the spectral data collected by the computational liquid crystal spectral imager corresponds to and matches the spectral data collected by the calibrated spectral imager, and forming a collection database based on the collected spectral data; A spectral dimension deep learning model guided by the transmittance of a liquid crystal computational modulator is established. The deep learning model uses the MST++ network as the basic framework, and adds a continuous spectral band selection method at the input end of the network to preprocess the input data; and replaces the spectral multi-head self-attention mechanism in the MST++ network with a spectral dimension multi-head self-attention mechanism guided by transmittance. In the spectral dimension multi-head self-attention mechanism guided by transmittance, the feature vector of the image block is re-weighted based on the transmittance data in the guidance database when calculating the attention value; Using the spectrum data of the computational liquid crystal spectroscopic imager in the acquisition database as the network input of the deep learning model, using the spectrum data of the calibrated spectrum imager in the acquisition database as the network output of the deep learning model, training the deep learning model so that the reconstruction result of the spectrum data of the computational liquid crystal spectroscopic imager gradually approaches the spectrum data of the calibrated spectrum imager, and obtaining a trained deep learning model; The spectral data re-collected by the computational liquid crystal spectral imager is input into the trained deep learning model to reconstruct the spectral information of the corresponding scene or corresponding object.

7. The data reconstruction method of the computational liquid crystal spectroscopic imager based on wide spectrum modulation according to claim 6, characterized in that: The method of adding a continuous spectral band selection method to the input end of the network to pre-process the input data includes: The spectrum data H of the computational liquid crystal spectrometer input into the network is divided into C parts, among which the first The parts include continuous spectral bands, Includes continuous bands, n represents the number of sampling bands; The spectral data in each part are summed and normalized to obtain the final input data.

8. The data reconstruction method of the computational liquid crystal spectroscopic imager based on wide spectrum modulation according to claim 6, characterized in that: The method replaces the spectral multi-head self-attention mechanism in the MST++ network with a spectral multi-head self-attention mechanism guided by transmittance, including: Calculate the modulator transmittance based on the liquid crystal in the guidance database and establish the transmittance guidance matrix: LT=(W1T L )☉(1+δ(f dw (W2W1T L ))) Where LT represents the transmittance guidance matrix; W1 and W2 both represent 1×1 convolutional layers with learnable parameters; T L represents the transmittance data of the liquid crystal modulator in the boot database; f dw (·) represents a 7×7 convolutional layer in the depth direction; δ(·) represents the sigmoid activation function; In the spectral dimension, the transmittance guidance matrix is ​​divided into N heads LT = [LT1, LT2, ... LT N ]; For any multi-head self-attention b j , by using LT j Reweight the feature vector of the image patch to guide the spectral attention change: Among them, b j represents the attention value of the jth image block; σ j represents a learnable parameter; represents the attention score between the i-th image block and the j-th image block; v j Represents the feature vector of the j-th image block.

9. The data reconstruction method of the computational liquid crystal spectroscopic imager based on wide spectrum modulation according to claim 8, characterized in that: A single-layer spectral dimension neural network is constructed according to the transmittance-guided spectral dimension multi-head self-attention mechanism, and the single-layer spectral dimension neural network adopts a U-shaped structure to extract multi-resolution spectral context information; By stacking the single-layer spectral-dimensional neural network, a spectral-dimensional deep learning model guided by the transmittance of the liquid crystal calculation modulator is formed.

10. The data reconstruction method of the computational liquid crystal spectroscopic imager based on wide spectrum modulation according to claim 6, characterized in that: When the computational liquid crystal spectral imager is used to collect spectral data, the number of sampling bands is equal to the number of modulations of the liquid crystal computational modulator in the guide database.

Images