Mask optimization-based low-illumination spectral imaging system and method

Abstract

The invention relates to the field of computational photography, in particular to a low-illumination spectral imaging system and method based on mask optimization, an imaging module in the system comprises a mask unit and an acquisition unit, and the acquisition unit comprises a dispersion element and a single-photon image acquisition device. The imaging module is used for imaging the original spectrum cube under low illumination to obtain a measured value; the reconstruction network module comprises a trained spectrum reconstruction network model; the spectrum reconstruction network model comprises a coding mask layer, a device simulation layer and a spectrum recovery layer, and the device simulation layer simulates an acquisition unit; a coding mask layer of the trained spectrum reconstruction network model controls a mask unit to generate an optimized mask, and a spectrum recovery layer of the trained spectrum reconstruction network model reconstructs a measurement value output by an imaging module to obtain a corresponding reconstructed spectrum cube. According to the system provided by the invention, the imaging accuracy of the snapshot spectral imaging system under the low-illumination condition can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computational imaging, and in particular, to a low-light spectral imaging system and method based on mask optimization. Background Art

[0002] Spectral imaging technology can simultaneously obtain the spatial information and spectral information of an object, and has wide applications in fields such as medical imaging, environmental monitoring, and agricultural detection. However, under low-light conditions, the performance of traditional spectral imaging systems is significantly limited. With the continuous growth of the demand for high-performance devices in various industries, low-light spectral imaging technology shows important application potential in low-light environments such as underwater exploration, deep space exploration, and night imaging. Traditional spectral imaging systems rely on long exposure times or high-sensitivity detectors to increase the signal intensity, and face problems such as low signal-to-noise ratio, poor reconstruction quality, and insufficient spatial resolution under low-light conditions, making it difficult to provide ideal image quality.

[0003] Existing spectral imaging technologies mainly include two types: scanning and snapshot. Scanning spectral imaging systems gradually acquire information in the spectral or spatial dimension during a mechanical scanning process by introducing optical devices (such as tunable liquid crystal filters, gratings, etc.), and then combine them to obtain a spectral cube. Although it can provide high spectral resolution, scanning devices are bulky, have a slow imaging speed, and are difficult to achieve dynamic imaging. Snapshot spectral imaging systems obtain spatial and spectral information simultaneously through a single exposure, significantly improving the imaging speed and dynamic performance. With the development of compressive sensing technology, snapshot systems have become a research hotspot, especially having obvious advantages in applications that require fast imaging and high temporal resolution. Although snapshot spectral imaging systems show significant advantages in the field of spectral imaging, they still face many challenges in practical applications. First, existing snapshot spectral imaging systems perform poorly under low-light conditions and are difficult to capture sufficient spectral information to ensure reconstruction quality, which limits their application in low-light environments. Second, most current reconstruction algorithms for snapshot spectral imaging systems do not fully consider the imaging physical process, resulting in a disconnection between the physical model and the reconstruction algorithm, thus affecting the reconstruction accuracy of the spectral cube. In addition, reconstructing a spectral cube from two-dimensional compressed data is an underdetermined problem. The reconstruction quality of snapshot spectral imaging systems largely depends on their compressive sampling design and the accompanying image reconstruction algorithm. Existing snapshot spectral imaging systems usually process the imaging encoding and image decoding processes separately in the design, lacking a joint optimization design for the encoding and decoding processes, and failing to maximize the system performance and imaging quality. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to some extent.

[0005] To this end, the first object of the present invention is to propose a low-light spectral imaging system based on mask optimization to improve the imaging accuracy of a snapshot spectral imaging system under low-light conditions.

[0006] The second object of the present invention is to propose a low-light spectral imaging method based on mask optimization.

[0007] To achieve the above object, the first aspect of the present invention proposes a low-light spectral imaging system based on mask optimization, including: The imaging module includes a mask unit and an acquisition unit. The acquisition unit includes a dispersion element and a single-photon image acquisition device. The imaging module is used to image the original spectral cube under low light to obtain a measurement value; The reconstruction network module includes a trained spectral reconstruction network model; the spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer connected in sequence. The device simulation layer is used to simulate the acquisition unit. The input of the spectral recovery layer is the measurement value, and the output is the reconstructed spectral cube; wherein the encoding mask layer of the trained spectral reconstruction network model is used to control the mask unit to generate an optimized mask, and the spectral recovery layer of the trained spectral reconstruction network model is used to reconstruct the measurement value output by the imaging module to obtain the corresponding reconstructed spectral cube.

[0008] In the low-light spectral imaging system based on mask optimization provided by the first aspect of the present invention, the mask unit includes a digital micromirror device.

[0009] In the low-light spectral imaging system based on mask optimization provided by the first aspect of the present invention, the spectral recovery layer includes a spectral perception layer and a spatial perception layer.

[0010] In the low-light spectral imaging system based on mask optimization provided by the first aspect of the present invention, both the spatial perception layer and the spectral perception layer adopt a neural network structure.

[0011] In the low-light spectral imaging system based on mask optimization provided by the first aspect of the present invention, the spectral recovery layer is a multi-source noise-constrained deep learning model based on the full-link physical process.

[0012] To achieve the above object, the second aspect of the present invention proposes a low-light spectral imaging method based on mask optimization, which is applied to the low-light spectral imaging system based on mask optimization provided by the first aspect of the present invention. The method includes: Obtain a trained spectral reconstruction network model; the spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer connected in sequence. The input of the spectral recovery layer is the measurement value, and the output is the reconstructed spectral cube; Using the encoding mask layer of the trained spectral reconstruction network model to control the mask unit of the imaging module to generate an optimized mask; Inputting the original spectral cube under low illuminance into the imaging module for imaging to obtain a measurement value; Inputting the measurement value into the spectral recovery layer of the trained spectral reconstruction network model for reconstruction to obtain the corresponding reconstructed spectral cube.

[0013] In the low-illuminance spectral imaging method based on mask optimization provided in the second aspect of the present invention, using the encoding mask layer of the trained spectral reconstruction network model to control the mask unit of the imaging module to generate an optimized mask includes: using the encoding mask layer of the trained spectral reconstruction network model to control the mask unit of the imaging module to generate a single-frame optimized mask or multiple-frame masks.

[0014] In the low-illuminance spectral imaging method based on mask optimization provided in the second aspect of the present invention, the training process of the spectral reconstruction network model includes: obtaining a training data set composed of multiple spectral cubes based on an open-source data set; using the training data set to train the spectral reconstruction network model to obtain a trained spectral reconstruction network model.

[0015] In the low-illuminance spectral imaging method based on mask optimization provided in the second aspect of the present invention, when training the spectral reconstruction network model, if both the spatial perception layer loss function and the spectral perception layer loss function are less than a threshold, the training ends.

[0016] In the low-illuminance spectral imaging method based on mask optimization provided in the second aspect of the present invention, the spectral recovery layer includes a spectral perception layer and a spatial perception layer, and both the spatial perception layer and the spectral perception layer adopt a neural network structure.

[0017] The low-light spectral imaging system and method based on mask optimization provided by the present invention, wherein the system includes an imaging module and a reconstruction network module. The imaging module includes a mask unit and an acquisition unit. The acquisition unit includes a dispersion element and a single-photon image acquisition device. The imaging module is used to image the original spectral cube under low light to obtain a measurement value. The reconstruction network module includes a trained spectral reconstruction network model. The spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer connected in sequence. The device simulation layer is used to simulate the acquisition unit. The input of the spectral recovery layer is the measurement value, and the output is the reconstructed spectral cube. The encoding mask layer of the trained spectral reconstruction network model is used to control the mask unit to generate an optimized mask, and the spectral recovery layer of the trained spectral reconstruction network model is used to reconstruct the measurement value output by the imaging module to obtain the corresponding reconstructed spectral cube. In this case, the reconstruction network module includes a trained spectral reconstruction network model. The spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer connected in sequence. The device simulation layer is used to simulate the acquisition unit. The input of the spectral recovery layer is the measurement value, and the output is the reconstructed spectral cube. The encoding mask layer of the trained spectral reconstruction network model is used to control the mask unit to generate an optimized mask, and the spectral recovery layer of the trained spectral reconstruction network model is used to reconstruct the measurement value output by the imaging module to obtain the corresponding reconstructed spectral cube. Therefore, the spectral reconstruction network model fully considers the imaging physical process, realizes the joint optimization of the imaging process and the spectral reconstruction network model, and thus improves the imaging accuracy of the snapshot spectral imaging system under low-light conditions.

[0018] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, wherein: Figure 1 is a block diagram of a low-light spectral imaging system based on mask optimization provided by an embodiment of the present invention; Figure 2 is a diagram of the low-light optimized single-frame mask snapshot spectral imaging process provided by an embodiment of the present invention; Figure 3 is a diagram of the low-light optimized multi-frame mask spectral imaging process provided by an embodiment of the present invention; Figure 4 is a diagram of a multi-source noise-constrained deep learning model based on the full-link physical process provided by an embodiment of the present invention; Figure 5 is a flowchart of a low-light spectral imaging method based on mask optimization provided by an embodiment of the present invention. Detailed implementation manners

[0020] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0021] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0022] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined. It should also be understood that the term "and / or" used in the present invention refers to and includes any or all possible combinations of one or more of the associated listed items.

[0023] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present invention and should not be construed as a limitation to the present invention.

[0024] The present invention provides a low-light spectral imaging system and method based on mask optimization to improve the imaging accuracy of a snapshot spectral imaging system under low-light conditions.

[0025] In the embodiments of the present invention, Figure 1 is a block diagram of a low-light spectral imaging system based on mask optimization provided by an embodiment of the present invention. As Figure 1As shown in the figure, the low-light spectral imaging system based on mask optimization includes an imaging module and a reconstruction network module.

[0026] In an embodiment of the present invention, the imaging module is used to image the original spectral cube under low light to obtain a measurement value. The imaging module includes a mask unit and an acquisition unit.

[0027] Specifically, the mask unit is used to encode spatial information into the original spectral cube under low light through an optimized mask to obtain a first intermediate spectral cube. The optimized mask in the mask unit is generated under the control of the trained encoding mask layer (described in detail later) in the reconstruction network module. Among them, the number of optimized masks can be a single frame or multiple frames (see Figure 1 ).

[0028] The acquisition unit includes a dispersive element and a single-photon image acquisition device. Among them, the dispersive element is used to encode spectral information into the first intermediate spectral cube to obtain a second intermediate spectral cube. The single-photon image acquisition device is used to image the second intermediate spectral cube to obtain a measurement value.

[0029] In some embodiments, taking the number of optimized masks as a single frame and the single-photon image acquisition device as a single-photon camera as an example, Figure 2 is a process diagram of low-light optimized single-frame mask snapshot spectral imaging provided by an embodiment of the present invention. As Figure 2 shown, the imaging process of the imaging module includes: Under low light, the light source passes through the sample to obtain the original spectral cube; the original spectral cube passes through a frame of optimized mask to obtain a first intermediate spectral cube, where the first intermediate spectral cube carries spatial information, and this frame of optimized mask is generated under the control of the trained encoding mask layer in the reconstruction network module; the first intermediate spectral cube passes through the dispersive element to obtain a second intermediate spectral cube, where the second intermediate spectral cube carries spectral information; the second intermediate spectral cube enters the single-photon camera for imaging to obtain a measurement value. The measurement value is input into the subsequent algorithm reconstruction part (described in detail later) to obtain the required reconstructed spectral cube.

[0030] In some embodiments, Figure 2 in the imaging module shown, the mask unit for obtaining a single-frame optimized mask can be implemented by an encoded aperture. At this time, Figure 2The imaging module shown uses a Coded Aperture Snapshot Spectral Imager (CASSI). CASSI is an efficient spectral imaging technology based on the theory of compressive sensing. Different from the traditional way of scanning hyperspectral information point by point or line by line, CASSI can capture the spectral and spatial information of the scene through a single snapshot, thus significantly improving the imaging efficiency and avoiding the problem of motion artifacts caused by long exposure. The basic principle of CASSI is to modulate the hyperspectral information (i.e., spectral cube) using a coded aperture and a dispersive element, and compress the modulated signal into two-dimensional measurement data. Then, through an optimization algorithm (i.e., the spectral recovery layer of the reconstruction algorithm), the spectral cube containing rich spectral information is reconstructed from the two-dimensional measurement data.

[0031] In addition, since Figure 2In the single-photon image acquisition device in the imaging module shown, a single-photon camera is used. Therefore, at this time, CASSI is a snapshot single-photon spectral imaging system based on coded aperture. Single-photon imaging technology is an active optical imaging technology. Along with the rapid development of single-photon detection technology, single-photon imaging technology has also achieved remarkable progress. Especially with the widespread use of single-photon avalanche diodes (SPADs), this type of detector can capture and record signals as weak as single photons. Using this technology, sufficient data can be collected in an almost completely dark environment to reconstruct a clear image, achieving imaging under extremely low illuminance or even single-photon conditions, greatly expanding the application boundaries of imaging technology. Single-photon imaging technology utilizes the quantum properties of photons to reconstruct a high-definition image of the target object by precisely collecting and locating the positions of a large number of individual photons on the image plane. SPAD detectors play a crucial role in this progress. Due to their high reliability, robustness, ease of operation, excellent detection efficiency, and extremely low time jitter in the visible to near-infrared wavelength range, they have become an indispensable tool for single-photon imaging technology. Although single-photon imaging technology has made significant progress in the field of optical imaging, this technology still faces a series of challenges. The pixel size of the detectors used in single-photon imaging technology is relatively large, which directly affects the spatial resolution of imaging. In many application scenarios, especially those that require high-precision imaging, the large pixel size limits the ability to capture details. Additionally, multi-source noise is an important problem faced by single-photon imaging technology. Since single-photon imaging relies on extremely low-intensity optical signals, any form of noise (whether from the detector itself, the external environment, or scattered light during the imaging process) can have a significant impact on the imaging results. These noise sources include, but are not limited to, current noise, background light noise, etc. Acting together, they may seriously reduce the signal-to-noise ratio of imaging, thereby affecting the imaging quality and accuracy. To overcome the problem of multi-source noise, the method of the present invention optimizes the reconstruction network module (described in detail later).

[0032] In some other embodiments, taking the number of optimized masks as multiple frames as an example, the mask unit includes a digital micromirror device. Among them, the digital micromirror device is controlled by the trained coded mask layer in the reconstruction network module to generate multiple frames of optimized masks. Figure 3 This is a process diagram of low-illuminance optimized multi-frame mask spectral imaging provided by the embodiments of the present invention. As Figure 3 shown, the imaging process of the imaging module includes: At low illuminance, the light source obtains an original spectral cube on the sample; the original spectral cube enters a digital micromirror device through a beam splitter. The digital micromirror device is controlled by an encoded mask layer that has been trained in a reconstruction network module to generate multiple frames of optimized masks in chronological order. Different frames of optimized masks encode corresponding spatial information into the original spectral cube to obtain multiple first intermediate spectral cubes. Then, in chronological order, the corresponding first intermediate spectral cubes are sent to a dispersive element. After passing through the dispersive element in sequence, the corresponding second intermediate spectral cubes are obtained. Among them, the second intermediate spectral cubes enter a sensor (such as a photon camera) for imaging to obtain multiple frames of measurement values.

[0033] Among them, a digital micromirror system (Digital Micromirror Devices, DMD) is used to implement multi-frame mask modulation of the spectral cube. Through a jointly optimized reconstruction network of encoding and decoding (i.e., a spectral reconstruction network model), a set of different encoded masks is adaptively learned and loaded onto the DMD to modulate the spectral cube, combined with a single-photon camera or a low-illuminance CMOS camera (complementary metal-oxide-semiconductor camera) for imaging. The high-speed response ability of the DMD enables the system to adaptively adjust under different lighting conditions. Especially in low-illuminance scenarios, through the combination of multi-frame modulation acquisition and a high-frame-rate camera, the signal-to-noise ratio of imaging can be significantly improved and the imaging quality can be enhanced. In a low-illuminance environment, the photon count is extremely low, resulting in a weak intensity of the imaging signal, and thus a low signal-to-noise ratio. In this case, traditional static mask methods are difficult to effectively capture sufficient photon signals, thereby affecting the reconstruction accuracy of the image. Through the multi-frame modulation acquisition method, the system can use different mask patterns for acquisition in each frame. This multiple acquisition can effectively increase the photon utilization rate, thereby enhancing the signal intensity and suppressing noise. The different encoding patterns for each acquisition provide multi-perspective information about the scene, so that even under low illuminance, the image reconstruction quality after accumulating multiple frames still maintains a high accuracy. In the case of low illuminance, the DMD can quickly adjust the mask pattern, increase the number of photons collected, and perform encoding in different patterns at multiple time points. This process can adaptively cope with the challenges in low-illuminance scenarios.

[0034] In an embodiment of the present invention, the reconstruction network module is used to jointly optimize the spectral imaging system parameters (i.e., the parameters of the imaging module) and the reconstruction algorithm using a deep learning algorithm under low-illuminance conditions to reconstruct a high-resolution spectral cube from the intensity measurement map (i.e., obtain Figure 1 the reconstruction result). Among them, the intensity measurement map is the measurement value output by the acquisition unit.

[0035] Specifically, the reconstruction network module includes a trained spectral reconstruction network model; the spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer connected in sequence. The device simulation layer is used to simulate the acquisition unit. Among them, the encoding mask layer of the trained spectral reconstruction network model is used to control the mask unit to generate an optimized mask, and the spectral recovery layer of the trained spectral reconstruction network model is used to reconstruct the measurement values output by the imaging module to obtain the corresponding reconstructed spectral cube.

[0036] Figure 4 This is the multi-source noise-constrained deep learning model diagram based on the full-link physical process provided by the embodiments of the present invention. First, the acquisition process of the trained spectral reconstruction network model (also called the reconstruction algorithm) will be described below. The acquisition process of the trained spectral reconstruction network model includes: 1) Construct a spectral reconstruction network model. As Figure 1 shown, the constructed spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer.

[0037] Encoding mask layer: Encoding is responsible for modulating the spatial structure of the spectral cube input to the model, and its structure is used as an optimizable parameter of the spectral reconstruction network model and is continuously optimized during the iterative process of the spectral reconstruction network model. In the present invention, the encoding mask layer is regarded as the optimizable part of the reconstruction algorithm and is denoted as the mask layer. The number of images output by the mask layer can be one frame or multiple frames; the encoding mask layer can be a binary mask layer. The encoding mask layer corresponds to the mask unit in the imaging module and plays a role in spatial modulation in the CASSI system. The input of the encoding mask layer is the spectral cube. The input of the encoding mask layer is the input of the spectral reconstruction network model.

[0038] Device simulation layer: Corresponding to the acquisition unit in the imaging physical model, after modulating the spatial information using a single frame or multiple frames of masks in the spatial dimension, the spectral information is modulated according to the chromatic dispersion step of the actual system in the spectral dimension, and the modulated spectral cube is digitally propagated to the single-photon camera plane, and a grayscale image (i.e., intensity map) is captured on the camera plane. In the present invention, the process of propagating the masked spectral cube to the dispersion element and the camera plane is simulated by a simulation device and is denoted as the device simulation layer, which is the second part of the reconstruction algorithm. The device simulation layer is a fixed parameter layer in the spectral reconstruction network model. The input of the device simulation layer is the output of the encoding mask layer. The output of the device simulation layer is the measurement value (i.e., grayscale image).

[0039] Spectral recovery layer: The deep learning network is used as the spectral recovery layer, which consists of two parts: a spectral perception layer and a spatial perception layer. Among them, the spectral perception layer includes an encoding layer, a shallow feature extraction module, downsampling, a shallow feature extraction module, downsampling, a deep feature extraction module, upsampling, a feature recovery module, upsampling, a feature recovery module, and a mapping layer (such as Figure 4 shown). Among them, the encoding layer, the shallow feature extraction module, downsampling, the shallow feature extraction module, and downsampling constitute the shallow feature encoding stage of the spectral perception stage, the deep feature extraction module constitutes the deep feature extraction stage of the spectral perception stage, and upsampling, the feature recovery module, upsampling, the feature recovery module, and the mapping layer constitute the feature reconstruction stage of the spectral perception stage. The structure of the spatial perception layer can be analogous to the structure of the spectral perception layer (such as Figure 4 shown). The shallow feature encoding stage, the deep feature extraction stage, and the feature reconstruction stage of the spatial perception stage can be analogous to the corresponding stages of the spectral perception stage. The spectral recovery layer corresponds to the numerical reconstruction part. In some embodiments, the spectral recovery layer can adopt a multi-source noise-constrained deep learning model based on the full-link physical process, which is used to decode and map the captured grayscale image into a spectral cube. The input of the spectral recovery layer is the measured value, and the output is the reconstructed spectral cube. The output of the spatial perception layer is the output of the spectral reconstruction network model.

[0040] Connecting the above three layers in series is the joint optimization reconstruction network architecture of encoding and decoding. This architecture introduces the parameters of the imaging physical model (i.e., the CASSI system parameters) into network training and combines them with the spectral reconstruction process to achieve joint optimization of software and hardware.

[0041] 2) Construct a training dataset. Based on the open-source dataset, a training dataset (also called a spectral dataset) composed of multiple spectral cubes is obtained.

[0042] The open-source training dataset can consist of hyperspectral images in the CAVE dataset and the KAIST dataset.

[0043] The CAVE dataset contains multispectral images of 32 static scenes. Each scene contains 31 bands, and the image resolution is 512×512 pixels, covering the visible spectral range from 400 to 700 nanometers, with one band every 10 nanometers. This dataset records the multispectral characteristics of various objects and materials, such as various textiles, real and simulated fruits, candies, and beverages.

[0044] In contrast, the KAIST dataset provides 30 high-resolution hyperspectral images with a spatial resolution as high as 2704×3376 pixels, containing more delicate details. The network makes the binary mask layer learn the optimal mask layer parameters through continuous iteration to achieve optimal spatial modulation.

[0045] Select multiple images based on the CAVE dataset or the KAIST dataset. The spectral cubes of the selected images form the training dataset.

[0046] 3) Train the spectral reconstruction network model. Use the training dataset to train the spectral reconstruction network model to obtain a trained spectral reconstruction network model. Among them, the input of the spectral reconstruction network model is the spectral cube. The input of the spectral perception layer is the single-frame or multi-frame intensity maps captured after the original spectral data (referring to the spectral cubes in the training dataset) pass through the masking layer and the device simulation layer, and the label is the single-frame or multi-frame images after the original spectral data pass through the masking layer (i.e., the input data of the device simulation layer). The input of the spatial perception layer is the output of the spectral perception layer, and the label is the original spectral data (i.e., the input data of the spectral reconstruction network model). The output of the spectral perception layer (i.e., the output of the spectral reconstruction network model) is the reconstructed high-resolution spectral cube.

[0047] When training the spectral reconstruction network model, if both the spatial perception layer loss function and the spectral perception layer loss function are less than the threshold, the training ends and a trained spectral reconstruction network model is obtained.

[0048] To effectively reduce the influence of noise, multi-source noise constraint terms are introduced into the spatial perception layer loss function and the spectral perception layer loss function during the reconstruction process, including environmental noise constraint and detector noise constraint. These two types of constraints are respectively optimized for the randomly occurring scattered light noise in the environment and the electronic noise of the detector itself. By adjusting the weights of these two constraint terms in multiple experiments, the influence of environmental noise and detector noise on the imaging quality can be flexibly balanced according to the actual situation. This weight adjustment strategy not only enables the reconstruction algorithm to more precisely suppress noise in specific application scenarios, but also further improves the accuracy and fidelity of the reconstructed images. Since the pixel size of the single-photon avalanche diode is relatively large, the discretized sampling of the sensor often results in an undersampling problem. Therefore, the process of the single-photon camera collecting light field information can be described as:

[0049] where D is the downsampling process, Y d is the captured single-channel grayscale image. T n ( x, y, z ) is the wavefront function of the sensor plane.

[0050] Such as Figure 4As shown, different from the spectral reconstruction algorithms in previous CASSI systems, the multi-source noise-constrained deep learning model based on the full-link physical process of the present invention fully considers the forward physical process of CASSI. The network is divided into two stages. The first stage simulates the spectral encoding process in the CASSI process, and the second stage simulates the spatial encoding process in CASSI. Through the learning of the two stages, the original spectral cube can be restored with high fidelity.

[0051] As Figure 4 shown, during training, the input of the first stage (i.e., the spectral perception stage) is the single-channel grayscale image collected by the camera plane in the device simulation layer Y d , and through the shallow feature encoding stage, deep feature extraction stage, and feature reconstruction stage of the spectral perception stage, the image after spatial masking is output, and the input data of the device simulation layer corresponding to the acquisition unit is used as the ground truth for supervision. The input of the second stage (i.e., the spatial perception stage) is the output of the first stage. Through the shallow feature encoding stage, deep feature extraction stage, and feature reconstruction stage of the spatial perception stage, the reconstructed spectral cube is output, and the original spectral data is used for supervision. Through the two-stage process, the forward physical model of CASSI is fully simulated, and the spatial modulation and spectral encoding are separately learned, which can improve the reconstruction quality and achieve high-quality spectral cube reconstruction.

[0052] In an embodiment of the invention, after obtaining the trained spectral reconstruction network model, during the actual imaging process, the encoding mask layer in the trained spectral reconstruction network model controls the mask unit in the imaging module to obtain one or more optimized masks required during actual imaging. The measurement value obtained by the imaging module during the actual imaging process is input into the spectral recovery layer of the trained spectral reconstruction network model for reconstruction to obtain the required reconstructed spectral cube.

[0053] In an embodiment of the invention, both the spectral perception layer and the spatial perception layer adopt a neural network structure. The neural network structure can be a deep learning model. The spectral recovery layer can be a multi-source noise-constrained deep learning model based on the full-link physical process.

[0054] Figure 4 The imaging process in Figure 2 uses the Figure 2 and Figure 4 shown imaging module. Specifically, in combination with a spectral cube as an example, this spectral cube is modulated by an optimized mask to obtain the first intermediate spectral cube , and the i th wavelength band of the first intermediate spectral cube Satisfy:

[0055] In the formula, Denotes element multiplication, Denotes the spectral cube I In the i th wavelength band, . Denotes the number of wavelength bands. 、 Denotes the length and width of the spectral cube I In.

[0056] The first intermediate spectral cube Is dispersed by a prism, and images of different frequencies are displaced in the dispersion direction. The displaced spectral image (i.e., the second intermediate spectral cube) Can be expressed as:

[0057] In the formula, Denotes the plane coordinates of a single-photon sensor (i.e., a single-photon camera), x, y Denotes the two-dimensional spatial coordinates of the first intermediate spectral cube . Denotes the i th band, Denotes the reference wavelength, Denotes the i th band's translation distance relative to the reference band. Finally, the signal is integrated on the single-photon camera to obtain the grayscale acquisition data of a single channel (i.e., the measurement value output by the imaging model) as:

[0058] In the formula, Z is the multi-source noise of the single-photon camera.

[0059] After obtaining the grayscale acquisition data, it is sent to the spectral recovery layer, and the original target image (i.e., the reconstructed spectral cube) is recovered through the spectral perception layer and the spatial perception layer (as shown in Figure 4 ).

[0060] For the single-frame optimized mask of the system of the present invention, during the imaging process, the optimized mask layer is fabricated by means of photolithography or 3D printing and added to the imaging module. The original spectral cube undergoes mask modulation and dispersion processes to form a grayscale image on the single-photon camera. The grayscale image collected by the sensor is input into the trained spectral recovery layer, and the original spectral cube can be reconstructed. For the single-frame mask, spectral and spatial information can be obtained simultaneously through a single exposure, significantly improving the imaging efficiency, and it is applicable to dynamic scenes and high temporal resolution requirements. For the system corresponding to the multi-frame optimized mask, the multi-frame mask is optimized, and more spectral information is obtained through multiple exposures, which helps to improve the reconstruction accuracy and detail presentation of the image. Compared with the traditional compressive sensing algorithm, the joint optimization algorithm of encoding and decoding proposed in the present invention jointly learns the imaging parameters and the decoding network, can reconstruct high-resolution and hyperspectral images, eliminate artifacts, and has the ability to reconstruct the spectra of dynamic objects.

[0061] The following are the method embodiments of the present invention. For the details not disclosed in the method embodiments of the present invention, please refer to the system embodiments of the present invention. The method embodiments of the present invention propose a low-light spectral imaging method based on mask optimization. The low-light spectral imaging method based on mask optimization is applied to the low-light spectral imaging system based on mask optimization in the above system embodiments.

[0062] Figure 5 It is a flowchart of the low-light spectral imaging method based on mask optimization provided by the embodiments of the present invention.

[0063] As Figure 5 shown, the low-light spectral imaging method based on mask optimization includes: Step S101, obtaining a trained spectral reconstruction network model; the spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer connected in sequence. The input of the spectral recovery layer is the measurement value, and the output is the reconstructed spectral cube; Step S102, using the encoding mask layer of the trained spectral reconstruction network model to control the mask unit of the imaging module to generate an optimized mask; Step S103, inputting the original spectral cube under low light into the imaging module for imaging to obtain a measurement value; Step S104, inputting the measurement value into the spectral recovery layer of the trained spectral reconstruction network model for reconstruction to obtain the corresponding reconstructed spectral cube.

[0064] In step S101, the training process of the spectral reconstruction network model includes: obtaining a training data set composed of multiple spectral cubes based on an open-source data set; training the spectral reconstruction network model using the training data set to obtain a trained spectral reconstruction network model. Among them, when training the spectral reconstruction network model, if both the spatial perception layer loss function and the spectral perception layer loss function are less than the threshold, the training ends.

[0065] In step S101, the spectral recovery layer includes a spectral perception layer and a spatial perception layer, and both the spectral perception layer and the spatial perception layer adopt a neural network structure.

[0066] In step S102, using the encoding mask layer of the trained spectral reconstruction network model to control the mask unit of the imaging module to generate an optimized mask includes: using the encoding mask layer of the trained spectral reconstruction network model to control the mask unit of the imaging module to generate a single-frame optimized mask or multiple-frame masks.

[0067] It should be noted that the foregoing explanation of the embodiment of the low-light spectral imaging system based on mask optimization also applies to the low-light spectral imaging method based on mask optimization of this embodiment, and will not be elaborated here.

[0068] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0069] In the low-light spectral imaging system and method based on mask optimization according to the embodiments of the present invention, the system includes an imaging module and a reconstruction network module. The imaging module includes a mask unit and an acquisition unit. The acquisition unit includes a dispersion element and a single-photon image acquisition device. The imaging module is configured to image the original spectral cube under low light to obtain a measurement value. The reconstruction network module includes a trained spectral reconstruction network model. The spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer connected in sequence. The device simulation layer is configured to simulate the acquisition unit. The input of the spectral recovery layer is the measurement value, and the output is the reconstructed spectral cube. The encoding mask layer of the trained spectral reconstruction network model is configured to control the mask unit to generate an optimized mask. The spectral recovery layer of the trained spectral reconstruction network model is configured to reconstruct the measurement value output by the imaging module to obtain the corresponding reconstructed spectral cube. In this case, the reconstruction network module includes a trained spectral reconstruction network model. The spectral reconstruction network model includes an encoding mask layer, a device simulation layer, and a spectral recovery layer connected in sequence. The device simulation layer is configured to simulate the acquisition unit. The input of the spectral recovery layer is the measurement value, and the output is the reconstructed spectral cube. The encoding mask layer of the trained spectral reconstruction network model is configured to control the mask unit to generate an optimized mask. The spectral recovery layer of the trained spectral reconstruction network model is configured to reconstruct the measurement value output by the imaging module to obtain the corresponding reconstructed spectral cube. Therefore, the spectral reconstruction network model fully considers the imaging physical process, realizes the joint optimization of the imaging process and the spectral reconstruction network model, and thus improves the imaging accuracy of the snapshot spectral imaging system under low-light conditions.

[0070] In order to improve the imaging accuracy and reconstruction performance of the snapshot spectral imaging system under low illumination conditions, the present invention proposes a low illumination spectral imaging system and method based on mask optimization, combines the deep learning algorithm with the hardware design of the spectral imaging system, and realizes high-resolution spectral image reconstruction under low illumination conditions through a joint optimization scheme. The optimization method adopted by the present invention can effectively improve the signal-to-noise ratio, improve the spatial resolution, and overcome the technical bottleneck of the traditional method in a low illumination environment. Specifically, it includes constructing a spectral reconstruction network model based on multi-source noise constraints of the full-link physical process. Joint training coding mask and reconstruction method. When training the network, the spectral data set is used as the training set and input into the spectral reconstruction network. The mask layer of the network can learn the optimal encoding method in the continuous training process, and enable the spectral recovery layer to learn the mapping relationship between the collected data and the spectral cube (that is, it can learn the optimal encoding method from the spectral and spatial information of the spectral cube), realizing the joint optimization of the system algorithm. Build an imaging module and capture the intensity map on the camera plane. The optimized mask is manufactured by digital micromirror devices, lithography, 3D printing, etc. and added to the imaging module. A prism is used to disperse information in different bands, and finally the spatially and spectrally modulated intensity map (i.e., acquisition data) is recorded on the camera plane to encode spectral and spatial information into the intensity map. The intensity map is input and a high-resolution, high-spectral original spectral cube is obtained. The trained deep learning algorithm jointly optimizes the parameters of the mask and the spectral recovery layer, and can quickly and high-quality reconstruct the original spectral cube under low illumination, avoiding the problems of slow reasoning speed and poor reconstruction quality of traditional algorithms, and has the ability to reconstruct the spectrum of dynamic objects. In addition, due to the Poisson process of photons, crosstalk of the detection array, and after-pulse noise, the image obtained by the camera under low illumination conditions often loses the clarity of the original target. Through multi-time point information acquisition, the camera can better capture details, learn the mapping relationship between the intensity map and the original spectral data, and reconstruct a high-resolution spectral cube. The system and method of the present invention use a deep learning algorithm to jointly optimize the parameters of the low-illumination spectral imaging system and the reconstruction algorithm, and reconstruct a high-resolution spectral image from a single-frame or multi-frame intensity measurement map. It can realize the joint optimization of spectral imaging system parameters and deep learning algorithms, and solve the problems of low signal-to-noise ratio, poor spectral reconstruction quality and low spatial resolution under low illumination conditions through the method of multiple masks and multiple acquisitions, realizing the joint optimization of software and hardware, which not only improves the reconstruction speed, but also improves the image reconstruction quality.

[0071] The schematic structural diagrams according to the disclosed embodiments of the present invention are shown in the accompanying drawings. These drawings are not drawn to scale, where certain details are enlarged for the purpose of clear expression, and some details may be omitted. The shapes of various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art can additionally design regions / layers with different shapes, sizes, and relative positions according to actual requirements.

[0072] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. The present invention is not limited herein.

[0073] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A low-illumination spectral imaging system based on mask optimization, characterized in that: include: The imaging module includes a mask unit and a collection unit, wherein the collection unit includes a dispersion element and a single-photon image collection device, and the imaging module is used to image the original spectral cube under low illumination to obtain a measurement value; The reconstruction network module includes a trained spectral reconstruction network model; the spectral reconstruction network model includes a coding mask layer, a device simulation layer and a spectral recovery layer connected in sequence, the device simulation layer is used to simulate the acquisition unit, the input of the spectral recovery layer is a measurement value, and the output is a reconstructed spectral cube; The encoded mask layer of the trained spectral reconstruction network model is used to control the mask unit to generate an optimized mask, and the spectral recovery layer of the trained spectral reconstruction network model is used to reconstruct the measurement values ​​output by the imaging module to obtain the corresponding reconstructed spectral cube.

2. The low-illumination spectral imaging system based on mask optimization according to claim 1, characterized in that: The mask unit includes a digital micromirror device.

3. The low-illumination spectral imaging system based on mask optimization according to claim 1, characterized in that: The spectrum restoration layer includes a spectrum perception layer and a space perception layer.

4. The low-illumination spectral imaging system based on mask optimization according to claim 3, characterized in that: The spatial perception layer and the spectral perception layer both adopt a neural network structure.

5. The low-illumination spectral imaging system based on mask optimization according to claim 1, characterized in that: The spectrum restoration layer is a multi-source noise constrained deep learning model based on the full-link physical process.

6. A low-illumination spectral imaging method based on mask optimization, characterized in that: The method applied to the low-illumination spectral imaging system based on mask optimization according to any one of claims 1 to 5 comprises: Acquire a trained spectrum reconstruction network model; the spectrum reconstruction network model comprises a coding mask layer, a device simulation layer and a spectrum recovery layer connected in sequence, the input of the spectrum recovery layer is a measurement value, and the output is a reconstructed spectrum cube; Using the encoded mask layer of the trained spectral reconstruction network model to control the mask unit of the imaging module to generate an optimized mask; Inputting the original spectral cube under low illumination into the imaging module for imaging to obtain a measurement value; The measured values ​​are input into the spectrum recovery layer of the trained spectrum reconstruction network model for reconstruction to obtain the corresponding reconstructed spectrum cube.

7. The low-illumination spectral imaging method based on mask optimization according to claim 6, characterized in that: Using the encoded mask layer of the trained spectral reconstruction network model to control the mask unit of the imaging module to generate an optimized mask includes: The encoded mask layer of the trained spectral reconstruction network model is used to control the mask unit of the imaging module to generate a single-frame optimized mask or a multi-frame mask.

8. The low-illumination spectral imaging method based on mask optimization according to claim 6, characterized in that: The training process of the spectral reconstruction network model includes: Obtain a training dataset consisting of multiple spectral cubes based on open source datasets; The spectral reconstruction network model is trained using the training data set to obtain a trained spectral reconstruction network model.

9. The low-illumination spectral imaging method based on mask optimization according to claim 8, characterized in that: When training the spectral reconstruction network model, if the spatial perception layer loss function and the spectral perception layer loss function are both less than a threshold, the training ends.

10. The low-illumination spectral imaging method based on mask optimization according to claim 6, characterized in that: The spectrum restoration layer includes a spectrum perception layer and a space perception layer, and both the space perception layer and the spectrum perception layer adopt a neural network structure.

Images