Quantum image sensing method and apparatus based on exposure modulation

Abstract

The application discloses a quantum image sensing method and device based on exposure modulation. The sensing end performs frame exposure modulation on the same scene in K continuous frames, and controls the expected number of incident photons of each frame; the pixel outputs a 1-bit binary image in each frame and obeys a Poisson distribution. The acquisition end performs maximum a posteriori estimation on K frames as a group to obtain a reference intensity estimation of each pixel. In order to improve robustness and speed, the reconstruction process adopts a RED-PRO iterative algorithm based on FISTA acceleration: after the gradient descent of the data consistency term, the projection is performed to the pre-trained denoiser manifold, and the pixel adaptive step is given based on the Fisher information. The method significantly improves the signal-to-noise ratio and suppresses artifacts in low-light, saturation and dynamic range limited scenes, and does not need to change the 1-bit sensing circuit architecture, and only needs to be compatible with the existing QIS / SPAD sensor and lens module through exposure modulation and software reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of image sensors and image processing technology, and specifically to a quantum image sensing method and apparatus based on exposure modulation. Background Technology

[0002] Quantum image sensors (QIS) are a novel single-photon-level imaging sensing technology. Their core feature is that the exposure of each pixel is divided into many high-speed frames, with each frame accumulating only a very small number of photons, allowing each frame's output to be quantized into a single bit of information. Specifically, in a single-bit QIS, each frame reads out one bit: if a pixel detects at least one photon within that frame, it outputs "1"; otherwise, it outputs "0". Since photon arrival exhibits random Poisson statistical properties, for an exposure frame with an average photon count of N, the probability that a pixel has not received a photon is... The probability of receiving at least one photon is 1- .

[0003] QIS (Quick Induction Sensor) was developed to address the performance limitations of traditional CMOS / CCD image sensors in extremely low light and high dynamic range conditions. Traditional image sensors struggle to detect single photons in low light due to readout noise limitations; conversely, they are prone to saturation in strong light, thus limiting their dynamic range. In contrast, QIS consists of numerous ultra-small pixels (called "jots"), possessing single-photon detection capability and an extremely high readout frame rate (>100kHz). By performing digital integration and denoising on these spatiotemporal cubic data, a certain quality of grayscale image can still be reconstructed even in extremely low light conditions where the average number of photons per pixel is less than one.

[0004] Currently, image reconstruction based on QIS mainly relies on statistical inference and regularization methods. On the one hand, since QIS readouts conform to Poisson-Bernoulli statistics, the photon arrival rate of pixels can be inferred through maximum likelihood estimation. On the other hand, to suppress noise and underdeterminism in reconstruction, effective prior regularization needs to be introduced. Existing methods include simple frame accumulation averaging, thresholding, and more advanced Bayesian estimation and iterative algorithms. However, existing technologies still have shortcomings in reconstruction quality and convergence speed. For example, direct accumulation cannot fully utilize the nonlinear information of QIS, Bayesian methods are computationally complex, and conventional iterative reconstruction is prone to slow convergence or getting trapped in local extrema under high noise and weak signal conditions. Therefore, a new scheme combining intelligent exposure modulation and efficient reconstruction algorithms is urgently needed to improve the image signal-to-noise ratio sacrificed by QIS for high frame rate and high dynamic range, and further promote the practical application of QIS. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a quantum image sensing method and apparatus based on exposure modulation. Without increasing the complexity of sensor hardware, it utilizes the modulation of exposure time under different frames to achieve a coding-like effect, thereby reducing the underdeterminacy of the target optimization problem during reconstruction and improving the final imaging effect. In addition, it integrates the advantages of FISTA and RED-PRO (Regularization by Denoising via Fixed-Point Projection) optimization algorithms to accelerate the convergence speed during reconstruction.

[0006] The method of the present invention includes the following steps: a quantum image sensing method based on exposure modulation, the method comprising the following steps:

[0007] (S1) For pixels within the same field of view, a pre-set exposure modulation mode is applied to each frame during continuous multi-frame exposure acquisition to adjust the desired number of incident photons in each frame.

[0008] (S2) Obtain the binary output image of each frame that follows a Poisson distribution through a quantum image sensor, establish a likelihood function for the expected number of incident photons for each pixel, and use the negative log-likelihood as the fidelity loss function with the optimization objective.

[0009] (S3) The maximum a posteriori estimation is used to iteratively optimize the reconstruction of the image for the negative log-likelihood.

[0010] (S5) During the iterative reconstruction process, the RED-PRO iterative algorithm based on FISTA acceleration is used. After each gradient descent / accelerated gradient descent, the algorithm proposed in RED-PRO (Regularization by Denoising via Fixed-Point Projection) is applied. The operator, utilizing a pre-trained denoising model, projects the image / video to be reconstructed onto a neural network denoiser. On the defined high-dimensional manifold, an implicitly regularized and fast-converging reconstruction effect is achieved.

[0011] Furthermore, the expected number of photons in the k-th frame is Where N is proportional to the pixel grayscale. The modulation function is known; the modulation function It is a linear modulation function, i.e. ,in For the k-th frame, there is a known coefficient that is proportional to the exposure time T.

[0012] Furthermore, during the iterative reconstruction process, the FISTA-accelerated RED-PRO iterative algorithm is used. After each gradient descent / accelerated gradient descent step, a pre-trained denoising model is used to project the image / video to be reconstructed onto the neural network denoiser. On the defined high-dimensional manifold, an implicitly regularized and fast-converging reconstruction effect is achieved.

[0013] Furthermore, the denoising operator To utilize neural network denoisers pre-trained using natural image / video or target domain data, including image denoisers DnCNN, DRUNet, Restormer, or video denoisers FastDVDNet, RVRT structures, and as prior operators in RED-PRO.

[0014] Furthermore, the FISTA-accelerated RED-PRO iterative algorithm mentioned above refers to replacing the gradient descent process of the original RED-PRO with Nesterov accelerated gradient technology to speed up the convergence.

[0015] Furthermore, the entire video frame is divided into groups of K frames, and each group is estimated independently.

[0016] In a second aspect, the present invention also provides a quantum image sensing device, comprising: a modulation control unit, an optical lens module, a quantum image sensor array, a readout and timing control circuit, and a processor; wherein the modulation control unit is used to generate control of frame-by-frame exposure time or incident intensity; the optical lens module acquires images based on the lens, the quantum image sensor array is a one-bit quantization readout structure with sub-electron noise output; and the processor is used for the maximum a posteriori estimation reconstruction process.

[0017] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the device to implement the aforementioned quantum image sensing method based on exposure modulation.

[0018] The beneficial effects of this invention are as follows:

[0019] This invention proposes the use of exposure modulation during QIS imaging, which achieves a coding-like effect, thereby reducing the ill-conditioning of reconstruction problems and improving reconstruction results to a certain extent.

[0020] This invention combines the Nesterov acceleration gradient technique from FISTA with the RED-PRO technique. The operator projects the reconstructed object onto the fixed point set of the denoiser, achieving a reconstruction that is both fast and robust.

[0021] The method of this invention can achieve high-quality image reconstruction with a relatively small number of synthesized frames. Without any neural network training or hardware modifications, it can significantly improve imaging performance when synthesizing a grayscale image from a very small number of 1-bit frames (e.g., 4 frames), thus promoting the practical application of QIS in the field of high-speed imaging. Attached Figure Description

[0022] Figure 1 This is an overall framework diagram of the quantum image sensing method based on exposure modulation provided in an embodiment of the present invention.

[0023] Figure 2 The simulation results with and without exposure modulation are shown in the embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] Figure 1 This is an overall framework diagram of the quantum image sensing method based on exposure modulation provided in an embodiment of the present invention, with reference to... Figure 1 The specific steps of this invention are as follows:

[0027] 1. Exposure Modulation Acquisition: For pixels within the same field of view, a pre-set exposure modulation mode is applied to each frame in consecutive multi-frame exposure acquisition. Specifically, given that the pixel grayscale value of the reconstructed image is proportional to its corresponding reference photon number N, the desired incident photon number in the k-th frame is achieved by controlling the exposure time or gain of each frame. .in Given a known modulation function, it can be calculated according to a predetermined rule based on the frame number k. For example, in the case of linear modulation, take... ( (where is the relative exposure coefficient of the k-th frame), thus achieving a similar encoding effect for the same scene information, which can significantly reduce the underdeterminacy of the optimization problem.

[0028] 2. Single-photon detection readout: Using a QIS sensor such as a single-photon detector (SPAD), the light signal after each frame of modulation exposure is read out to obtain a binarized output image frame. Since a single-bit QIS pixel can only represent the presence or absence of one photon per frame, the output... This indicates that at least one photon was detected in this pixel of the frame. This indicates that no photon was detected. According to the Poisson statistics of photon arrival, The probability is determined by the expected number of photons in that frame. Decision, Obedience It can also be conveniently written in the form of a Bernoulli distribution: .in For the quantum effect of the sensor, This is equivalent to the actual average number of incident photons per pixel in the k-th frame. The formula above shows that the longer the exposure time, the higher the desired number of photons. The larger the value, the higher the probability of a "1" appearing in that frame; conversely, a short exposure frame is more likely to output a "0". This is achieved by using K frames with different... The combination of observations can yield rich statistical information about the reference photon number N.

[0029] 3. Constructing the Likelihood Function: Taking the binary readouts of the above K frames as a set of observations, a statistical likelihood function is constructed for the reference photon number N of each pixel. Based on the independence of Bernoulli outputs, the joint likelihood of each frame can be expressed as:

[0030]

[0031] Log-likelihood can generally be taken as the optimization objective; therefore, the log-likelihood in the above equation can be expressed as:

[0032]

[0033] 4. While maximum likelihood estimation can be directly performed using the log-likelihood from the previous step, the results are generally poor. Therefore, we consider using a maximum a posteriori estimation framework, such as RED-PRO, to project the reconstructed image onto the denoiser's manifold. This process can also be viewed as an implicit regularization of the reconstructed image. After obtaining the gradient, FISTA-accelerated RED-PRO can be performed using the following framework:

[0034]

[0035] The superscripts t and t-1 represent the variables in the current iteration step or the variables in the previous iteration step, respectively. Intermediate variables representing momentum information are also known as acceleration extrapolation points; This represents the result of performing a gradient descent at the acceleration extrapolation point; This represents a noise denoiser; This represents the intermediate variable obtained after the noise denoiser is applied. Represents the target variable being optimized; This represents the FISTA acceleration factor.

[0036] The aforementioned exposure modulation can be achieved not only by adjusting the camera's exposure time, but also by controlling the emitted light intensity of the illumination source.

[0037] The likelihood function can be supplemented with common QIS noise statistics such as dark counts and after-pulse to achieve more accurate modeling.

[0038] The FISTA-accelerated RED-PRO framework can actually be easily replaced by iterative algorithms such as RED, PnP-FISTA, and native RED-PRO. However, the FISTA-accelerated RED-PRO significantly outperforms prior constraint algorithms based on denoiser fixed points, such as RED and RED-PRO, in terms of convergence speed. Become Compared to PnP-FISTA, it exhibits better convergence and avoids non-convergence in the optimization process due to the expansion of the denoiser.

[0039] Alternatively, the gradient expression can be written directly in closed form, eliminating the need for automatic differentiation. This saves memory and speeds up reconstruction. The first derivative of the log-likelihood is...

[0040]

[0041] in = , .

[0042] Example: This invention can use any quantum sensor, including but not limited to SPADs, to acquire 1-bit raw data. During acquisition, the exposure time is adjusted for each frame according to a pre-set exposure modulation function. Then, reconstruction is performed using the loss function and reconstruction algorithm of this invention, ultimately yielding high-quality reconstruction results. This scheme is highly superior when the K value is small, as illustrated by the simulation below.

[0043] This invention performs Poisson distribution sampling on a 512*512 pixel grayscale image to simulate the sampling process of a quantum sensor under a certain light flux, obtaining several 1-bit images. In the simulation, the grayscale value of the image is proportional to the desired number of photons N; therefore, only this scaling factor needs to be changed to simulate exposure modulation. Figure 2 As shown, 1-bit simulation images were generated using a constant scaling factor and a pre-known scaling factor, respectively, representing the comparison between no-exposure modulation and exposure modulation. The diagrams clearly show that in the no-exposure modulation scenario, the probability of the overall "1" appearing in all four images is roughly the same, while in the exposure modulation scenario, the human eye can clearly perceive a transition from dark to bright in all four images.

[0044] To promote the use of QIS in high-speed imaging, it is necessary to synthesize grayscale images using as few 1-bit frames as possible. This simulation uses four images. In practical applications, if a 100kHz single-photon camera (such as Piimaging's SPAD512) is used as the QIS, the final imaging frame rate can reach as high as 25kHz without any resolution loss or frame rate limitation.

[0045] The FISTA-accelerated RED-PRO algorithm is used for maximum a posteriori estimation, with negative log-likelihood as the loss function. In this simulation, the RVRT video denoiser is used as the prior, and the image can be reconstructed in as few as a few dozen iterations.

[0046] from Figure 2 The simulation results below also show that, in the synthesis process with a very small number of frames, the results using exposure modulation are significantly better than those using no exposure modulation. From a quantitative perspective, compared to the traditional method of direct synthesis without exposure modulation, using exposure modulation can increase the PSNR from 12.44dB to 26.05dB.

[0047] Corresponding to the aforementioned embodiment of a quantum image sensing method based on exposure modulation, the present invention also provides an embodiment of a quantum image sensing device, comprising: a modulation control unit, an optical lens module, a quantum image sensor array, a readout and timing control circuit, and a processor; wherein the modulation control unit is used to generate control of frame-by-frame exposure time or incident intensity; the optical lens module acquires images based on the lens, the quantum image sensor array is a one-bit quantization readout structure with sub-electron noise output; and the processor is used for the maximum a posteriori estimation reconstruction process.

[0048] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0049] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0050] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the device to implement the aforementioned quantum image sensing method based on exposure modulation.

[0051] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0052] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A quantum image sensing method based on exposure modulation, characterized in that, The method includes the following steps: (S1) For pixels within the same field of view, a pre-set exposure modulation mode is applied to each frame during continuous multi-frame exposure acquisition to adjust the desired number of incident photons in each frame. (S2) Obtain the binary output image of each frame that follows a Poisson distribution through a quantum image sensor, establish a likelihood function for the expected number of incident photons for each pixel, and use the negative log-likelihood as the fidelity loss function with the optimization objective. (S3) Maximum a posteriori estimation is used to iteratively optimize and reconstruct the image for negative log-likelihood; During the iterative reconstruction process, the FISTA-accelerated RED-PRO iterative algorithm is used. Based on the native RED-PRO, its gradient descent process is replaced with Nesterov-accelerated gradient techniques to speed up convergence. After each gradient descent / accelerated gradient descent step, the algorithm proposed in RED-PRO is applied... The operator, utilizing a pre-trained denoising model, projects the image / video to be reconstructed onto a neural network denoiser. On the defined high-dimensional manifold, an implicitly regularized and fast-converging reconstruction effect is achieved.

2. The method according to claim 1, characterized in that, The expected number of photons in the k-th frame is Where N is proportional to the pixel grayscale. The modulation function is known; the modulation function It is a linear modulation function, i.e. ,in For the k-th frame, there is a known coefficient that is proportional to the exposure time T.

3. The method according to claim 1, characterized in that, The neural network denoiser For neural network denoisers pre-trained using natural image / video or target domain data, including image denoisers DnCNN, DRUNet, Restormer or video denoisers FastDVDNet, RVRT structures, and used as prior operators in RED-PRO.

4. The method according to claim 1, characterized in that, The entire video frame is divided into groups of 1K frames, and each group is estimated independently.

5. A quantum image sensing device for implementing the method of any one of claims 1-4, characterized in that, include: Modulation control unit, optical lens module, quantum image sensor array, readout and timing control circuit, processor; The modulation control unit is used to generate control over the frame-by-frame exposure time or incident intensity. The optical lens module acquires images based on the lens; the quantum image sensor array is a one-bit quantization readout structure with sub-electron noise output; and the processor is used for the maximum a posteriori estimation reconstruction process.

6. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, causes the device to perform the method of any one of claims 1-4.

Images