Quantum image sensing method and device based on exposure modulation

Abstract

The invention discloses a quantum image sensing method and device based on exposure modulation. The sensing end performs frame-by-frame exposure modulation on the same scene in continuous K frames, and regulates and controls the expected number of incident photons of each frame; the pixels output a 1-bit binary image in each frame and obey Poisson distribution. And the collection end takes K frames as a group to carry out maximum posteriori estimation to obtain reference intensity estimation of each pixel. In order to improve robustness and speed, an RED-PRO iterative algorithm based on FISTA acceleration is adopted in the reconstruction process, after gradient descent of a data consistency item, the data is projected to a pre-training de-noising device manifold, and a pixel adaptive step length is given based on Fisher information. According to the method, the signal-to-noise ratio is remarkably improved and artifacts are inhibited in a scene with low illumination, saturation and limited dynamic range, the 1-bit sensing circuit architecture does not need to be changed, and the existing QIS / SPAD sensor and the lens module can be compatible only through exposure modulation and software reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image sensors and image processing technology, and in particular to a quantum image sensing method and device based on exposure modulation. BACKGROUND

[0002] A quantum image sensor (QIS) is a novel single-photon-level imaging sensing technology, whose core feature is to divide the exposure of each pixel into many high-speed frames, and each frame only accumulates a small number of photons, so that the output of each frame can be quantized to one bit of information. Specifically, in a single-bit QIS, each frame reads out a bit: if the pixel detects at least one photon in the frame, it outputs "1", otherwise it outputs "0". Since the photons have random Poisson statistical characteristics, for an exposure frame with an average number of photons N, the probability that the pixel does not receive a photon is , and the probability that at least one photon is received is .

[0003] The emergence of QIS is to solve the performance limitations of traditional CMOS / CCD image sensors in extremely low illumination and high dynamic range. Traditional image sensors are difficult to detect single photons due to readout noise limitations in weak light; in strong light, they are easily saturated, thereby limiting the dynamic range. In contrast, QIS is composed of numerous super-small pixels (called "jots"), with single-photon detection capability and extremely high readout frame rate (>100khz). By digitally integrating and denoising these spatiotemporal cubic data, even in extremely weak light with an average of less than one photon per pixel, a certain quality of grayscale image can still be restored.

[0004] Currently, image reconstruction for QIS mainly relies on statistical inference and regularization methods. On the one hand, since QIS readout conforms to the Poisson-Bernoulli statistical law, the photon arrival rate of the pixel can be inferred by maximum likelihood estimation; on the other hand, to suppress noise and underdetermination in reconstruction, effective prior regularization needs to be introduced. The existing methods include simple frame accumulation averaging, threshold processing, and more advanced Bayesian estimation and iterative algorithms. However, the existing technology still has deficiencies in reconstruction quality and convergence speed. For example, direct accumulation cannot fully utilize the non-linear information of QIS, Bayesian methods are computationally complex, and conventional iterative reconstruction is prone to slow convergence or local extremum under high noise and weak signal conditions. Therefore, there is an urgent need for a new solution that combines intelligent exposure modulation and efficient reconstruction algorithm to improve the image signal-to-noise ratio sacrificed by QIS for high frame rate and high dynamic range, and further promote the practicality of QIS. SUMMARY

[0005] To overcome the shortcomings of existing technologies, this invention provides a quantum image sensing method and apparatus based on exposure modulation. Without increasing the complexity of sensor hardware, it utilizes the modulation of exposure time under different frames to achieve a coding-like effect, thereby reducing the underdeterminacy of the target optimization problem during reconstruction and improving the final imaging effect. In addition, it integrates the advantages of FISTA and RED-PRO (Regularization by Denoising via Fixed-Point Projection) optimization algorithms to accelerate the convergence speed during reconstruction.

[0006] The method of the present invention includes the following steps: a quantum image sensing method based on exposure modulation, the method comprising the following steps:

[0007] (S1) For pixels within the same field of view, a pre-set exposure modulation mode is applied to each frame during continuous multi-frame exposure acquisition to adjust the desired number of incident photons in each frame.

[0008] (S2) Obtain the binary output image of each frame that follows a Poisson distribution through a quantum image sensor, establish a likelihood function for the expected number of incident photons for each pixel, and use the negative log-likelihood as the fidelity loss function with the optimization objective.

[0009] (S3) The maximum a posteriori estimation is used to iteratively optimize the reconstruction of the image for the negative log-likelihood.

[0010] (S5) During the iterative reconstruction process, the RED-PRO iterative algorithm based on FISTA acceleration is used. After each gradient descent / accelerated gradient descent, the algorithm proposed in RED-PRO (Regularization by Denoising via Fixed-Point Projection) is applied. The operator, utilizing a pre-trained denoising model, projects the image / video to be reconstructed onto a neural network denoiser. On the defined high-dimensional manifold, an implicitly regularized and fast-converging reconstruction effect is achieved.

[0011] Furthermore, the expected number of photons in the k-th frame is Where N is proportional to the pixel grayscale. The modulation function is known; the modulation function It is a linear modulation function, i.e. ,in For the k-th frame, there is a known coefficient that is proportional to the exposure time T.

[0012] Further, in the process of iterative reconstruction, the RED-PRO iterative algorithm based on FISTA acceleration is used, and after each gradient descent / accelerated gradient descent, the pre-trained denoising model is used to project the image / video to be reconstructed to the neural network denoiser On the defined high-dimensional flow pattern, the implicit regularization and fast convergence of the reconstruction effect are realized.

[0013] Further, the denoising operator The neural network denoiser pre-trained using natural images / videos or target domain data includes image denoiser DnCNN, DRUNet, Restormer or video denoiser FastDVDNet, RVRT structure, and is used as a prior operator in RED-PRO.

[0014] Further, the FISTA accelerated RED-PRO iterative algorithm refers to replacing the gradient descent process of the original RED-PRO with Nesterov accelerated gradient technology to speed up the convergence speed.

[0015] Further, the whole video frame is divided into a group every K frames, and each group is estimated independently.

[0016] In a second aspect, the application further provides a quantum image sensing device, comprising: a modulation control unit, an optical lens module, a quantum image sensor array, a readout and time control circuit, and a processor; wherein the modulation control unit is used to generate frame exposure time or incident intensity control; the optical lens module acquires images based on the lens, and the quantum image sensor array is a one-bit quantization readout structure with sub-electron noise output; and the processor is used for maximum a posteriori estimation reconstruction process.

[0017] In a third aspect, the application further provides a computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, causes the device to implement the quantum image sensing method based on exposure modulation.

[0018] The application has the following beneficial effects:

[0019] The application uses exposure modulation in the QIS imaging process, which has a similar coding effect, can reduce the ill-conditioned nature of the reconstruction problem to some extent, and improves the reconstruction effect.

[0020] The application fuses the Nesterov accelerated gradient technology in FISTA and the RED-PRO uses The operator projects the reconstruction object to the fixed point set of the denoiser, realizes fast and robust reconstruction, and has the advantages of

[0021] The method can realize high-quality image reconstruction with less synthetic frames. Without any neural network training and without any hardware modification, when a gray-scale image is synthesized from a small number of 1-bit frames (such as 4 frames), the imaging effect can be significantly improved, and the practicality of QIS in the field of high-speed imaging is promoted. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 An overall framework diagram of the quantum image sensing method based on exposure modulation is provided for an embodiment of the present application.

[0023] Figure 2 A simulation result comparison diagram with and without exposure modulation is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] Figure 1 An overall framework diagram of the quantum image sensing method based on exposure modulation is provided for an embodiment of the present application, with reference to Figure 1 The specific steps of the present application are as follows:

[0027] 1. Exposure modulation acquisition: for the pixels in the same field of view, a pre-set exposure modulation mode is applied to each frame in the continuous multi-frame exposure acquisition. Specifically, the pixel gray value of the reconstructed image is proportional to the corresponding reference photon number N, then by controlling the exposure time or gain of each frame, the expected incident photon number of the kth frame is . Wherein For known modulation function, it can be calculated according to frame index k by predetermined rule. For example, take is the relative exposure coefficient for the k-th frame, thus plays the effect of similar encoding under the same scene information, which can significantly reduce the underdetermined degree of optimization problem.

[0028] 2. Single photon detection readout: using QIS sensors such as single photon detectors (SPAD), the light signal after modulated exposure of each frame is read, and a binary output image frame is obtained . Since a single-bit QIS pixel can only represent "yes or no" a photon in each frame, the output indicates that at least one photon is detected by the pixel in the frame, and indicates that no photon is detected. According to the Poisson statistics of photon arrival, the probability is determined by the expected number of photons of the frame, which is subject to , which can also be conveniently written in the form of Bernoulli distribution: . Where is the quantum efficiency of the sensor, which is equivalent to the actual average number of incident photons of the pixel in the k-th frame. The above formula shows that the longer the exposure time, that is, the greater the expected number of photons , the higher the probability of "1" in the frame; on the contrary, short exposure frame is more likely to output "0". Through the observation combination of K frames with different , rich statistical information about the reference photon number N can be obtained.

[0029] 3. Constructing likelihood function: taking the above K-frame binary readout as a set of observations, a statistical likelihood function is established for the reference photon number N of each pixel. According to the independence of Bernoulli output, the joint likelihood of each frame can be represented as:

[0030]

[0031] Generally, the log-likelihood can be taken as the optimization target, so the log-likelihood of the above formula can be represented as

[0032]

[0033] 4. The log-likelihood in the last step can be directly used for maximum likelihood estimation, but the results of maximum likelihood estimation are generally poor. Therefore, consider using the framework of maximum a posteriori estimation, that is, use the framework of RED-PRO to project the reconstructed image onto the manifold of the denoiser, which can also be regarded as an implicit regularization of the reconstructed image process. After obtaining the gradient, the FISTA accelerated RED-PRO can be carried out according to the following framework:

[0034]

[0035] wherein the superscripts t and t-1 denote the variable in the current iteration step or the variable in the last iteration step, respectively; denotes an intermediate variable with momentum information, also known as an accelerated extrapolation point; denotes the result of a gradient descent at the accelerated extrapolation point; denotes a denoiser; denotes an intermediate variable after the denoiser; denotes an optimized target variable; denotes a FISTA acceleration coefficient.

[0036] The exposure modulation can be achieved by adjusting the camera exposure time, and can also be achieved by adjusting the outgoing light intensity of the illumination light source.

[0037] The likelihood function can be added to the common noise statistics of QIS such as dark count and post-pulse to perform more accurate modeling.

[0038] The FISTA accelerated RED-PRO framework can also be simply replaced by RED, PnP-FISTA, native RED-PRO and other iterative algorithms, but the FISTA accelerated RED-PRO is significantly superior to RED, RED-PRO and other prior constraint algorithms based on the fixed point of the denoiser in convergence speed, and the convergence speed is changed from to Compared with PnP-FISTA, the convergence is better, and the optimization process will not be divergent due to the expansibility of the denoiser.

[0039] Optionally, the expression of the gradient can be directly written in a closed form, which saves the step of automatic derivation, saves memory and speeds up the reconstruction, and the first-order derivative of the log-likelihood is

[0040]

[0041] wherein = , .

[0042] Embodiment: The present application can use any quantum sensor, including but not limited to SPAD and the like, to perform 1-bit raw data acquisition, and during acquisition, the exposure time is adjusted according to the preset exposure modulation function, and then the loss function and reconstruction algorithm of the present application are used for reconstruction, and finally a high-quality reconstruction result can be obtained. When the K value is small, the present scheme has great superiority, which will be described in combination with simulation.

[0043] The present application carries out Poisson distribution sampling on a 512*512 pixel grayscale image, simulates the process of quantum sensor sampling under a certain light flux, and obtains several 1-bit pictures. In the simulation, the grayscale value of the image is proportional to the expected number of photons N, so only the proportional coefficient needs to be changed to simulate exposure modulation. As shown in Figure 2 , a constant proportional coefficient and a pre-known proportional coefficient are used respectively to generate 1-bit simulation pictures, representing the contrast between no exposure modulation and exposure modulation. From the schematic diagram, it can be seen that the overall probability of the appearance of the "1" in the four pictures is similar when there is no exposure modulation, and when there is exposure modulation, the four pictures can be obviously perceived by the human eye to have a process from dark to bright.

[0044] In order to promote the use of QIS in the field of high-speed imaging, it is necessary to synthesize grayscale images with as few 1-bit frames as possible. In this simulation, four pictures are used. In actual use, if a 100khz single-photon camera (such as the SPAD512 of piimaging company) is used as QIS, the final imaging frame rate can be as high as 25khz without any loss of resolution and limitation of frame number.

[0045] The RED-PRO algorithm accelerated by FISTA is used for maximum a posteriori estimation, and the negative log-likelihood is used as the loss function. In this simulation, the RVRT video denoiser is used as the prior, and the image can be reconstructed in as few as dozens of iterations.

[0046] From Figure 2 The simulation results below show that in the synthesis process of a small number of frames, the result of using exposure modulation is significantly better than that of using no exposure modulation. From the quantitative point of view, compared with the traditional method of directly synthesizing without exposure modulation, the use of exposure modulation can improve the PSNR from 12.44dB to 26.05dB.

[0047] Corresponding to the foregoing embodiment of the quantum image sensing method based on exposure modulation, the present application also provides an embodiment of a quantum image sensing device, comprising: a modulation control unit, an optical lens module, a quantum image sensor array, a readout and time control circuit, a processor; wherein the modulation control unit is used to generate frame exposure time or incident intensity control; the optical lens module acquires images based on the lens, and the quantum image sensor array is a one-bit quantization readout structure with sub-electron noise output; the processor is used for maximum a posteriori estimation reconstruction process.

[0048] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0049] For the apparatus embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The apparatus embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0050] The application further provides a computer readable storage medium, which stores a computer program, and the program causes the apparatus to implement the method for quantum image sensing based on exposure modulation when executed by a processor.

[0051] The computer readable storage medium can be an internal storage unit of any data processing device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0052] The specific embodiments described above do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A quantum image sensing method based on exposure modulation, characterized in that, The method includes the following steps: (S1) For pixels within the same field of view, a pre-set exposure modulation mode is applied to each frame during continuous multi-frame exposure acquisition to adjust the desired number of incident photons in each frame. (S2) Obtain the binary output image of each frame that follows a Poisson distribution through a quantum image sensor, establish a likelihood function for the expected number of incident photons for each pixel, and use the negative log-likelihood as the fidelity loss function with the optimization objective. (S3) Maximum a posteriori estimation is used to iteratively optimize and reconstruct the image for negative log-likelihood; (S5) During the iterative reconstruction process, the RED-PRO iterative algorithm based on FISTA acceleration is used. After each gradient descent / accelerated gradient descent, the algorithm proposed in RED-PRO is applied. The operator, utilizing a pre-trained denoising model, projects the image / video to be reconstructed onto a neural network denoiser. On the defined high-dimensional manifold, an implicitly regularized and fast-converging reconstruction effect is achieved.

2. The method according to claim 1, characterized in that, The expected number of photons in the k-th frame is Where N is proportional to the pixel grayscale. The modulation function is known; the modulation function It is a linear modulation function, i.e. ,in For the k-th frame, there is a known coefficient that is proportional to the exposure time T.

3. The method according to claim 1, characterized in that, During the iterative reconstruction process, the FISTA-accelerated RED-PRO iterative algorithm is used. After each gradient descent / accelerated gradient descent, a pre-trained denoising model is used to project the image / video to be reconstructed onto the neural network denoiser. On the defined high-dimensional manifold, an implicitly regularized and fast-converging reconstruction effect is achieved.

4. The method according to claim 3, characterized in that, The denoising operator To utilize neural network denoisers pre-trained using natural image / video or target domain data, including image denoisers DnCNN, DRUNet, Restormer, or video denoisers FastDVDNet, RVRT structures, and as prior operators in RED-PRO.

5. The method according to claim 1, characterized in that, The FISTA-accelerated RED-PRO iterative algorithm mentioned above refers to replacing the gradient descent process of the original RED-PRO with Nesterov accelerated gradient technology to speed up the convergence.

6. The method according to claim 1, characterized in that, The entire video frame is divided into groups of 1K frames, and each group is estimated independently.

7. A quantum image sensing device, characterized in that, include: Modulation control unit, optical lens module, quantum image sensor array, readout and timing control circuit, processor; The modulation control unit is used to generate control over the frame-by-frame exposure time or incident intensity. The optical lens module acquires images based on the lens; the quantum image sensor array is a one-bit quantization readout structure with sub-electron noise output; and the processor is used for the maximum a posteriori estimation reconstruction process.

8. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, causes the device to perform the method of any one of claims 1-7.

Images