Camera design optimization method based on single photon density imaging

Abstract

The invention discloses a camera design optimization method based on single photon density imaging, and the method comprises the steps: capturing incident photon information through a single photon avalanche diode (SPAD) device, and constructing a photon cube for representing the light detection condition at each pixel point and the feature information of photons. The arrival of photons can be modeled as a Poisson process, a time contrast projection technology is adopted to generate a photon event, an index response curve is utilized to avoid underflow, and an event triggering mechanism is optimized, so that the sensitivity under a low light condition is improved, and the optimization of imaging performance is realized. According to the invention, the imaging effect is achieved through the photonic cube information collected by the SPAD and a corresponding algorithm. And higher video reconstruction quality and lower bandwidth cost are realized through optimization of the algorithm, and the method has better practical value.

Description

Technical Field

[0001] The present invention relates to a camera design optimization method based on single photon density imaging, belonging to the technical field of the combination of computer vision and computational photography. Background Art

[0002] The earliest imaging originated from the pinhole imaging principle. Light passes through a small hole and forms an inverted image on the opposite wall in a dark room. This principle laid the foundation for the development of optical imaging. In the development of imaging technology, image capture and processing technology have always been accompanied. Thanks to the continuous innovation of sensor technology, the resolution and efficiency of information acquisition have been significantly improved. In the era of film photography, the core of image processing lies in precise exposure control and complex development process. With the vigorous development of digital photography technology, image processing has been refined to the precision of pixel level, which greatly promotes the overall improvement of image quality.

[0003] Light field photography technology captures the entire light field information, making it easier to process the light field data at a granular level after acquisition, thus realizing the function of post-shooting digital editing. The development of this technology has inspired people to explore the limits of imaging technology. If single photons can be accurately calculated and processed, the acquisition speed and resolution of imaging will be greatly improved.

[0004] Single-photon imaging is an advanced technology that indirectly obtains information about an object based on the interaction between photons and matter. It constructs an image by detecting the number of photons reflected or emitted from the surface of an object. This technology has extremely high sensitivity and can accurately capture weak light signals, providing detailed imaging.

[0005] Photon density imaging is an innovation in photon imaging technology. It obtains images by measuring the density distribution of photons in a unit volume. Compared with traditional imaging methods, photon density imaging has significant advantages in spatial resolution and contrast, especially when dealing with targets with limited number of photons or weak signals, it shows excellent imaging capabilities. This technology has far-reaching significance for improving imaging accuracy and enhancing image quality in weak signal environments, and provides a new solution to imaging problems in complex environments.

[0006] In summary, the present invention proposes a camera design optimization method based on single-photon density imaging. This method uses single-photon devices to capture photon signals, combines time contrast projection technology to generate photon events, and introduces an exponential response curve to avoid underflow. By optimizing the event triggering mechanism, the light sensitivity is improved under low light conditions, thereby optimizing the imaging performance. Summary of the invention

[0007] This paper proposes a camera design optimization method based on single photon density imaging. This method uses a single photon avalanche diode (SPAD) as the core device to accurately capture the incident photon information and construct a photon cube - a binary frame sequence composed of a time series, which is used to characterize the photon detection event at each pixel and its related information. In this framework, the arrival of the incident photon is represented by a Poisson process, and the state of each pixel in the photon cube is represented as an independent Bernoulli random variable Among them B t (x) represents the photon detection at time t and pixel position x, that is, 1 represents the detection of photons and 0 represents the non-detection of photons. Ф(x,t) represents the number of photons received at time t and pixel x, η is the photon detection efficiency, r q is the counting rate of the sensor, ω exp is the exposure time.

[0008] Projection calculation on the generated photon cube is the main step of imaging. Adaptive time-integrated projection algorithm is used to accumulate the number of photons in a specific time region. By accumulating the photon detection results over multiple time periods By obtaining the photon number image in the time region, a clearer image can be generated in a static scene. The temporal contrast projection of the photon cube is used to extract the temporal brightness change information from the photon detection data.

[0009] The present invention provides a camera design optimization method based on single photon density imaging. The technical solutions adopted are mainly as follows:

[0010] Step 1: Use a single-photon avalanche diode (SPAD) cell array as an imaging device to capture photon data in the cube with high temporal resolution;

[0011] Step 2: Collect single photon data and pre-process the captured photon data in the cube. Since the number of photons obtained per unit time is small, a longer period of photon accumulation is required to obtain enough sampled photons to construct an image.

[0012] Step 3: Construct an image based on the detected single photon positions and arrival times, and use signal processing techniques to reduce noise and optimize image quality;

[0013] Step 4: Project the data collected in the cube photons and use the time contrast for event detection to achieve event detection under sufficient lighting conditions.

[0014] Further, the specific steps of step 1 are as follows:

[0015] Step 1.1: Use SPAD to convert incident photons into electricity, such as Figure 1As shown, the photons incident on the SPAD are absorbed by the semiconductor material, which absorbs the photons to obtain energy and generates an excited electron, converting the photon information into an electrical signal;

[0016] Step 1.2: Set the size, cell pitch and frame rate (100kHz) of the SPAD array, calibrate the dark current of the SPAD cell array to reduce false triggering in the absence of light; adjust the SPAD gain to optimize the photon detection rate of the cell;

[0017] Step 1.3: Synchronize the SPAD cell array with the external clock to achieve accurate frame rate control and set the exposure time of each cell to adapt to the lighting environment;

[0018] Step 1.4: Exposure Cycle Each SPAD unit independently detects the incident photons and converts them into binary values ​​(0 or 1).

[0019] Furthermore, the specific steps of step 2 are as follows:

[0020] Step 2.1: Start the light source and adjust the light intensity to the single-photon level, initialize the data and start data acquisition. The unit detector records the arrival time and position of each photon event;

[0021] Step 2.2: Since the number of single photons captured in a short time is small, a long period of cumulative capture is required to ensure signal quality. The signal accumulation value of each unit detector is represented by the image grayscale value;

[0022] Step 2.3: Record the time from the emission of a photon to its arrival at the detector, and calculate the more accurate information of each pixel position by averaging the time distribution of multiple photon events;

[0023] Further, the specific steps of step 3 are as follows:

[0024] Step 3.1: Filter the collected photon events to remove background noise and abnormal data to obtain a clearer initial image;

[0025] Step 3.2: Apply denoising algorithms such as average filtering or Gaussian filtering to the accumulated photon data to reduce the impact of noise and improve image clarity;

[0026] Step 3.3: Use image reconstruction algorithms to enhance resolution and detail on the collected photon data.

[0027] Further, the specific steps of step 4 are as follows:

[0028] Step 4.1: Single-photon cameras are based on a novel image sensor array unit that, unlike conventional frame rate cameras, does not capture continuous image frames at a fixed frame rate, but rather records brightness changes in the scene asynchronously at the pixel location;

[0029] Step 4.2: Each pixel independently detects the change in brightness, and a record is generated when the change exceeds a preset threshold, |V(x, t)-V ref (x)|>τ, the SPAD sensor generates a light-sensitive voltage V(x, t) after detecting photons: the deviation of this voltage from the reference voltage is V ref (x), after a single capture, the new voltage value is updated to the reference voltage V ref (x), provides a benchmark for subsequent detection. τ is the contrast threshold;

[0030] Step 4.3: The photon cube is composed of photon detection data captured by SPAD at a high frame rate. Each layer represents the photon detection result at a single moment. Since the random noise of single-frame data is large, in order to stabilize the brightness, the exponential moving average (EMA) is used to smooth the time frame data to obtain the current photon intensity estimate: Where β is the smoothing factor, B t (x) represents the photon detection result. The event triggering condition is as follows: |h(μ(x))-h(μ ref (x))|>τ, where h is a scalar function applied to the EMA to transform the estimated value of the photon intensity. A logarithmic or exponential function is usually chosen to adapt to the response characteristics of the sensor;

[0031] Step 4.4: The logarithmic function h(x) = log(x) is prone to underflow in low light conditions. For this purpose, the exponential response curve of the SPAD camera h(x) = 1-e is used. -ɑΦ(x，t) ,like Figure 2 As shown in the figure, ɑ is a constant independent of the photon flow, Φ(x, t) is the actual light intensity of pixel x at time t, and this response curve can avoid the underflow problem of the logarithmic function under low light conditions; the smoothing coefficient β and the contrast threshold τ are dynamically adjusted according to the pixel position or the incident light intensity to enhance the sensitivity of the camera and avoid the camera being unable to record due to insufficient brightness changes in low light, thereby achieving the purpose of optimizing the camera function.

[0032] The beneficial effects of the present invention are that through single-photon detection, time contrast projection technology, exponential response curve and optimized event triggering mechanism, the camera can still obtain high-quality imaging effects under low-light conditions, and has certain practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the process of single photon imaging.

[0034] Figure 2 is the nonlinear response curve of SPAD. DETAILED DESCRIPTION

[0035] The present invention provides a camera design optimization method based on photon density imaging, which is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] 1. Single-photon imaging

[0037] Single-photon imaging is a new type of sensitive imaging technology that constructs a mapping image of a scene by detecting a very small number of incident photons and performing photoelectric conversion. Single-photon detectors (such as single-photon avalanche diodes, SPADs) are used to capture photon information under low light intensity, and images are constructed through statistical methods and data processing. The specific working principle is as follows:

[0038] Photon detection: The single-photon imaging system uses a highly sensitive detector to convert the incident signal of a single photon into an electrical signal and capture incident photons of extremely low light intensity.

[0039] Time-resolved measurement: In time-resolved single-photon imaging, the detector records not only the spatial position of each photon, but also the arrival time of the photon. Based on the photon's flight time, the path length of the photon reflected from the light source is calculated to determine the depth information of the object's distance.

[0040] Statistical accumulation: Since only a few photons (even a single photon) can be captured each time in single-photon imaging, photons need to be accumulated in time intervals. The imaging system collects flux photons through time intervals, counts and reconstructs scene imaging. In this process, the pixel position accumulates enough photons in multiple time intervals to form a clear image.

[0041] Post-processing: The initial images generated by single-photon imaging are often sparse and require post-processing. Common post-processing methods include denoising, image reconstruction, and time correlation analysis.

[0042] Figure 1 The process of single photon imaging of the present invention is given. SPAD captures photon events at a high frame rate, and each pixel independently records the detection results of photons to form a two-dimensional image frame of a time series. Photon cubes are formed through accumulation in a time interval, and scene information is extracted through accumulation and time contrast projection operations. Real-time basic processing is completed on the chip to improve efficiency, and complex post-processing is performed outside the chip to achieve imaging effects.

[0043] 2. Photon Cube

[0044] A photon cube is a three-dimensional data structure designed to store single-photon camera detection data, which records the photon detection information of the sensor in time and space. It is a high-dimensional data volume consisting of a series of binary image frames captured by a single-photon detector in a continuous exposure cycle. It provides a flexible way to capture and process imaging data at the photon level. A photon cube usually has three dimensions:

[0045] Spatial dimension: used to capture the specific location where a photon is incident on a two-dimensional plane, corresponding to the two-dimensional pixel array of the imaging detector, where each pixel records the detection of a photon event at its location.

[0046] Time dimension: used to record the arrival time of photons, corresponding to continuous time intervals. In each time interval, all pixels will independently record the arrival of photons.

[0047] Detection value (0 or 1): used to record whether a photon event is detected at a specific time frame and pixel position. 0 indicates that no photon event is detected, and 1 indicates that a photon event is detected.

[0048] As a multi-dimensional data processing and imaging technology, Photon Cube provides higher resolution, accuracy and diverse analysis dimensions for the imaging process by combining time, space and detection information.

[0049] In summary, the present invention proposes a camera design optimization method based on single photon density imaging. The method is based on the projection method of the photon cube. By adopting an exponential response curve, exponential moving average smoothing, and adjusting the contrast threshold, the underflow problem of the camera under low light conditions is alleviated, the sensitivity of event generation is improved, and the camera function is optimized.

[0050] The above describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.

[0051] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

Claims

1. The present invention provides a camera design optimization method based on single photon density imaging. It belongs to the technical field of the combination of computer vision and computational photography. The technical solutions adopted are mainly as follows: Step 1: Use a single-photon avalanche diode (SPAD) cell array as an imaging device to capture photon data in the cube with high temporal resolution; Step 2: Collect single photon data and pre-process the captured photon data in the cube. Since the number of photons obtained per unit time is small, a longer period of photon accumulation is required to obtain enough sampled photons to construct an image. Step 3: Construct an image based on the detected single photon positions and arrival times, and use signal processing techniques to reduce noise and optimize image quality; Step 4: Project the data collected in the cube photons and use the time contrast for event detection to achieve event detection under sufficient lighting conditions.

2. The camera design optimization method based on single photon density imaging according to claim 1, characterized in that: The specific steps of step 1 are as follows: Step 1.1: Use SPAD to perform photoelectric conversion on the incident photons. As shown in Figure 1, the photons incident on the SPAD are absorbed by the semiconductor material, and the absorbed photons generate energy to generate an excited electron, converting the photon information into an electrical signal. Step 1.2: Set the size, cell pitch and frame rate (100kHz) of the SPAD array, calibrate the dark current of the SPAD cell array to reduce false triggering in the absence of light; adjust the SPAD gain to optimize the photon detection rate of the cell; Step 1.3: Synchronize the SPAD cell array with the external clock to achieve accurate frame rate control and set the exposure time of each cell to adapt to the lighting environment; Step 1.4: Exposure Cycle Each SPAD unit independently detects the incident photons and converts them into binary values ​​(0 or 1).

3. The camera design optimization method based on single photon density imaging according to claim 1, characterized in that: The specific steps of step 2 are as follows: Step 2.1: Start the light source and adjust the light intensity to the single-photon level, initialize the data and start data acquisition. The unit detector records the arrival time and position of each photon event; Step 2.2: Since the number of single photons captured in a short time is small, a long period of cumulative capture is required to ensure signal quality. The signal accumulation value of each unit detector is represented by the image grayscale value; Step 2.3: Record the time from when the photon is emitted to when it reaches the detector, and calculate the more accurate information of each pixel position by averaging the time distribution of multiple photon events.

4. The camera design optimization method based on single photon density imaging according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3.1: Filter the collected photon events to remove background noise and abnormal data to obtain a clearer initial image; Step 3.2: Apply denoising algorithms such as average filtering or Gaussian filtering to the accumulated photon data to reduce the impact of noise and improve image clarity; Step 3.3: Use image reconstruction algorithms to enhance resolution and detail on the collected photon data.

5. The camera design optimization method based on single photon density imaging according to claim 1, characterized in that: The specific steps of step 4 are as follows: Step 4.1: Single-photon cameras are based on a novel image sensor array unit that, unlike conventional frame rate cameras, does not capture continuous image frames at a fixed frame rate, but rather records brightness changes in the scene asynchronously at the pixel location; Step 4.2: Each pixel independently detects the change in brightness, and a record is generated when the change exceeds a preset threshold, |V(x, t)-V ref (x)|>τ, the SPAD sensor will generate a light-sensing voltage V(x, t) after detecting photons; the deviation of this voltage from the reference voltage is V ref (x), after a single capture, the new voltage value is updated to the reference voltage V ref (x), provides a benchmark for subsequent detection. τ is the contrast threshold; Step 4.3: The photon cube is composed of photon detection data captured by SPAD at a high frame rate. Each layer represents the photon detection result at a single moment. Since the random noise of single-frame data is large, in order to stabilize the brightness, the exponential moving average (EMA) is used to smooth the time frame data to obtain the current photon intensity estimate: Where β is the smoothing factor, B t (x) represents the photon detection result. The event triggering condition is as follows: |h(μ(x))-h(μ ref (x))|>τ, where h is a scalar function applied to the EMA to transform the estimated value of the photon intensity. A logarithmic or exponential function is usually chosen to adapt to the response characteristics of the sensor; Step 4.4: The logarithmic function h(x) = log(x) is prone to underflow in low light conditions. For this purpose, the exponential response curve of the SPAD camera h(x) = 1-e is used. -αФ(x,t) , as shown in Figure 2, where ɑ is a constant independent of the photon flow, Φ(x, t) is the actual light intensity of pixel x at time t, and this response curve can avoid the underflow problem of the logarithmic function under low light conditions; the smoothing coefficient β and the contrast threshold τ are dynamically adjusted according to the pixel position or the incident light intensity to enhance the sensitivity of the camera, avoiding the camera being unable to record due to insufficient brightness changes in low light, thereby achieving the purpose of optimizing the camera function.

Images