A photon shot noise calibration method for CMOS sensors

Abstract

The present invention discloses a method for calibrating the photon shot noise of a CMOS sensor. The method utilizes the illumination uniformity of a flat-field frame to approximate the true signal by an average value. After obtaining the system gain, a low-light image is acquired. The low-light image is subtracted from the black level and then divided by the system gain to convert the image into a photon count. A Poisson distribution is applied to the photon count and the photon count is multiplied by the system gain to restore the photon count to an image with photon shot noise, thereby simulating the generation of photon shot noise. The present invention designs an experimental apparatus consisting of a standardized 24-color card and a camera when acquiring the flat-field frame. Through the design of pixel blocks, the present invention extracts the signal values ​​of all channels of each pixel block in each image at each ISO, calculates the true signal value and the digital signal variance, and performs linear fitting on each channel to obtain the system gain of each channel at the current ISO. The system gain of each channel is summed and averaged as the system gain at the current ISO. The calibration method of the present invention is simple to operate, low-cost, accurate, and efficient.

Description

Technical Field

[0001] The present invention belongs to the field of camera parameter calibration, and in particular relates to a photon shot noise calibration method for a CMOS sensor. Background Art

[0002] Camera noise parameter calibration has important applications in computer vision and image processing. Despite the continuous advancement of CMOS image sensor technology and significant improvements in camera imaging quality, image noise remains difficult to avoid, especially in low-light conditions or at high sensitivity settings. The goal of camera noise parameter calibration is to accurately quantify the noise characteristics of camera sensors, including fixed pattern noise, readout noise, and photon noise, through experiments and modeling.

[0003] Noise calibration plays a key role in multiple applications. For example, in image denoising, noise parameters are an important basis for designing denoising algorithms, significantly improving denoising accuracy and fidelity. In photogrammetry and 3D reconstruction, understanding noise characteristics can enhance the robustness of feature extraction, matching, and depth estimation. In high-precision fields such as medical imaging and astronomical observation, noise calibration helps optimize signal processing algorithms and improve image quality.

[0004] During the calibration process, photon shot noise is often calibrated using flat-field frames. Flat-field frames are images with uniform illumination intensity, typically obtained using a D65 standard light source and an integrating sphere. This method uses an adjustable power supply to control the light source brightness, simulating varying intensities of daylight, and provides uniform diffuse light through the light outlet of the integrating sphere. If parallel light at infinite distance is required, a collimator can be used to generate a parallel beam, and the camera can be focused at infinity to ensure uniform illumination. If specialized equipment is lacking, a white paper taped to a uniform wall can be used as an alternative to capture the image under uniform lighting conditions.

[0005] In contrast, the first method, while accurate, is complex and costly, and relies heavily on the experimental environment and equipment, limiting its flexibility. The second method is simple and inexpensive, but the diffuse reflective properties of white paper are easily affected by ambient light, potentially reducing the accuracy of the experimental results. Therefore, it is necessary to optimize existing solutions and develop more flexible, efficient, and cost-effective solutions for photon shot noise calibration. Summary of the Invention

[0006] The purpose of the present invention is to address the shortcomings of existing photon shot noise calibration schemes and propose a photon shot noise calibration method that is simple to operate, low-cost, accurate and efficient.

[0007] The present invention solves the problems of existing photon shot noise calibration solutions by the following approaches: a photon shot noise calibration method for a CMOS sensor, the method comprising:

[0008] Photons are incident on the image point through the lens and converted into electrons through photoelectric reaction. The electrons are then converted into voltage through capacitors. After system gain amplification and A / D conversion, digital signal output is generated. The digital signal output is expressed by the following formula:

[0009] D=KI+N c

[0010] Where D is the Raw format image output by the CMOS image sensor, I is the number of photons captured by the image sensor in a single imaging process, K is the system gain, and N c is the sum of noise sources caused by the physical model of the image sensor;

[0011] According to the principles of quantum mechanics, the photon shot noise N p It obeys the Poisson distribution P(·), which is expressed as:

[0012] N p +I~P(I)

[0013] According to the expression of digital signal output and the characteristic that photon shot noise conforms to Poisson distribution, the digital signal image output by the CMOS image sensor is expressed as:

[0014] D=K(I+N p )+N

[0015] Where N represents the sum of other noise sources unrelated to the light signal, N+N p =N c ; Express the noise variance as:

[0016] Var(D)=K 2 Var(I+N p )+Var(N)

[0017] Where Var(·) represents the variance operator; since I+N p It follows a Poisson distribution with a variance equal to its mean, so:

[0018] Var(D)=K 2 I + Var(N) = K(KI) + Var(N)

[0019] The above formula shows that there is a linear relationship between the digital signal variance Var(D) and the true signal KI. By utilizing the illumination uniformity of the flat field frame, the true signal KI is approximated by the average value.

[0020] After calculating the system gain K, the camera is used to capture images under low-light conditions to obtain a low-light image. The black level is subtracted from the low-light image and the image is divided by the system gain K to convert it into the number of photons I. A Poisson distribution is applied to the number of photons I, and the image is then multiplied by the system gain K to restore the number of photons I to an image with photon shot noise. This simulates the generation of photon shot noise and achieves photon shot noise calibration.

[0021] Furthermore, flat-field frames were obtained using an experimental setup consisting of a standardized 24-color card and a camera. During measurement, photography was performed under ordinary fluorescent light or during the day. Six or more different ISO values ​​were selected during photography, and several images were taken continuously at each ISO value. During the data processing stage, the fitting effect was judged by the fitting coefficient.

[0022] Furthermore, the ISO value is specifically selected as follows: selecting ISO according to linear growth; or selecting ISO according to exponential growth; or selecting ISO according to linear growth / exponential growth in different intervals.

[0023] Furthermore, the positions of the standardized 24 color cards are kept unchanged when adjusting ISO, so that 24 different sets of data points can be more easily obtained during subsequent data processing.

[0024] Furthermore, the flat-field frames in Raw format collected by the experimental device are converted into PNG format through the ISP process to obtain the pixel data of each color block of the standardized 24-color card, specifically:

[0025] Several square color blocks are arranged in an array on a black background of a standardized 24-color card. First, a rectangular window (i.e., the pixel block size) is set. Then, a mouse event callback function in the OpenCV package is used to select one pixel in each color block on the flat-field frame as the top-left corner of the pixel block. The rectangle function in the OpenCV package is then used to generate the rectangular coordinates of the 24 pixel blocks. These rectangular coordinates include four parameters: the horizontal coordinate of the top-left corner of the rectangle, the vertical coordinate of the top-left corner of the rectangle, the width of the rectangle, and the length of the rectangle. The signal values ​​of all channels of each pixel block in each image are extracted at each ISO.

[0026] For each pixel block, the true signal value KI and the digital signal variance Var(D) are calculated, and then a linear fit is performed on (Var(D),KI) for each channel. The slope of the fitting line is the system gain K of the channel at the current ISO of the camera. c , the system gain K of each channel c The sum is taken as the average value as the system gain K at the current ISO.

[0027] Furthermore, for each pixel block, the true signal value KI is calculated as follows: the signal of each channel in each image is averaged within the range of the pixel block, and the average value of all images at the current ISO within the range of the pixel block is accumulated, and then divided by the total number of images at the current ISO.

[0028] Furthermore, for each pixel block, the digital signal variance Var(D) is calculated as follows: the variance of the signal of each channel in each image within the range of the pixel block is calculated, and the variance of all images at the current ISO within the range of the pixel block is accumulated, and then divided by the total number of images at the current ISO.

[0029] Furthermore, the black base plate of the standardized 24-color card has a size of 20 cm*14 cm, and square color blocks are arranged in an array of 4 horizontal columns and 6 vertical columns. The size of each color block is 2 cm*2 cm, and the pixel block size is set to 25 pixels*25 pixels.

[0030] The beneficial effects of the present invention are as follows: The present invention utilizes the illumination uniformity of the flat-field frame to approximate the true signal by the average value; when acquiring the flat-field frame, an experimental device consisting of a standardized 24-color chart and a camera is designed. The 24-color chart covers 24 standardized colors and brightnesses, from neutral colors to saturated colors, covering a wide range of signal strengths, providing 24 sets of high-quality data points for the experiment, effectively reducing the risk of data overfitting. In addition, the high-precision design of the standardized color chart avoids the reflection unevenness and experimental errors that may be caused by non-standardized white paper, ensuring the accuracy and repeatability of the experiment. When shooting, 6 or more different ISO values ​​are selected, and several identical images are taken continuously at each ISO value to avoid data overfitting. In the data processing stage, the fitting effect is judged by the fitting coefficient. After the flat-field frame is collected in Raw format, it is converted to PNG format. The pixel point data of each color block of the standardized 24-color chart is obtained through the relevant functions of the OpenCV package. The pixel block size is set, and the signal values ​​of all channels of each pixel block in each image are extracted at each ISO. By averaging the true signal value and the digital signal variance, and performing a linear fit on each channel, the slope of the fitted line represents the system gain of each channel at the camera's current ISO. The system gain of each channel is then summed and averaged to provide the system gain at the current ISO, thereby achieving photon shot noise calibration. The photon shot noise calibration method provided by this invention is simple to operate, low-cost, accurate, and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a physical model of a CMOS image sensor provided by an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of a photon shot noise calibration experimental device provided in an embodiment of the present invention;

[0033] Figure 3-Figure 6 These are the measurement results of each channel of Canon EOS200D2 at ISO=1600;

[0034] Figure 7-10 These are the measurement results of each channel of Canon EOS200D2 at ISO=3200;

[0035] Figure 11-14 These are the measurement results of each channel of Panasonic LUMIX S5 at ISO=1600;

[0036] Figures 15-18 These are the measurement results of each channel of Panasonic LUMIX S5 at ISO=3200. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0039] This embodiment provides a method for calibrating the photon shot noise of a CMOS sensor, and the implementation steps are as follows:

[0040] Figure 1 This is the physical model of a CMOS image sensor. As shown in the diagram, photons pass through the lens and strike the image point within a certain timeframe. These photons are converted into a certain number of electrons through a photoelectric reaction. These electrons are then converted into voltage by a capacitor. This voltage is then amplified by the system gain and converted to an analog-to-digital converter (A / D). Therefore, this digital signal output can be expressed as follows:

[0041] D=KI+N c

[0042] Where D is the Raw format image output by the CMOS image sensor, I is the number of photons captured by the image sensor in a single imaging process, K is the total gain of the entire system, and N c is the sum of noise sources caused by the physical model of the image sensor.

[0043] During the exposure process, incident light strikes the pixel area of ​​the light sensor in the form of photons, and according to the photoelectric effect, photoelectrons proportional to the light intensity are released. Due to the quantum nature of light, there is an inevitable uncertainty in the number of electrons collected in the pixel area. This uncertainty leads to the photon shot noise N pAccording to the principles of quantum mechanics, photon shot noise obeys the Poisson distribution:

[0044] N p +I~P(I)

[0045] where P(·) represents the Poisson distribution and I is the number of photons captured by the image sensor in a single imaging process.

[0046] According to the expression of digital signal output and the characteristic that photon shot noise conforms to Poisson distribution, the digital signal image output by the CMOS image sensor can be expressed as:

[0047] D=K(I+N p )+N

[0048] Where N represents the sum of other noise sources unrelated to the light signal, N+N p =N c .

[0049] Because I+N p It conforms to the Poisson distribution, so the corresponding noise variance can be expressed as:

[0050] Var(D)=K 2 Var(I+N p )+Var(N)

[0051] Where Var(·) represents the variance operator, which is used to calculate the variance of a given random variable. p It follows a Poisson distribution with a variance equal to its mean, so:

[0052] Var(D)=K 2 I + Var(N) = K(KI) + Var(N)

[0053] The above equation shows that there is a linear relationship between the digital signal variance Var(D) and the true signal KI. Since the true signal is difficult to obtain, we can use the illumination uniformity of the flat field frame to approximate the true signal KI by the average value.

[0054] The experimental device proposed in the present invention can be used when acquiring the flat field frame. The experimental device consists of a standardized 24-color card and a camera. The 24-color card covers 24 standardized colors and brightnesses, from neutral colors to saturated colors, covering a wide range of signal strengths, providing 24 sets of high-quality data points for the experiment, effectively reducing the risk of data overfitting. In addition, the high-precision design of the standardized color card avoids the reflection unevenness and experimental errors that may be caused by non-standardized white paper, ensuring the accuracy and repeatability of the experiment. The specific schematic diagram is as follows Figure 2 shown.

[0055] During measurement, capture images under standard fluorescent lighting or during the day. Images should be kept bright to prevent the introduction of noise. Select six or more different ISO values, and capture at least 20 consecutive images at each ISO value to avoid overfitting. Avoid large differences in ISO values, as this can lead to overfitting or poor fitting. Alternatively, select ISO values ​​in a linear progression. For example, in increments of 400, select values ​​such as 800, 1200, 1600, and 2000. Alternatively, select ISO values ​​in an exponential progression within the lower ISO range and then in a linear progression within the higher ISO range. For example, select 800, 1600, 3200, 6400, and 12800, and then select ISO values ​​at regular intervals thereafter to prevent overfitting. To simplify the experimental procedure, keep the color chart position unchanged when adjusting ISO. This will facilitate the acquisition of 24 different data points during subsequent data processing. During data processing, assess the fitting performance using the fitting coefficients. If the results are poor, repeat the above method to collect data.

[0056] Using the above method, we captured flat-field frames using the Canon EOS200D2 and Panasonic LUMIX S5 as examples. For the Canon EOS200D2, we selected an ISO range of 400 to 12800, with an exponential increment of 2. For the Panasonic LUMIX S5, we selected an ISO range of 640 to 6400, with a linear increment of 160. The acquired Raw flat-field frames were converted to PNG format through the ISP process. The pixel data for each block of a standardized 24-color palette was obtained using OpenCV functions. Specifically, the standardized 24-color palette consists of a black background measuring 20cm by 14cm, with four horizontal and six vertical columns of color blocks arranged in an array. Each block is 2cm by 2cm square. To select pixels, a rectangular window is set to a 25x25 pixel block size. The EVENT_LBUTTONDOWN mouse event callback function in OpenCV is then used to select a pixel within each block in the flat-field frame as the top-left corner of the block. The rectangle function in OpenCV, called rectangle, is then used to generate the coordinates of each 24-pixel block. These coordinates take four parameters: the horizontal coordinate of the top-left corner, the vertical coordinate of the top-left corner, the width of the rectangle, and the length of the rectangle. The signal values ​​for all channels of each pixel block in each image are then extracted at each ISO.

[0057] For each pixel block, the following values ​​are calculated:

[0058] 1. True signal value KI: The signal of each channel in each image is averaged within the pixel block range, and the average value of all images at the current ISO within the pixel block range is accumulated, and then divided by the total number of images at the current ISO.

[0059] 2. Digital signal variance Var(D): The variance of the signal of each channel in each image is calculated within the range of the pixel block. The variance of all images at the current ISO within the range of the pixel block is accumulated and then divided by the total number of images at the current ISO.

[0060] Next, a linear fit is performed on (Var(D),KI) of each channel using the linear_regression function in Numpy. The slope of the fitted line is the system gain K of the channel at the current ISO of the camera. c , the system gain K of each channel c The sum is taken as the average value as the system gain K at the current ISO.

[0061] Figure 3-6 These are the measurement results of each channel of Canon EOS200D2 at ISO=1600. Figure 7-10 The measurement results of each channel at ISO=3200. Figure 11-14 These are the measurement results of each channel of Panasonic LUMIX S5 at ISO=1600. Figure 15-18 The results are for each channel when ISO=3200. Goodness of Fit refers to the degree of fit of the regression line to the observed values. The statistic used to measure goodness of fit is the coefficient of determination (also known as the coefficient of determination) R. 2 , R 2 The maximum value is 1, R 2 The closer the value of R is to 1, the better the regression line fits the observed value; on the contrary, 2 The smaller the value, the worse the regression line fits the observed values. Finally, we can find that the system gain K of the Canon EOS200D2 is 7.0444 at ISO = 1600 and 14.0768 at ISO = 3200. The system gain K of the Panasonic LUMIX S5 is 2.6474 at ISO = 1600 and 5.3162 at ISO = 3200.

[0062] After the value of the system gain K is determined, the camera is used to capture images under low-light conditions to obtain a low-light image. The black level (the offset value generated by the CMOS image sensor when no light signal is received) is subtracted from the low-light image and the image is divided by the value of the system gain K to convert it into the number of photons I. A Poisson distribution is then applied to the number of photons I, and finally, the number of photons I is multiplied by the system gain K to restore it to an image with photon shot noise. This simulates the generation of photon shot noise, thereby completing the entire photon shot noise calibration process.

[0063] The above description is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present invention without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A method for calibrating photon shot noise of a CMOS sensor, characterized in that: include: Photons are incident on the image point through the lens and converted into electrons through photoelectric reaction. The electrons are then converted into voltage through capacitors. After system gain amplification and A / D conversion, digital signal output is generated. The digital signal output is expressed by the following formula: D=KI+N c Where D is the Raw format image output by the CMOS image sensor, I is the number of photons captured by the image sensor in a single imaging process, K is the system gain, and N c is the sum of noise sources caused by the physical model of the image sensor; According to the principles of quantum mechanics, the photon shot noise N p It obeys the Poisson distribution P(·), which is expressed as: N p +I~P(I) According to the expression of digital signal output and the characteristic that photon shot noise conforms to Poisson distribution, the digital signal image output by the CMOS image sensor is expressed as: D=K(I+N p )+N Where N represents the sum of other noise sources unrelated to the light signal, N+N p =N c ; Express the noise variance as: Yes(D)=K 2 ·Yes(I+N p )+Yes(N) Where Var(·) represents the variance operator; since I+N p It follows a Poisson distribution with a variance equal to its mean, so: Yes(D)=K 2 I+Var(N)=K(KI)+Var(N) The above formula shows that there is a linear relationship between the digital signal variance Var(D) and the true signal KI. By utilizing the illumination uniformity of the flat field frame, the true signal KI is approximated by the average value. After calculating the system gain K, the camera is used to capture images under low-light conditions to obtain a low-light image. The black level is subtracted from the low-light image and the image is divided by the system gain K to convert it into the number of photons I. A Poisson distribution is applied to the number of photons I, and the image is then multiplied by the system gain K to restore the number of photons I to an image with photon shot noise. This simulates the generation of photon shot noise and achieves photon shot noise calibration.

2. The photon shot noise calibration method for a CMOS sensor according to claim 1, wherein: Flat-field frames were acquired using an experimental setup consisting of a standardized 24-color chart and a camera. Measurements were taken under ordinary fluorescent light or during the day. Six or more different ISO values ​​were selected, and several images were taken continuously at each ISO value. During the data processing stage, the fitting effect was evaluated using the fitting coefficients.

3. The photon shot noise calibration method for a CMOS sensor according to claim 2, characterized in that: The ISO value selection is specifically: selecting ISO according to linear growth; or selecting ISO according to exponential growth; or selecting ISO according to linear growth / exponential growth in different intervals.

4. The photon shot noise calibration method for a CMOS sensor according to claim 2, wherein: Keeping the position of the standardized 24-color chart unchanged when adjusting ISO makes it easier to obtain 24 different sets of data points during subsequent data processing.

5. The photon shot noise calibration method for a CMOS sensor according to claim 2, wherein: The flat-field frames in Raw format collected by the experimental device are converted to PNG format through the ISP process to obtain the pixel data of each color block of the standardized 24-color card, specifically: Several square color blocks are arranged in an array on a black background of a standardized 24-color card. First, a rectangular window (i.e., the pixel block size) is set. Then, a mouse event callback function in the OpenCV package is used to select one pixel in each color block on the flat-field frame as the top-left corner of the pixel block. The rectangle function in the OpenCV package is then used to generate the rectangular coordinates of the 24 pixel blocks. These rectangular coordinates include four parameters: the horizontal coordinate of the top-left corner of the rectangle, the vertical coordinate of the top-left corner of the rectangle, the width of the rectangle, and the length of the rectangle. The signal values ​​of all channels of each pixel block in each image are extracted at each ISO. For each pixel block, the true signal value KI and the digital signal variance Var(D) are calculated, and then a linear fit is performed on (Var(D),KI) for each channel. The slope of the fitting line is the system gain K of the channel at the current ISO of the camera. c , the system gain K of each channel c The sum is taken as the average value as the system gain K at the current ISO.

6. The photon shot noise calibration method for a CMOS sensor according to claim 5, characterized in that: For each pixel block, the true signal value KI is calculated as follows: the signal of each channel in each image is averaged within the pixel block range, and the average value of all images at the current ISO within the pixel block range is accumulated, and then divided by the total number of images at the current ISO.

7. The photon shot noise calibration method for a CMOS sensor according to claim 5, wherein: For each pixel block, the digital signal variance Var(D) is calculated as follows: the variance of the signal of each channel in each image within the range of the pixel block is calculated, and the variance of all images at the current ISO within the range of the pixel block is accumulated, and then divided by the total number of images at the current ISO.

8. The photon shot noise calibration method for a CMOS sensor according to claim 5, wherein: The black base plate of the standardized 24-color card has a size of 20cm*14cm, and square color blocks are arranged in an array with 4 horizontal and 6 vertical columns. The size of each color block is 2cm*2cm, and the pixel block size is set to 25 pixels*25 pixels.

Images