A noise modeling and denoising method for low-light-level image intensifier

Abstract

The application provides a noise modeling and denoising method for a low-light image intensifier, comprising the following steps: step 1, establishing an imaging noise model for the low-light image intensifier; step 2, collecting output data of the low-light image intensifier under different illumination conditions, calibrating parameters of the imaging noise model, and superimposing the calibrated imaging noise model on a noise-free image sample to construct a training data set conforming to actual imaging characteristics; step 3, designing an image denoising network based on a convolutional neural network, wherein the network input is a noisy image, and the output is a clean image with noise suppressed; and step 4, training and optimizing the convolutional neural network based on the training data set, and verifying on synthetic data and real data. The method can effectively enhance the adaptability and generalization performance of the neural network to complex noise, and ultimately help to realize more accurate and stable image enhancement and denoising effect in the low-light imaging scene.

Description

Technical Field

[0001] This invention belongs to the field of noise modeling and denoising technology for low-light image intensifiers, and particularly relates to a noise modeling and denoising method for low-light image intensifiers. Background Technology

[0002] In many high-tech fields such as national defense, astronomical observation, and cutting-edge biomedicine, the ability to detect and image extremely weak light signals with high sensitivity is crucial. In these scenarios, the incident photon flux is extremely low, and the effective signal is often on the same order of magnitude as or even lower than the system noise. Therefore, researchers have begun to study low-light image intensifiers to address this problem.

[0003] Currently, standard CMOS (Complementary Metal-Oxide-Semiconductor) or CCD (Charge-coupled Device) digital image sensors are the most widely used imaging devices. However, in extremely low-light environments with very few photons, the incident light signal is extremely weak, and the number of photoelectrons generated is even less than the number of noise electrons in the readout circuit itself. This causes the effective signal to be overwhelmed by strong readout noise, resulting in an extremely low signal-to-noise ratio, or even failure to image. To overcome this limitation, researchers have proposed a low-light image intensifier technology based on a microchannel plate (MCP). In this way, the low-light image intensifier amplifies the original weak signal by tens of thousands of times, making its intensity far exceed the readout noise of the back-end sensor, thus achieving effective imaging under extremely dark conditions. However, under extremely low light conditions, unlike the classical noise model, the low-light image intensifier introduces new and more complex noise sources during photocathode conversion, electron multiplication of the MCP, and bombardment of the phosphor screen. Therefore, accurately establishing a noise model is the key to low-light image denoising.

[0004] In recent years, deep learning methods, represented by convolutional neural networks, have brought revolutionary breakthroughs to image denoising. Unlike traditional methods that rely on manually designed prior models, deep learning models can autonomously learn complex nonlinear mappings from noisy images to clean images from large-scale datasets in an end-to-end manner. Due to the data requirements of supervised learning, training samples need noise-free ground truth images corresponding to the noisy images as labels. However, it is difficult to directly obtain noise-free labeled images in real-world scenarios. Therefore, synthesizing training datasets by building accurate noise models can effectively drive deep learning models to accurately denoise images.

[0005] Existing technologies mostly use simplified assumptions such as a single Gaussian to describe the noise of low-light image intensifiers, which makes it difficult to accurately characterize the noise of multiple devices such as photocathode shot, microchannel plate gain fluctuations, fluorescent screen and CMOS; and lack traceable calibration procedures and high-fidelity training data, resulting in insufficient generalization and poor repeatability of denoising networks. Summary of the Invention

[0006] Objective of the Invention: The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a noise modeling and denoising method for low-light image intensifiers, comprising the following steps:

[0007] Step 1: Establish an imaging noise model for the low-light image intensifier. The imaging noise model comprehensively considers the dark current noise of the photocathode, the noise of the microchannel plate, the noise of the fluorescent screen, and the noise of the CMOS (Complementary Metal-Oxide-Semiconductor) sensor.

[0008] Step 2: Collect the output data of the low-light image intensifier under different lighting conditions, calibrate the parameters of the imaging noise model, and superimpose the calibrated imaging noise model on the noise-free image samples to construct a training dataset that conforms to the actual imaging characteristics.

[0009] Step 3: Design an image denoising network based on a convolutional neural network. The network input is a noisy image, and the output is a clean image with the noise suppressed.

[0010] Step 4: Train and optimize the convolutional neural network based on the training dataset, and validate it on synthetic and real data.

[0011] In step 1, the dark current noise of the photocathode is modeled in the following way:

[0012] Exposure time Noise model of photocathode Represented as:

[0013] ,

[0014] in, This indicates that the photon shot noise caused by quantum fluctuations of incident photons follows a Poisson distribution:

[0015] ,

[0016] ,

[0017] in Indicates the Poisson distribution. The expected number of total photons reaching the sensor during the exposure time. Photon flux The quantum efficiency of the photocathode;

[0018] The dark current noise of the photocathode follows a Poisson distribution:

[0019] ,

[0020] in, denoted as the expected count rate per unit time of the dark current of the photocathode.

[0021] In step 1, the noise of the microchannel plate is modeled in the following way:

[0022] ,

[0023] in, Here is the noise model for the microchannel plate, where K is the gain coefficient of the microchannel plate. It is the relative gain deviation. It follows a zero-mean Gaussian distribution , It is the standard deviation that characterizes the gain non-uniformity between channels of a microchannel plate.

[0024] In step 1, the screen noise is modeled as follows:

[0025] Photon count per pixel during the exposure time Represented as:

[0026] ,

[0027] in, The average photon yield of the fluorescent screen. The perturbation of luminous efficiency non-uniformity satisfies a zero-mean Gaussian distribution. , The standard deviation represents the non-uniformity of the luminous efficiency of the fluorescent screen.

[0028] In step 1, the noise of the CMOS sensor is modeled as follows:

[0029] ,

[0030] in, It is a noise model for CMOS sensors. The signal noise generated by the CMOS sensor receiving photons from the fluorescent screen follows a Poisson distribution. R follows a zero-mean Gaussian distribution. Readout noise, This represents the standard deviation of the readout noise; It conforms to the Poisson distribution CMOS dark current noise, This is the expectation of CMOS dark current noise.

[0031] In step 1, the imaging noise model Represented as:

[0032] ,

[0033] .

[0034] Step 2 includes: calibrating the model parameters of the imaging noise model using an integrating sphere, and then superimposing the calibrated imaging noise model onto noise-free image samples to construct a training dataset that conforms to the actual imaging rules. The training dataset includes noisy images and real images (clean images), which can provide high-quality data support for neural network training.

[0035] In step 3, the convolutional neural network is UNet, and the output of the UNet convolutional neural network is... The convolutional neural network UNet references the UNet network architecture in the paper "Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[C] / / International Conference on Medical image computing and computer-assisted intervention. Cham: Springer international publishing, 2015: 234-241". This UNet architecture employs a symmetrical encoder-decoder architecture. The encoder extracts multi-scale features of the image layer by layer through downsampling and convolution, while the decoder restores the image resolution layer by layer through upsampling and convolution. Feature stitching is achieved between each encoder and decoder layer via skip connections. The network input is a noisy image, and the output is a clean image with suppressed noise. .

[0036] In step 4, the output of the convolutional neural network UNet... ,calculate Compared with real images in the training dataset loss function between :

[0037] ,

[0038] in, The mean absolute error function, For structural similarity functions, For the perceptual similarity function, and This is the balance coefficient of the loss function.

[0039] Finally, by minimizing the loss function To train and optimize the convolutional neural network UNet.

[0040] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0041] This invention proposes a parameterizable and calibrable accurate noise model covering the entire chain from photocathode to microchannel plate to fluorescent screen to CMOS, along with a noise parameter calibration method. Based on this model, large-scale, high-fidelity synthetic datasets can be generated, thereby effectively driving deep networks to achieve higher-precision denoising.

[0042] Beneficial effects: This method deeply analyzes the potential noise sources of different hardware modules in low-light image intensifiers and constructs an accurate noise model. This model not only provides key support for the synthesis of large-scale high-fidelity datasets, but also effectively enhances the adaptability and generalization performance of neural networks to complex noise, ultimately helping to achieve more accurate and stable image enhancement and denoising effects in low-light imaging scenarios. Attached Figure Description

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0044] Figure 1 This is a flowchart of the method of the present invention.

[0045] Figure 2 This is a diagram of a low-light image intensifier.

[0046] Figure 3 The images show a noisy real-world image captured by the low-light image intensifier (left) and a denoised image obtained using the present invention (right). Detailed Implementation

[0047] Reference Figure 1 This invention provides a noise modeling and denoising method for low-light image intensifiers, comprising the following steps:

[0048] Step 1: Establish an imaging noise model suitable for low-light image intensifiers. Unlike traditional CMOS cameras, which are mainly affected by readout noise, dark current noise, and photon shot noise, low-light image intensifiers exhibit more complex noise characteristics during imaging due to their unique hardware structure. For example... Figure 2 As shown, the low-light image intensifier not only includes a CMOS sensor, but also an image enhancement component consisting of a photocathode, a microchannel plate (MCP), and a fluorescent screen, any of which may introduce additional noise. Based on the physical imaging mechanism of the low-light image intensifier, this invention systematically models the noise models of each functional device and finally establishes a complete and realistic link-level noise model.

[0049] (1) Photocathode Dark Current Noise: In a low-light image intensifier, the main function of the photocathode is to convert incident photons into electrons. Its noise mainly comes from photon shot noise and its own dark current noise. During exposure time... Internal, single-pixel electronic counting This can be expressed as the following formula:

[0050] ,

[0051] in, The photon shot noise is caused by quantum fluctuations of incident photons, and it follows a Poisson distribution:

[0052] ,

[0053] Indicates the Poisson distribution. This represents the photon flux. Let be the quantum efficiency of the photocathode (which is a constant).

[0054] The dark current noise of the photocathode also follows a Poisson distribution:

[0055] ,

[0056] in, denoted as the expected count rate per unit time of the dark current of the photocathode.

[0057] (2) Microchannel plate noise: The microchannel plate is used to cascade and multiply the primary electrons output from the photocathode, and is the core gain unit of the low-light image intensifier. In engineering, the main noise can be characterized as the fixed multiplicative bias caused by the non-uniform gain between channels. The following noise model is established:

[0058] ,

[0059] Where K is the gain coefficient of the microchannel plate. It is the relative gain deviation, which conforms to a zero-mean Gaussian distribution. , It is the standard deviation that characterizes the gain non-uniformity between channels of a microchannel plate.

[0060] (3) Fluorescent Screen Noise: In low-light image intensifiers, the main function of the fluorescent screen is to convert electrons output from the microchannel plate into visible photons; its noise mainly originates from the non-uniformity of the fluorescent screen's luminous efficiency. During the exposure time, the photon count per unit pixel... Represented as:

[0061] ,

[0062] in, The average photon yield of the fluorescent screen (the average number of photons produced per electron). The non-uniformity perturbation of luminous efficiency follows a zero-mean Gaussian distribution, denoted as . , The standard deviation represents the non-uniformity of the luminous efficiency of the fluorescent screen.

[0063] (4) Traditional CMOS noise: CMOS receives photons from the phosphor screen and completes readout. Its noise mainly comes from photon shot noise, dark current noise, and readout noise, and is modeled as follows:

[0064] ,

[0065] in, The signal noise generated by the CMOS sensor receiving photons from the fluorescent screen follows a Poisson distribution. ; It is readout noise that follows a zero-mean Gaussian distribution. This represents the standard deviation of the readout noise; It is CMOS dark current noise that conforms to a Poisson distribution. This is the expectation of CMOS dark current noise.

[0066] In summary, the noise model of the low-light image intensifier proposed in this invention is expressed as follows:

[0067] .

[0068] Step 2: Calibrate the noise model parameters based on the actual low-light image intensifier. First, acquire multiple exposures of dark frames under complete illumination. In this case, with the intensifier processing off, the CMOS readout noise R and dark current noise can be estimated. Subsequently, with the intensifier on but the light blocked, the dark current of the photocathode was estimated. Then, using an integrating sphere to create a uniform flat field, the luminous flux and the high voltage of the microchannel plate are varied: the quantum efficiency of the photocathode is obtained by fitting the mean and variance. Pixel normalization is performed on the flat-field stack to calculate the inter-channel gain inhomogeneity of the microchannel plate. With phosphor screen luminescence non-uniformity Finally, the luminance reference was used for calculation. The value of .

[0069] Next, the calibrated noise model is superimposed onto noise-free image samples to construct a training dataset that conforms to actual imaging patterns. This training dataset includes noisy images and real images (clean images), providing high-quality data support for neural network training.

[0070] Step 3: Design an image denoising network based on the convolutional neural network UNet. The network input is a noisy image and the output is a clean image with the noise suppressed.

[0071] Step 4, for the output of the convolutional neural network UNet Calculate its comparison with real images in the training dataset. loss function between :

[0072] ,

[0073] in, The mean absolute error function, For structural similarity functions, For the perceptual similarity function, and This is the balance coefficient of the loss function.

[0074] Finally, by minimizing the loss function To train and optimize neural networks.

[0075] During network training, noisy images were used as input, with a batch size of 20. The ADAM optimizer was employed, and the learning rate was set to 0.001. The learning rate was halved every 10 training epochs. The network output consisted of clean, noise-free images, and the training process was repeated for a total of 200 epochs. Finally, the network performance was tested on a synthetic test set and real-world images.

[0076] Due to the complex hardware structure and diverse noise sources of low-light image intensifiers, current denoising methods face a series of problems, such as limited available datasets and poor denoising performance. This invention constructs an accurate noise model for the low-light image intensifier, which not only enables the synthesis of large-scale training datasets but also makes the noise features learned by the network more closely resemble real imaging scenes. Ultimately, the network achieves state-of-the-art performance on real datasets. Figure 3The actual noise reduction effect of the present invention is demonstrated. Figure 3 The image on the left is an actual image captured by the low-light image intensifier, which contains a lot of noise. Figure 3 The image on the right is the image after noise reduction according to the present invention, which shows that the noise has been effectively suppressed.

[0077] This invention provides a noise modeling and denoising method for low-light image intensifiers. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A noise modeling and denoising method for low-light image intensifiers, characterized in that, Includes the following steps: Step 1: Establish an imaging noise model for the low-light image intensifier. The imaging noise model comprehensively considers the dark current noise of the photocathode, the noise of the microchannel plate, the noise of the fluorescent screen, and the noise of the CMOS sensor. Step 2: Collect the output data of the low-light image intensifier under different lighting conditions, calibrate the parameters of the imaging noise model, and superimpose the calibrated imaging noise model on the noise-free image samples to construct a training dataset that conforms to the actual imaging characteristics. Step 3: Design an image denoising network based on a convolutional neural network. The network input is a noisy image, and the output is a clean image with the noise suppressed. Step 4: Train and optimize the convolutional neural network based on the training dataset, and verify it on synthetic and real data; In step 1, the dark current noise of the photocathode is modeled in the following way: Exposure time Noise model of photocathode Represented as: ， in, This indicates that the photon shot noise caused by quantum fluctuations of incident photons follows a Poisson distribution: ， ， in Indicates the Poisson distribution. The expected number of total photons reaching the sensor during the exposure time. Photon flux The quantum efficiency of the photocathode; The dark current noise of the photocathode follows a Poisson distribution: ， in, denoted as the expected count rate per unit time of the dark current of the photocathode.

2. The method according to claim 1, characterized in that, In step 1, the noise of the microchannel plate is modeled in the following way: ， in, Here is the noise model for the microchannel plate, where K is the gain coefficient of the microchannel plate. It is the relative gain deviation. It follows a zero-mean Gaussian distribution , It is the standard deviation that characterizes the gain non-uniformity between channels of a microchannel plate.

3. The method according to claim 2, characterized in that, In step 1, the screen noise is modeled as follows: Photon count per pixel during the exposure time Represented as: ， in, The average photon yield of the fluorescent screen. The perturbation of luminous efficiency non-uniformity satisfies a zero-mean Gaussian distribution. , The standard deviation represents the non-uniformity of the luminous efficiency of the fluorescent screen.

4. The method according to claim 3, characterized in that, In step 1, the noise of the CMOS sensor is modeled as follows: ， in, It is a noise model for CMOS sensors. The signal noise generated by the CMOS sensor receiving photons from the fluorescent screen follows a Poisson distribution. R follows a zero-mean Gaussian distribution. Readout noise, This represents the standard deviation of the readout noise; It conforms to the Poisson distribution CMOS dark current noise, This is the expectation of CMOS dark current noise.

5. The method according to claim 4, characterized in that, In step 1, the imaging noise model Represented as: ， 。 6. The method according to claim 5, characterized in that, Step 2 includes: calibrating the model parameters of the imaging noise model using an integrating sphere, and then superimposing the calibrated imaging noise model onto a noise-free image sample to construct a training dataset that conforms to the actual imaging rules. The training dataset includes noisy images and real images.

7. The method according to claim 6, characterized in that, In step 3, the convolutional neural network is UNet, and the output of the UNet convolutional neural network is... .

8. The method according to claim 7, characterized in that, In step 4, the output of the convolutional neural network UNet... ,calculate Compared with real images in the training dataset loss function between : ， in, The mean absolute error function, For structural similarity functions, For the perceptual similarity function, and These are the balance coefficients of the loss function; Finally, by minimizing the loss function To train and optimize the convolutional neural network UNet.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images