Single-photon image noise modeling enhancement method and system based on guidance of physical principle
By establishing a single-photon noise model based on physical principles and a deep learning image reconstruction model, the problem of inaccurate noise modeling of single-photon cameras is solved, and high-quality denoising and reconstruction of single-photon images are achieved.
Patent Information
- Application Number
- CN202411421994.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-10-12
AI Technical Summary
The existing single-photon camera noise modeling technology is inaccurate and incomplete, making it difficult to effectively remove strong and complex noise in single-photon data, limiting image quality.
Based on the physical principle of single-photon camera noise, a noise model was established, and combined with deep learning methods, an end-to-end image reconstruction model was proposed. Through self-attention calculation of spatial domain and channel domain, structural feature enhancement module was designed to achieve accurate calibration of noise parameters. and image denoising.
Accurate modeling and high-quality image reconstruction of single-photon camera noise is achieved, which is better than the prior art and can quickly and efficiently remove noise in single-photon images and generate high-quality images.
Smart Images

Figure CN118917121A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a single-photon camera noise modeling and image enhancement method and system guided by physical principles, belonging to the technical field of computer image processing. Background Art
[0002] In recent years, single-photon cameras have been widely used in many fields such as ultra-dark light imaging, high dynamic range imaging, fluorescence lifetime imaging and quantum computing. Single-photon cameras use single-photon avalanche diode arrays to collect image data. This sensor has extremely high single-photon sensitivity and time resolution, and can still image normally in extremely dark light environments and high-speed dynamic scenes.
[0003] However, these properties also make single-photon sensors extremely susceptible to noise, and the causes of noise are very complex. At the same time, single-photon cameras also face problems such as severe quantization as low as 1 bit per frame. These problems have greatly limited the use of single-photon cameras to obtain high-quality images.
[0004] Most of the existing single-photon camera noise modeling technologies are limited to some of the components, or simply approximate the noise, and have not yet formed a complete noise model that conforms to the actual noise distribution. In addition, most of the current image reconstruction methods for single-photon data are based on mathematical methods such as statistics and interpolation, which have certain limitations in terms of information utilization and algorithm performance.
[0005] Therefore, how to use high-performance methods to remove strong and complex noise in single-photon data is the key to further improving the imaging performance of single-photon cameras. Summary of the invention
[0006] The purpose of the present invention is to creatively propose a single-photon camera noise modeling and image enhancement method and system based on the physical principle of noise, based on the needs of single-photon camera imaging and in view of the problems of inaccuracy and incompleteness of existing single-photon image noise modeling technology. The present invention constructs a noise model based on the physical principle of single-photon camera noise and applies deep learning methods to the image enhancement process.
[0007] The innovative points of the present invention include: in the noise modeling and noise parameter calibration part, based on the physical principles of each noise source of the single-photon camera and the statistical laws it obeys, the time interval between two adjacent photon detection events is taken as the research object, and a noise model is established in the form of probability. At the same time, according to the properties of the noise model, a scheme for calibrating the parameters of each noise component is proposed. The noise model proposed in the present invention can concisely and accurately describe the real noise distribution of the single-photon camera, and the proposed calibration scheme is easy and fast to calculate, and can decouple each noise component and calibrate it separately.
[0008] In the image reconstruction process, the present invention proposes an end-to-end image reconstruction model based on deep learning and Transformer modules. The model fully utilizes the information and dependencies in the data by calculating self-attention in the spatial domain and channel domain respectively. The present invention also designs a structural feature enhancement module to more comprehensively extract local high-frequency structural features for denoising. Experimental results show that the proposed image reconstruction model is superior to the existing technology in real single-photon image denoising.
[0009] In order to achieve the above object, the present invention adopts the following technical solutions.
[0010] A single photon image noise modeling enhancement method based on physical principles comprises the following steps: Step 1: Based on various noise physics principles, establish a complete single-photon camera noise model and design a noise parameter calibration scheme.
[0011] Specifically, the possible noise sources of the single-photon camera during the imaging process are first identified as incident photon shot noise and photon detection quantum efficiency related to the incident photon signal, as well as dark count noise, post-pulse noise and crosstalk noise related to the single-photon camera's own device materials. The probability of the time interval between two adjacent photon detection events is mathematically derived to form a noise model.
[0012] Then, a correction term is proposed based on the maximum frame number limit of a single shot in actual sampling to make the noise model conform to the actual situation.
[0013] Finally, combining the image properties of the noise model, a scheme is designed to decouple and calibrate each noise source separately.
[0014] Step 2: Use a single-photon camera to capture noise data and perform data preprocessing. Based on the preprocessed real noise data, calibrate the parameters of each noise source.
[0015] Specifically, a single-photon camera is used to capture real noise data for the subsequent noise calibration process. Considering that the calibration of dark count noise and post-pulse noise requires the probability distribution of the time interval between two adjacent photon detection events, and the calibration of crosstalk noise requires the total number of counts of each of the two sensors and the number of simultaneous counts, the present invention extracts the three key data of the histogram of the time interval between two adjacent photon detection events, the total number of counts of each sensor, and the number of simultaneous counts of each sensor and its neighboring sensors from the real single-photon data during data preprocessing and saves them (can be saved in an HDF5 file) for use in the subsequent calibration process.
[0016] After preprocessing the real single-photon data, the data required for noise calibration are obtained. Using these data, the noise calibration scheme in step 1 is used to complete the calibration of the noise source parameters of the single-photon camera. Specifically, for the calibration of dark count noise, a linear fitting method based on RANSAC can be used; for the calibration of post-pulse noise, a nonlinear fitting method can be used; for the calibration of crosstalk noise, it can be completed by solving transcendental equations.
[0017] Step 3: Generate simulation data based on the noise parameter calibration results.
[0018] After the noise parameter calibration is completed, the noise parameters of a specific single-photon camera are obtained. The present invention designs a single-photon simulation data synthesis scheme, firstly using the RAW image to estimate the incident photon rate of each pixel of the scene, and then simulating the complete imaging process of the single-photon camera in combination with the calibrated noise parameters, and finally generating single-photon simulation data with different brightness, different noise intensity, and different bit depth.
[0019] Step 4: Use the generated paired simulation data to train a deep learning image reconstruction model, and use the trained network for real single-photon image reconstruction tasks.
[0020] Based on the deep learning method and Transformer module structure, the present invention adopts an encoder-decoder structure similar to U-Net and proposes an end-to-end image reconstruction model.
[0021] Furthermore, in order to improve the model's ability to utilize different types of feature information in the data, the present invention proposes a spatial-channel Transformer block as its basic module, and also proposes a structural feature enhancement module as the feedforward layer of the Transformer block.
[0022] Specifically, the training network uses the simulated data synthesized in step 4. In order to explicitly improve the model's ability to reconstruct high-frequency details, the present invention designs a loss function strategy that guides the model to pay more attention to the recovery of high-frequency edges. After the training is completed, the real noisy data taken by the single-photon camera is input into the image reconstruction model, and the reconstructed result image is calculated. The reconstruction effect can be evaluated using evaluation indicators such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).
[0023] Furthermore, to achieve the purpose of the present invention, based on the above method, the present invention also proposes a single-photon image noise modeling enhancement system guided by physical principles, including a single-photon data shooting module, a single-photon noise calibration module, a single-photon data simulation module, an image reconstruction model training module and a single-photon imaging module.
[0024] The single-photon data capturing module is used to capture binary single-photon data using a single-photon camera, including noise data for calibrating noise and scene data for imaging; The single photon noise calibration module is used to complete the calibration process of each noise source parameter of the single photon camera according to the real single photon noise data captured; The single photon data simulation module is used to synthesize simulation data using RAW images according to the calibrated noise parameters; The image reconstruction model training module is used to complete the training process of the image reconstruction model using the synthesized simulation data; The single-photon imaging module is used to reconstruct the real scene data captured by the single-photon data capturing module using the trained image reconstruction model to remove noise and improve the image quality.
[0025] The relationship between the above modules is as follows: The output end of the single-photon data shooting module is connected to the input end of the single-photon noise calibration module and the single-photon imaging module; the output end of the single-photon noise calibration module is connected to the input end of the single-photon data simulation module; the output end of the single-photon data simulation module is connected to the input end of the image reconstruction model training module; the output end of the image reconstruction model training module is connected to the input end of the single-photon imaging module; the output end of the single-photon imaging module is connected to the output high-quality single-photon image. Beneficial Effects
[0026] Compared with the prior art, the present invention has the following advantages: 1. Based on the physical principle of single-photon camera noise generation, this paper models the time interval between adjacent photon detections and proposes a single-photon camera noise model. This noise model takes all noise sources into account and can accurately describe the real noise distribution of single-photon cameras.
[0027] 2. Based on the mathematical properties of the noise model, the present invention designs a scheme for calibrating noise parameters based on real data. The scheme is based on mature curve fitting technology and transcendental equation solving technology, with simple and efficient calculations, and can decouple and calibrate each noise source separately, and can obtain relatively accurate noise parameters.
[0028] 3. The present invention designs a single-photon simulation data synthesis scheme and an image reconstruction model based on deep learning. Through the simulation data synthesis scheme, single-photon simulation data that conforms to the real noise distribution can be quickly and massively obtained, which alleviates the dependence on the real data set to a certain extent. At the same time, the proposed image reconstruction model can quickly and efficiently remove strong and complex noise in single-photon images, thereby generating high-quality results. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a schematic diagram of the statistical distribution and noise parameters obeyed by each noise source in the method of the present invention; Figure 2 It is a schematic diagram of the noise calibration scheme in the method of the present invention; Figure 3 It is a flow chart of simulation data generation in the method of the present invention; Figure 4 It is a diagram of the architecture of the neural network model for single-photon image reconstruction in the method of the present invention; Figure 5 It is a schematic diagram of the structure of the system of the present invention. DETAILED DESCRIPTION
[0030] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0031] Example A single photon image noise modeling enhancement method based on physical principles comprises the following steps: Step 1: Based on various noise physics principles, establish a complete single-photon camera noise model and design a noise parameter calibration scheme.
[0032] like Figure 1 As shown in the figure, the noise sources of the single-photon camera mainly include the incident photon shot noise and photon detection quantum efficiency related to the incident photon signal, as well as the dark count noise, post-pulse noise and crosstalk noise related to the single-photon camera's own device materials. Among them, the incident photon shot noise and dark count noise obey the non-homogeneous Poisson process, the photon detection quantum efficiency and the probability of crosstalk noise between two sensors can be regarded as constants, and the probability of post-pulse noise is described by a multi-exponential decline.
[0033] In this embodiment, a noise model is established for a quantum imaging sensor (QIS) single-photon camera. A single-photon camera outputs 1 as long as a photon is detected within a fixed integration time, otherwise it outputs 0, so the statistical law of the noise can be obtained from the output count value distribution. Due to the event correlation of post-pulse noise and its decay over time, the interval frame number between two adjacent frames with a detection count of 1 is selected as the research object, and the impact of various types of noise is clarified by modeling the interval time.
[0034] After a pixel output count of the single-photon camera reaches 1, The probability that the frame output count is 1 is:
[0035] in, Indicates The probability that the avalanche current is not detected in the frame is described in detail according to the statistical law obeyed by each noise source:
[0036] in, the quantum efficiency of detecting incident photons for that pixel; represents the probability that the incident photon does not trigger avalanche breakdown, represents the probability that dark counts do not trigger avalanche breakdown, The first The probability of a postpulse occurring within a frame, Indicates the probability of crosstalk occurring; is the incident photon rate, is the dark count rate, represents the neighborhood of the single photon sensor; Indicates that the pixel is The probability of crosstalk between single-photon sensors; Indicates the neighborhood The sum of the dark count rate of the single-photon sensor and the effective incident photon rate; Indicates the integration time.
[0037] Substituting equation 2 into equation 1, we get the time interval between two adjacent photon detection events to be exactly The probability is:
[0038] Formula 3 is the single photon noise model proposed by the present invention.
[0039] The present invention is based on the premise that there is no upper limit to the number of frames that a single-photon camera can acquire each time. However, in practice, single-photon cameras are limited by factors such as the size of their own storage space and are difficult to meet this condition. The limited maximum number of frames will reduce the probability of large time intervals. Therefore, it is necessary to further consider the influence of this factor in the noise model and add a correction term to correct this deviation in the probability distribution. Specifically, if the total number of frames captured by a single-photon camera in a single shot is , then for the length The time interval is only in the front The correction term is therefore able to produce a length of The ratio of the integral time of the time interval to the total time interval, that is:
[0040] This correction term should be added as a multiplicative factor in Equation 3, and the logarithm of Equation 3 should be taken to decouple the parts of the multiplicative coupling, resulting in:
[0041] In Formula 5, the first term is a correction term, the third term is a linear term, and the second and fourth terms are both nonlinear terms.
[0042] According to theoretical and experimental analysis, the second and fourth terms will approach a constant when they are large enough. Here, the sum of the limits of these two terms is recorded as Therefore, for larger (usually the data with the length of the time interval ranking in the last 90%), after subtracting the correction term, Formula 5 will approximately become a linear formula, such as Figure 2 As shown on the right side of the curve, the parameters are calibrated using linear fitting using the RANSAC algorithm paradigm. (usually the data with the first 10% time interval length), the logarithmic probability curve is nonlinear, such as Figure 2 As shown on the left side of the curve, the only parameters left are the post-pulse probability coefficient and the deep energy level lifetime. For these parameters, the existing nonlinear fitting mathematical tools are used to fit the parameter values according to the nonlinear part of the head of the logarithmic probability curve. Although fitting the nonlinear part is inevitable, the complexity and difficulty of fitting will be greatly reduced because the linear coefficients have been independently calibrated before.
[0043] In the linear coefficient of the linear term of the above calibration scheme, the crosstalk noise probability In fact, the non-homogeneous Poisson mean is still coupled together, and it is impossible to solve them separately using only the count interval distribution output by the single-photon camera. In this regard, the present invention designs a calibration method for the crosstalk probability, which can calibrate the crosstalk noise probability without additional data collection. The calibration is performed independently to decouple the crosstalk noise and the non-homogeneous Poisson noise.
[0044] Specifically, Intra-frame single photon sensor and The total count frames are and , the number of simultaneous counting frames due to crosstalk noise is , at this time, the number of simultaneous counting frames caused by the non-homogeneous Poisson process that has nothing to do with crosstalk is and the number of frames where crosstalk occurs Described as:
[0045] According to formula 6, we can get and .
[0046] Considering that multiple avalanche currents may be generated in each integration time, and each avalanche current may cause crosstalk, it is impossible to determine how many counts have occurred in this integration time based on the output count. This problem can be solved by analyzing the probability of crosstalk. Expectations for:
[0047] in, and Represents single photon sensor , The sum of the effective incident photon rate and the dark count rate, Single Photon Sensor and The probability of crosstalk between .
[0048] When the total number of frames When it is very large, I think , so by solving the transcendental equation in equation 7, we can get the crosstalk probability between two single-photon sensors: The above analysis is based on the non-homogeneous Poisson process. However, due to the influence of factors such as afterpulses, the noise of the single-photon data captured in practice does not obey the perfect non-homogeneous Poisson process. Considering that the probability of afterpulses is exponentially decaying, the influence of afterpulses is eliminated as much as possible by using a large integration time and recording the count value every several frames. At this point, the crosstalk probability is calibrated by using the above calibration scheme.
[0049] Step 2: Use a single-photon camera to capture noise data for data preprocessing. Calibrate the parameters of each noise source based on the real noise data after preprocessing.
[0050] Specifically, a single-photon camera is used to capture real binary single-photon noise data for subsequent noise calibration process. According to the noise model and calibration scheme in step 1, the calibration of dark count noise and post-pulse noise requires the probability distribution of the time interval between two adjacent photon detection events, while the calibration of crosstalk noise requires the use of the total number of counts and the number of simultaneous counts of each of the two sensors. Therefore, in the data preprocessing process, the present invention extracts the time interval between two adjacent photon detection events from the real single-photon data to construct a logarithmic probability curve of the time interval, and calibrates the parameters of dark count noise and post-pulse noise based on the logarithmic probability curve using linear fitting and nonlinear fitting techniques. At the same time, the total number of counts of each sensor and the number of simultaneous counts of each sensor and the sensors in its neighborhood are extracted from the single-photon data, and the crosstalk probability between each single-photon sensor pair is calibrated by solving a set of equations and transcendental equations. The results of data preprocessing can be saved in an HDF5 file for use in the calibration process.
[0051] Step 3: Generate simulation data based on the noise parameter calibration results.
[0052] Deep learning methods rely on paired data sets to train neural networks. Existing single-photon simulation data synthesis schemes usually use grayscale images or RGB color images to estimate the number of photons in each pixel of the scene. However, since RGB images are obtained by performing a series of post-processing operations on the RAW raw data output by the camera, the subsequent processing operations usually introduce complex nonlinear transformations, thereby changing the scene photon information implicit in the RAW image. At the same time, the bit depth of grayscale images and RGB images is usually 8 bits, so at most only 256 different photon numbers can be estimated, which will introduce non-negligible quantization errors and lead to serious deviations in the estimation of scene photon information.
[0053] To address these problems, the present invention chooses to directly use the RAW image to estimate the number of photons in each pixel in the scene, and combines the proposed noise model to simulate the complete imaging process of the single-photon camera to synthesize high-quality simulation data.
[0054] Specifically, the simulation data synthesis solution flow chart proposed by the present invention is as follows: Figure 3 As shown in Figure 1, it includes the scene information simulation stage and the simulation image generation stage. It is observed that the Bayer array and single photon sensor of CCD / CMOS camera are most sensitive to green light, and the quantum efficiency decay trend of both towards purple and red wavelengths is also very similar. According to this rule, the incident photon rate of each pixel is predicted directly using each 2x2 pixel block of the Bayer array in the RAW image, as shown in Figure 1. Figure 3 Shown in the left part.
[0055] After the incident photon rate of each pixel is obtained, the incident photon rate and the calibrated noise parameters are used to synthesize the single-photon simulation data by simulating the complete imaging process of the single-photon camera in the simulation image generation stage, such as Figure 3 As shown in the right part. Specifically, different illumination scaling factors are first used to multiply the incident photon rate estimated in the scene information simulation stage to simulate scenes of different brightness. Then, different integration times are used to multiply the calibrated dark count rate to simulate different dark count noise intensities at each integration time. Finally, the scaled incident photon rate and dark count rate are added, the Poisson mean of the obtained non-homogeneous Poisson process is sampled, and the after-pulse probability is calculated in each frame and the influence of crosstalk is considered.
[0056] Step 4: Use the generated paired simulation data to train a deep learning image reconstruction model and use the trained network for real single-photon image reconstruction tasks.
[0057] Based on the deep learning method and Transformer module structure, an end-to-end image reconstruction model is established, in which the spatial-channel Transformer block is used as its basic module, and the structural feature enhancement module is used as the feedforward layer of the Transformer block; the network is trained using simulated data; after the training is completed, the real noisy data is input into the image reconstruction model, and the reconstructed result image is calculated.
[0058] After the channel features and spatial features enter the SCTB, a 1x1 convolution is first used to integrate the input features along the channel dimension, and then the feature map is evenly divided into two blocks along the channel dimension, and the channel domain Transformer block and the spatial domain Transformer block are used to perform self-attention calculations in the spatial domain and channel domain for these two sub-feature maps respectively; among them, the sliding window mechanism of Swin-Transformer is used in the spatial domain Transformer block; in the channel domain Transformer block, a channel self-attention mechanism is used that only performs global perception along the channel and does not care about the specific position information; after passing through the channel domain Transformer and the spatial domain Transformer, the feature maps calculated by the two modules are spliced along the channel dimension, and a 1x1 convolution is used for gated fusion.
[0059] The structural feature enhancement module uses a symmetrical dual-path channel-by-channel convolution to extract local structural information with stronger nonlinearity and expression capabilities. The module first uses a 1x1 convolution to increase the channel dimension of the input feature. times, and then use the GELU activation function to process the upgraded features, and use a 1x1 convolution to perform preliminary feature extraction; then, two branches are used to perform different convolution methods respectively, one of which uses 3x3 channel-by-channel convolution to extract local features, and the other branch uses 3x3 channel-by-channel dilated convolution to extract feature information within a larger receptive field; in these two branches, the structural feature enhancement module uses two symmetrical channel-by-channel convolutions to extract features, and uses element-level multiplication to fuse the features; finally, the structural enhancement feature map that has passed the above two branches is spliced along the channel dimension, and a 1x1 convolution is used to fuse features of different spatial scales, and the number of channels is restored to the number of channels of the input feature map, so as to obtain the final calculation result.
[0060] First, we introduce the logarithmic loss The loss is calculated in the logarithmic domain to increase the difference between different pixel values, that is: in, represents the reconstructed image, represents the true value image, , Represent the width and height of the image respectively; then, LPIPS loss and SSIM loss are introduced; among them, LPIPS loss is used to utilize the perception ability of the neural network to ensure the similarity of high-level structures, and SSIM loss is used to ensure the similarity of local structures by calculating the statistical information of brightness, contrast and structure; edge loss is introduced ,use Direction and The Sobel operators of the directions are denoted as and , to extract the edge information of the reconstructed image and the real image, and calculate the loss of the extracted edge information to guide the model to pay more attention to the recovery of high-frequency edges; the edge loss is calculated as: Combine the above loss functions to get the complete loss function : in, Respectively represent the weight of each loss; represents the Charbonnier loss, Represents SSIM loss; after training, the real noisy data taken by the single-photon camera is input into the image reconstruction model, and the reconstructed result image is calculated.
[0061] It includes a single-photon data shooting module, a single-photon noise calibration module, a single-photon data simulation module, an image reconstruction model training module and a single-photon imaging module; wherein the single-photon data shooting module is used to use a single-photon camera to shoot binary single-photon data, including noise data for calibrating noise and scene data for imaging; the single-photon noise calibration module is used to complete the calibration process of each noise source parameter of the single-photon camera according to the captured real single-photon noise data; the single-photon data simulation module is used to use the RAW image to synthesize simulation data according to the calibrated noise parameters; the image reconstruction model training module is used to use the synthesized simulation data to complete the training process of the image reconstruction model; the single-photon The imaging module is used to use the trained image reconstruction model to reconstruct the real scene data captured by the single-photon data shooting module to remove the noise and improve the image quality; the connection relationship between the above modules is: the output end of the single-photon data shooting module is connected to the input end of the single-photon noise calibration module and the single-photon imaging module; the output end of the single-photon noise calibration module is connected to the input end of the single-photon data simulation module; the output end of the single-photon data simulation module is connected to the input end of the image reconstruction model training module; the output end of the image reconstruction model training module is connected to the input end of the single-photon imaging module; the output end of the single-photon imaging module is connected to the output high-quality single-photon image.
Claims
1. A single-photon image noise modeling enhancement method based on physical principles, characterized in that: The following steps are involved: Step 1: Based on various noise physical principles, a complete single-photon camera noise model is established, and a noise parameter calibration scheme is designed; first, the noise sources that may appear in the imaging process of the single-photon camera are clearly defined as the incident photon shot noise and photon detection quantum efficiency related to the incident photon signal, as well as the dark count noise, post-pulse noise and crosstalk noise related to the single-photon camera's own device materials; the probability of the time interval between two adjacent photon detection events is mathematically derived, and a noise model is formed; then, combined with the maximum frame number limit of a single shot in actual sampling, a correction term is proposed to make the noise model conform to the actual situation; finally, combined with the image properties of the noise model, a scheme is designed to decouple and calibrate each noise source separately; Step 2: Use a single-photon camera to capture noise data and perform data preprocessing. According to the real noise data after preprocessing, the parameters of each noise source are calibrated; Use a single-photon camera to capture real noise data for the subsequent noise calibration process; During the data preprocessing process, the histogram of the time interval between two adjacent photon detection events, the total number of counts of each sensor, and the number of simultaneous counts of each sensor and the sensors in its neighborhood are extracted from the real single-photon data and saved; after preprocessing the real single-photon data, the various data required for noise calibration are obtained, and these data are used to complete the calibration of the various noise source parameters of the single-photon camera using the noise calibration scheme; Step 3: Generate simulation data based on the noise parameter calibration results; first use the RAW image to estimate the incident photon rate of each pixel in the scene, and then simulate the complete imaging process of the single-photon camera in combination with the calibrated noise parameters, and finally generate single-photon simulation data with different brightness, different noise intensity, and different bit depth; Step 4: Use the generated paired simulation data to train the deep learning image reconstruction model, and use the trained network for the real single-photon image reconstruction task.
2. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 1, characterized in that: In step 1, after a pixel output count of the single-photon camera reaches 1, The probability that the frame output count is 1 is: in, Indicates The probability that the avalanche current is not detected in the frame is described in detail according to the statistical law obeyed by each noise source: in, the quantum efficiency of detecting incident photons for that pixel; represents the probability that the incident photon does not trigger avalanche breakdown, represents the probability that dark counts do not trigger avalanche breakdown, The first The probability of a postpulse occurring within a frame, Indicates the probability of crosstalk occurring; is the incident photon rate, is the dark count rate, represents the neighborhood of the single photon sensor; Indicates that the pixel is The probability of crosstalk between single-photon sensors; Indicates the neighborhood The sum of the dark count rate of the single-photon sensor and the effective incident photon rate; represents the integration time; Substituting Equation 2 into Equation 1, the time interval between two adjacent photon detection events is exactly The probability is: Equation 3 is the single photon noise model; the correction term is the one that can produce a length of The ratio of the integral time of the time interval to the total time interval: in, is the total number of frames captured by the single-photon camera in a single shot; this correction term is added to Equation 3 as a multiplicative factor, and the logarithm of Equation 3 is taken to decouple the various parts of the multiplicative coupling, resulting in: In Formula 5, the first term is a correction term, the third term is a linear term, and the second and fourth terms are both nonlinear terms.
3. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 2, characterized in that: Design a calibration method for the crosstalk probability, and calculate the crosstalk noise probability without collecting additional data Independently calibrate to achieve decoupling of crosstalk noise and non-homogeneous Poisson noise; Intra-frame single photon sensor and The total count frames are and , the number of simultaneous counting frames due to crosstalk noise is Described as: According to formula 6, we can get and ; Number of simultaneous counting frames due to crosstalk noise Expectations for: in, and Represents single photon sensor , The sum of the effective incident photon rate and the dark count rate, Single Photon Sensor and The probability of crosstalk between When it is very large, I think , solve equation 7 to obtain the crosstalk probability between two single-photon sensors .
4. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 1, characterized in that: In step 2, for the calibration of dark count noise, a linear fitting method based on RANSAC is used; for the calibration of post-pulse noise, a nonlinear fitting method is used; for the calibration of crosstalk noise, it is completed by solving transcendental equations.
5. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 1, characterized in that: Step 3 includes scene information simulation and simulated image generation; using each 2x2 pixel block of the Bayer array in the RAW image to predict the incident photon rate of each pixel; then, in the simulated image generation stage, the incident photon rate and the calibrated noise parameters are used to synthesize single-photon simulation data by simulating the complete imaging process of the single-photon camera; first, different illumination scaling factors are used to multiply the incident photon rate estimated in the scene information simulation stage to simulate scenes of different brightness; then, different integration times are used to multiply the calibrated dark count rate to simulate different dark count noise intensities at each integration time; finally, the scaled incident photon rate and dark count rate are added, the Poisson mean of the non-homogeneous Poisson process is sampled, and the after-pulse probability is calculated in each frame and the influence of crosstalk is considered, and single-photon data of different frames are obtained by cyclic sampling of each frame.
6. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 1, characterized in that: In step 4, an end-to-end image reconstruction model is established based on the deep learning method and the Transformer module structure, in which the spatial-channel Transformer block is used as its basic module, and the structural feature enhancement module is used as the feedforward layer of the Transformer block; the network is trained using simulated data; after the training is completed, the real noisy data is input into the image reconstruction model, and the reconstructed result image is calculated.
7. The single photon image noise modeling enhancement method based on physical principle as claimed in claim 6, characterized in that: After the channel features and spatial features enter the SCTB, a 1x1 convolution is first used to integrate the input features along the channel dimension, and then the feature map is evenly divided into two blocks along the channel dimension, and the channel domain Transformer block and the spatial domain Transformer block are used to perform self-attention calculations in the spatial domain and channel domain for these two sub-feature maps respectively; among them, the sliding window mechanism of Swin-Transformer is used in the spatial domain Transformer block; in the channel domain Transformer block, a channel self-attention mechanism is used that only performs global perception along the channel and does not care about the specific position information; after passing through the channel domain Transformer and the spatial domain Transformer, the feature maps calculated by the two modules are spliced along the channel dimension, and a 1x1 convolution is used for gated fusion.
8. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 6, characterized in that: The structural feature enhancement module uses a symmetrical dual-path channel-by-channel convolution to extract local structural information with stronger nonlinearity and expression capabilities. The module first uses a 1x1 convolution to increase the channel dimension of the input feature. times, and then use the GELU activation function to process the upgraded features, and use a 1x1 convolution to perform preliminary feature extraction; then, two branches are used to perform different convolution methods respectively, one of which uses 3x3 channel-by-channel convolution to extract local features, and the other branch uses 3x3 channel-by-channel dilated convolution to extract feature information within a larger receptive field; in these two branches, the structural feature enhancement module uses two symmetrical channel-by-channel convolutions to extract features, and uses element-level multiplication to fuse the features; finally, the structural enhancement feature map that has passed the above two branches is spliced along the channel dimension, and a 1x1 convolution is used to fuse features of different spatial scales, and the number of channels is restored to the number of channels of the input feature map, so as to obtain the final calculation result.
9. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 6, characterized in that: First, we introduce the logarithmic loss The loss is calculated in the logarithmic domain to increase the difference between different pixel values, that is: in, represents the reconstructed image, represents the true value image, , Represent the width and height of the image respectively; then, LPIPS loss and SSIM loss are introduced; among them, LPIPS loss is used to utilize the perception ability of the neural network to ensure the similarity of high-level structures, and SSIM loss is used to ensure the similarity of local structures by calculating the statistical information of brightness, contrast and structure; edge loss is introduced ,use Direction and The Sobel operators of the directions are denoted as and , to extract the edge information of the reconstructed image and the real image, and calculate the loss of the extracted edge information to guide the model to pay more attention to the recovery of high-frequency edges; the edge loss is calculated as: Combine the above loss functions to get the complete loss function : in, Respectively represent the weight of each loss; represents the Charbonnier loss, Represents SSIM loss; after training, the real noisy data taken by the single-photon camera is input into the image reconstruction model, and the reconstructed result image is calculated.
10. A single photon image noise modeling enhancement system based on physical principles for implementing the method of claim 1, characterized in that: It includes a single-photon data shooting module, a single-photon noise calibration module, a single-photon data simulation module, an image reconstruction model training module and a single-photon imaging module; wherein the single-photon data shooting module is used to use a single-photon camera to shoot binary single-photon data, including noise data for calibrating noise and scene data for imaging; the single-photon noise calibration module is used to complete the calibration process of each noise source parameter of the single-photon camera according to the captured real single-photon noise data; the single-photon data simulation module is used to use the RAW image to synthesize simulation data according to the calibrated noise parameters; the image reconstruction model training module is used to use the synthesized simulation data to complete the training process of the image reconstruction model; the single-photon The imaging module is used to use the trained image reconstruction model to reconstruct the real scene data captured by the single-photon data shooting module to remove the noise and improve the image quality; the connection relationship between the above modules is: the output end of the single-photon data shooting module is connected to the input end of the single-photon noise calibration module and the single-photon imaging module; the output end of the single-photon noise calibration module is connected to the input end of the single-photon data simulation module; the output end of the single-photon data simulation module is connected to the input end of the image reconstruction model training module; the output end of the image reconstruction model training module is connected to the input end of the single-photon imaging module; the output end of the single-photon imaging module is connected to the output high-quality single-photon image.
Citation Information
Patent Citations
High-resolution single-photon imaging method and system combined with physical noise model
CN112461360A
CNN-Transform-based photon counting image high-efficiency joint denoising super-division method and CNN-Transform-based photon counting image high-efficiency joint denoising super-division system
CN117557467A
Method and system for single-photon imaging
WO2024178501A1
Cited By
Ultra-weak light detection system and method based on SPAD array and time correlation photon counting
CN120593892A
Artificial intelligence-based PET detector signal simulation generation method
CN121145684A
A method for generating a PET detector signal simulation based on artificial intelligence
CN121145684B
Multi-target sensing and tracking method for detecting unmanned aerial vehicle swarm through single-photon laser radar
CN121186804A
Low-illumination image enhancement method based on RAW image data and event camera data fusion
CN121767254A
