Single-photon image noise modeling enhancement method and system based on physical principles

Abstract

The present invention proposes a single-photon image noise modeling enhancement method and system guided by physical principles, which belongs to the field of computer image processing technology. First, starting from the physical principles of single-photon noise, a complete noise model is constructed with probability, and a noise calibration scheme is designed according to the properties of the noise model. Then, in combination with the noise model, RAW images are used to synthesize high-quality paired single-photon data to provide strong support for the training of neural networks. Finally, a high-performance image reconstruction model is used to utilize the self-attention mechanism in the channel domain and the spatial domain, the structural feature enhancement, and the loss strategy of explicitly guiding high-frequency texture recovery to achieve better image reconstruction and denoising effects. The present invention can accurately characterize the real noise distribution of a single-photon camera and restore clear detail textures from noisy images with high quality.

Description

Technical Field

[0001] The invention relates to a single-photon camera noise modeling and image enhancement method and system guided by physical principles, belonging to the technical field of computer image processing. Background Art

[0002] In recent years, single-photon cameras have been widely used in many fields such as ultra-dark light imaging, high dynamic range imaging, fluorescence lifetime imaging and quantum computing. Single-photon cameras use single-photon avalanche diode arrays to collect image data. This sensor has extremely high single-photon sensitivity and time resolution, and can still image normally in extremely dark light environments and high-speed dynamic scenes.

[0003] However, these properties also make single-photon sensors extremely susceptible to noise, and the causes of noise are very complex. At the same time, single-photon cameras also face problems such as severe quantization as low as 1 bit per frame. These problems have greatly limited the use of single-photon cameras to obtain high-quality images.

[0004] Most of the existing single-photon camera noise modeling technologies are limited to some of the components, or simply approximate the noise, and have not yet formed a complete noise model that conforms to the actual noise distribution. In addition, most of the current image reconstruction methods for single-photon data are based on mathematical methods such as statistics and interpolation, which have certain limitations in terms of information utilization and algorithm performance.

[0005] Therefore, how to use high-performance methods to remove strong and complex noise in single-photon data is the key to further improving the imaging performance of single-photon cameras. Summary of the invention

[0006] The purpose of the present invention is to creatively propose a single-photon camera noise modeling and image enhancement method and system based on the physical principle of noise, based on the needs of single-photon camera imaging and in view of the problems of inaccuracy and incompleteness of existing single-photon image noise modeling technology. The present invention constructs a noise model based on the physical principle of single-photon camera noise and applies deep learning methods to the image enhancement process.

[0007] The innovative points of the present invention include: in the noise modeling and noise parameter calibration part, based on the physical principles of each noise source of the single-photon camera and the statistical laws it obeys, the time interval between two adjacent photon detection events is taken as the research object, and a noise model is established in the form of probability. At the same time, according to the properties of the noise model, a scheme for calibrating the parameters of each noise component is proposed. The noise model proposed in the present invention can concisely and accurately describe the real noise distribution of the single-photon camera, and the proposed calibration scheme is easy and fast to calculate, and can decouple each noise component and calibrate it separately.

[0008] In the image reconstruction process, the present invention proposes an end-to-end image reconstruction model based on deep learning and Transformer modules. The model fully utilizes the information and dependencies in the data by calculating self-attention in the spatial domain and channel domain respectively. The present invention also designs a structural feature enhancement module to more comprehensively extract local high-frequency structural features for denoising. Experimental results show that the proposed image reconstruction model is superior to the existing technology in real single-photon image denoising.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions.

[0010] A single photon image noise modeling enhancement method based on physical principles comprises the following steps:

[0011] Step 1: Based on various noise physics principles, establish a complete single-photon camera noise model and design a noise parameter calibration scheme.

[0012] Specifically, the possible noise sources of the single-photon camera during the imaging process are first identified as incident photon shot noise and photon detection quantum efficiency related to the incident photon signal, as well as dark count noise, post-pulse noise and crosstalk noise related to the single-photon camera's own device materials. The probability of the time interval between two adjacent photon detection events is mathematically derived to form a noise model.

[0013] Then, a correction term is proposed based on the maximum frame number limit of a single shot in actual sampling to make the noise model conform to the actual situation.

[0014] Finally, combining the image properties of the noise model, a scheme is designed to decouple and calibrate each noise source separately.

[0015] Step 2: Use a single-photon camera to capture noise data and perform data preprocessing. Based on the preprocessed real noise data, calibrate the parameters of each noise source.

[0016] Specifically, a single-photon camera is used to capture real noise data for the subsequent noise calibration process. Considering that the calibration of dark count noise and post-pulse noise requires the probability distribution of the time interval between two adjacent photon detection events, and the calibration of crosstalk noise requires the total number of counts of each of the two sensors and the number of simultaneous counts, the present invention extracts the three key data of the histogram of the time interval between two adjacent photon detection events, the total number of counts of each sensor, and the number of simultaneous counts of each sensor and its neighboring sensors from the real single-photon data during data preprocessing and saves them (can be saved in an HDF5 file) for use in the subsequent calibration process.

[0017] After preprocessing the real single-photon data, the data required for noise calibration are obtained. Using these data, the noise calibration scheme in step 1 is used to complete the calibration of the noise source parameters of the single-photon camera. Specifically, for the calibration of dark count noise, a linear fitting method based on RANSAC can be used; for the calibration of post-pulse noise, a nonlinear fitting method can be used; for the calibration of crosstalk noise, it can be completed by solving transcendental equations.

[0018] Step 3: Generate simulation data based on the noise parameter calibration results.

[0019] After the noise parameter calibration is completed, the noise parameters of a specific single-photon camera are obtained. The present invention designs a single-photon simulation data synthesis scheme, firstly using the RAW image to estimate the incident photon rate of each pixel of the scene, and then simulating the complete imaging process of the single-photon camera in combination with the calibrated noise parameters, and finally generating single-photon simulation data with different brightness, different noise intensity, and different bit depth.

[0020] Step 4: Use the generated paired simulation data to train a deep learning image reconstruction model, and use the trained network for real single-photon image reconstruction tasks.

[0021] Based on the deep learning method and Transformer module structure, the present invention adopts an encoder-decoder structure similar to U-Net and proposes an end-to-end image reconstruction model.

[0022] Furthermore, in order to improve the model's ability to utilize different types of feature information in the data, the present invention proposes a spatial-channel Transformer block as its basic module, and also proposes a structural feature enhancement module as the feedforward layer of the Transformer block.

[0023] Specifically, the training network uses the simulated data synthesized in step 4. In order to explicitly improve the model's ability to reconstruct high-frequency details, the present invention designs a loss function strategy that guides the model to pay more attention to the recovery of high-frequency edges. After the training is completed, the real noisy data taken by the single-photon camera is input into the image reconstruction model, and the reconstructed result image is calculated. The reconstruction effect can be evaluated using evaluation indicators such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0024] Furthermore, to achieve the purpose of the present invention, based on the above method, the present invention also proposes a single-photon image noise modeling enhancement system guided by physical principles, including a single-photon data shooting module, a single-photon noise calibration module, a single-photon data simulation module, an image reconstruction model training module and a single-photon imaging module.

[0025] The single-photon data capturing module is used to capture binary single-photon data using a single-photon camera, including noise data for calibrating noise and scene data for imaging;

[0026] The single photon noise calibration module is used to complete the calibration process of each noise source parameter of the single photon camera according to the real single photon noise data captured;

[0027] The single photon data simulation module is used to synthesize simulation data using RAW images according to the calibrated noise parameters;

[0028] The image reconstruction model training module is used to complete the training process of the image reconstruction model using the synthesized simulation data;

[0029] The single-photon imaging module is used to reconstruct the real scene data captured by the single-photon data capturing module using the trained image reconstruction model to remove noise and improve the image quality.

[0030] The relationship between the above modules is as follows:

[0031] The output end of the single-photon data shooting module is connected to the input end of the single-photon noise calibration module and the single-photon imaging module; the output end of the single-photon noise calibration module is connected to the input end of the single-photon data simulation module; the output end of the single-photon data simulation module is connected to the input end of the image reconstruction model training module; the output end of the image reconstruction model training module is connected to the input end of the single-photon imaging module; the output end of the single-photon imaging module is connected to the output high-quality single-photon image. Beneficial Effects

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] 1. Based on the physical principle of single-photon camera noise generation, this paper models the time interval between adjacent photon detections and proposes a single-photon camera noise model. This noise model takes all noise sources into account and can accurately describe the real noise distribution of single-photon cameras.

[0034] 2. Based on the mathematical properties of the noise model, the present invention designs a scheme for calibrating noise parameters based on real data. The scheme is based on mature curve fitting technology and transcendental equation solving technology, with simple and efficient calculations, and can decouple and calibrate each noise source separately, and can obtain relatively accurate noise parameters.

[0035] 3. The present invention designs a single-photon simulation data synthesis scheme and an image reconstruction model based on deep learning. Through the simulation data synthesis scheme, single-photon simulation data that conforms to the real noise distribution can be quickly and massively obtained, which alleviates the dependence on the real data set to a certain extent. At the same time, the proposed image reconstruction model can quickly and efficiently remove strong and complex noise in single-photon images, thereby generating high-quality results. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the statistical distribution and noise parameters obeyed by each noise source in the method of the present invention;

[0037] Figure 2 It is a schematic diagram of the noise calibration scheme in the method of the present invention;

[0038] Figure 3 It is a flow chart of simulation data generation in the method of the present invention;

[0039] Figure 4 It is a diagram of the architecture of the neural network model for single-photon image reconstruction in the method of the present invention;

[0040] Figure 5 It is a schematic diagram of the structure of the system of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0042] Example

[0043] A single photon image noise modeling enhancement method based on physical principles comprises the following steps:

[0044] Step 1: Based on various noise physics principles, establish a complete single-photon camera noise model and design a noise parameter calibration scheme.

[0045] like Figure 1 As shown in the figure, the noise sources of the single-photon camera mainly include the incident photon shot noise and photon detection quantum efficiency related to the incident photon signal, as well as the dark count noise, post-pulse noise and crosstalk noise related to the single-photon camera's own device materials. Among them, the incident photon shot noise and dark count noise obey the non-homogeneous Poisson process, the photon detection quantum efficiency and the probability of crosstalk noise between two sensors can be regarded as constants, and the probability of post-pulse noise is described by a multi-exponential decline.

[0046] In this embodiment, a noise model is established for a quantum imaging sensor (QIS) single-photon camera. A single-photon camera outputs 1 as long as a photon is detected within a fixed integration time, otherwise it outputs 0, so the statistical law of the noise can be obtained from the output count value distribution. Due to the event correlation of post-pulse noise and its decay over time, the interval frame number between two adjacent frames with a detection count of 1 is selected as the research object, and the impact of various types of noise is clarified by modeling the interval time.

[0047] After a pixel output count of the single-photon camera reaches 1, The probability that the frame output count is 1 is:

[0048]

[0049] in, Indicates The probability that the avalanche current is not detected in the frame is described in detail according to the statistical law obeyed by each noise source:

[0050]

[0051] in, the quantum efficiency of detecting incident photons for that pixel; represents the probability that the incident photon does not trigger avalanche breakdown, represents the probability that dark counts do not trigger avalanche breakdown, The first The probability of a postpulse occurring within a frame, Indicates the probability of crosstalk occurring; is the incident photon rate, is the dark count rate, represents the neighborhood of the single photon sensor; Indicates that the pixel is The probability of crosstalk between single-photon sensors; Indicates the neighborhood The sum of the dark count rate of the single-photon sensor and the effective incident photon rate; Indicates the integration time.

[0052] Substituting equation 2 into equation 1, we get the time interval between two adjacent photon detection events to be exactly The probability is:

[0053]

[0054] Formula 3 is the single photon noise model proposed by the present invention.

[0055] The present invention is based on the premise that there is no upper limit to the number of frames that a single-photon camera can acquire each time. However, in practice, single-photon cameras are limited by factors such as the size of their own storage space and are difficult to meet this condition. The limited maximum number of frames will reduce the probability of large time intervals. Therefore, it is necessary to further consider the influence of this factor in the noise model and add a correction term to correct this deviation in the probability distribution. Specifically, if the total number of frames captured by a single-photon camera in a single shot is , then for the length The time interval is only in the front The correction term is therefore able to produce a length of The ratio of the integral time of the time interval to the total time interval, that is:

[0056]

[0057] This correction term should be added as a multiplicative factor in Equation 3, and the logarithm of Equation 3 should be taken to decouple the parts of the multiplicative coupling, resulting in:

[0058]

[0059] In Formula 5, the first term is a correction term, the third term is a linear term, and the second and fourth terms are both nonlinear terms.

[0060] According to theoretical and experimental analysis, the second and fourth terms will approach a constant when they are large enough. Here, the sum of the limits of these two terms is recorded as Therefore, for larger (usually the data with the length of the time interval ranking in the last 90%), after subtracting the correction term, Formula 5 will approximately become a linear formula, such as Figure 2 As shown on the right side of the curve, the parameters are calibrated using linear fitting using the RANSAC algorithm paradigm. (usually the data with the first 10% time interval length), the logarithmic probability curve is nonlinear, such as Figure 2 As shown on the left side of the curve, the only parameters left are the post-pulse probability coefficient and the deep energy level lifetime. For these parameters, the existing nonlinear fitting mathematical tools are used to fit the parameter values ​​according to the nonlinear part of the head of the logarithmic probability curve. Although fitting the nonlinear part is inevitable, the complexity and difficulty of fitting will be greatly reduced because the linear coefficients have been independently calibrated before.

[0061] In the linear coefficient of the linear term of the above calibration scheme, the crosstalk noise probability In fact, the non-homogeneous Poisson mean is still coupled together, and it is impossible to solve them separately using only the count interval distribution output by the single-photon camera. In this regard, the present invention designs a calibration method for the crosstalk probability, which can calibrate the crosstalk noise probability without additional data collection. The calibration is performed independently to decouple the crosstalk noise and the non-homogeneous Poisson noise.

[0062] Specifically, Intra-frame single photon sensor and The total count frames are and , the number of simultaneous counting frames due to crosstalk noise is , at this time, the number of simultaneous counting frames caused by the non-homogeneous Poisson process that has nothing to do with crosstalk is and the number of frames where crosstalk occurs Described as:

[0063]

[0064] According to formula 6, we can get and .

[0065] Considering that multiple avalanche currents may be generated in each integration time, and each avalanche current may cause crosstalk, it is impossible to determine how many counts have occurred in this integration time based on the output count. This problem can be solved by analyzing the probability of crosstalk. Expectations for:

[0066]

[0067] in, and Represents single photon sensor , The sum of the effective incident photon rate and the dark count rate, Single Photon Sensor and The probability of crosstalk between .

[0068] When the total number of frames When it is very large, I think , so by solving the transcendental equation in equation 7, we can get the crosstalk probability between two single-photon sensors: The above analysis is based on the non-homogeneous Poisson process. However, due to the influence of factors such as afterpulses, the noise of the single-photon data captured in practice does not obey the perfect non-homogeneous Poisson process. Considering that the probability of afterpulses is exponentially decaying, the influence of afterpulses is eliminated as much as possible by using a large integration time and recording the count value every several frames. At this point, the crosstalk probability is calibrated by using the above calibration scheme.

[0069] Step 2: Use a single-photon camera to capture noise data for data preprocessing. Calibrate the parameters of each noise source based on the real noise data after preprocessing.

[0070] Specifically, a single-photon camera is used to capture real binary single-photon noise data for subsequent noise calibration process. According to the noise model and calibration scheme in step 1, the calibration of dark count noise and post-pulse noise requires the probability distribution of the time interval between two adjacent photon detection events, while the calibration of crosstalk noise requires the use of the total number of counts and the number of simultaneous counts of each of the two sensors. Therefore, in the data preprocessing process, the present invention extracts the time interval between two adjacent photon detection events from the real single-photon data to construct a logarithmic probability curve of the time interval, and calibrates the parameters of dark count noise and post-pulse noise based on the logarithmic probability curve using linear fitting and nonlinear fitting techniques. At the same time, the total number of counts of each sensor and the number of simultaneous counts of each sensor and the sensors in its neighborhood are extracted from the single-photon data, and the crosstalk probability between each single-photon sensor pair is calibrated by solving a set of equations and transcendental equations. The results of data preprocessing can be saved in an HDF5 file for use in the calibration process.

[0071] Step 3: Generate simulation data based on the noise parameter calibration results.

[0072] Deep learning methods rely on paired data sets to train neural networks. Existing single-photon simulation data synthesis schemes usually use grayscale images or RGB color images to estimate the number of photons in each pixel of the scene. However, since RGB images are obtained by performing a series of post-processing operations on the RAW raw data output by the camera, the subsequent processing operations usually introduce complex nonlinear transformations, thereby changing the scene photon information implicit in the RAW image. At the same time, the bit depth of grayscale images and RGB images is usually 8 bits, so at most only 256 different photon numbers can be estimated, which will introduce non-negligible quantization errors and lead to serious deviations in the estimation of scene photon information.

[0073] To address these problems, the present invention chooses to directly use the RAW image to estimate the number of photons in each pixel in the scene, and combines the proposed noise model to simulate the complete imaging process of the single-photon camera to synthesize high-quality simulation data.

[0074] Specifically, the simulation data synthesis solution flow chart proposed by the present invention is as follows: Figure 3 As shown in Figure 1, it includes the scene information simulation stage and the simulation image generation stage. It is observed that the Bayer array and single photon sensor of CCD / CMOS camera are most sensitive to green light, and the quantum efficiency decay trend of both towards purple and red wavelengths is also very similar. According to this rule, the incident photon rate of each pixel is predicted directly using each 2x2 pixel block of the Bayer array in the RAW image, as shown in Figure 1. Figure 3 Shown in the left part.

[0075] After the incident photon rate of each pixel is obtained, the incident photon rate and the calibrated noise parameters are used to synthesize the single-photon simulation data by simulating the complete imaging process of the single-photon camera in the simulation image generation stage, such as Figure 3 As shown in the right part. Specifically, different illumination scaling factors are first used to multiply the incident photon rate estimated in the scene information simulation stage to simulate scenes of different brightness. Then, different integration times are used to multiply the calibrated dark count rate to simulate different dark count noise intensities at each integration time. Finally, the scaled incident photon rate and dark count rate are added, the Poisson mean of the obtained non-homogeneous Poisson process is sampled, and the after-pulse probability is calculated in each frame and the influence of crosstalk is considered.

[0076] Step 4: Use the generated paired simulation data to train a deep learning image reconstruction model and use the trained network for real single-photon image reconstruction tasks.

[0077] Based on the deep learning method and Transformer module structure, an end-to-end image reconstruction model is established, in which the spatial-channel Transformer block is used as its basic module, and the structural feature enhancement module is used as the feedforward layer of the Transformer block; the network is trained using simulated data; after the training is completed, the real noisy data is input into the image reconstruction model, and the reconstructed result image is calculated.

[0078] After the channel features and spatial features enter the SCTB, a 1x1 convolution is first used to integrate the input features along the channel dimension, and then the feature map is evenly divided into two blocks along the channel dimension, and the channel domain Transformer block and the spatial domain Transformer block are used to perform self-attention calculations in the spatial domain and channel domain for these two sub-feature maps respectively; among them, the sliding window mechanism of Swin-Transformer is used in the spatial domain Transformer block; in the channel domain Transformer block, a channel self-attention mechanism is used that only performs global perception along the channel and does not care about the specific position information; after passing through the channel domain Transformer and the spatial domain Transformer, the feature maps calculated by the two modules are spliced ​​along the channel dimension, and a 1x1 convolution is used for gated fusion.

[0079] The structural feature enhancement module uses a symmetrical dual-path channel-by-channel convolution to extract local structural information with stronger nonlinearity and expression capabilities. The module first uses a 1x1 convolution to increase the channel dimension of the input feature. times, and then use the GELU activation function to process the upgraded features, and use a 1x1 convolution to perform preliminary feature extraction; then, two branches are used to perform different convolution methods respectively, one of which uses 3x3 channel-by-channel convolution to extract local features, and the other branch uses 3x3 channel-by-channel dilated convolution to extract feature information within a larger receptive field; in these two branches, the structural feature enhancement module uses two symmetrical channel-by-channel convolutions to extract features, and uses element-level multiplication to fuse the features; finally, the structural enhancement feature map that has passed the above two branches is spliced ​​along the channel dimension, and a 1x1 convolution is used to fuse features of different spatial scales, and the number of channels is restored to the number of channels of the input feature map, so as to obtain the final calculation result.

[0080] First, we introduce the logarithmic loss The loss is calculated in the logarithmic domain to increase the difference between different pixel values, that is: in, represents the reconstructed image, represents the true value image, , Represent the width and height of the image respectively; then, LPIPS loss and SSIM loss are introduced; among them, LPIPS loss is used to utilize the perception ability of the neural network to ensure the similarity of high-level structures, and SSIM loss is used to ensure the similarity of local structures by calculating the statistical information of brightness, contrast and structure; edge loss is introduced ,use Direction and The Sobel operators of the directions are denoted as and , to extract the edge information of the reconstructed image and the real image, and calculate the loss of the extracted edge information to guide the model to pay more attention to the recovery of high-frequency edges; the edge loss is calculated as: Combine the above loss functions to get the complete loss function : in, Respectively represent the weight of each loss; represents the Charbonnier loss, Represents SSIM loss; after training, the real noisy data taken by the single-photon camera is input into the image reconstruction model, and the reconstructed result image is calculated.

[0081] It includes a single-photon data shooting module, a single-photon noise calibration module, a single-photon data simulation module, an image reconstruction model training module and a single-photon imaging module; wherein the single-photon data shooting module is used to use a single-photon camera to shoot binary single-photon data, including noise data for calibrating noise and scene data for imaging; the single-photon noise calibration module is used to complete the calibration process of each noise source parameter of the single-photon camera according to the captured real single-photon noise data; the single-photon data simulation module is used to use the RAW image to synthesize simulation data according to the calibrated noise parameters; the image reconstruction model training module is used to use the synthesized simulation data to complete the training process of the image reconstruction model; the single-photon The imaging module is used to use the trained image reconstruction model to reconstruct the real scene data captured by the single-photon data shooting module to remove the noise and improve the image quality; the connection relationship between the above modules is: the output end of the single-photon data shooting module is connected to the input end of the single-photon noise calibration module and the single-photon imaging module; the output end of the single-photon noise calibration module is connected to the input end of the single-photon data simulation module; the output end of the single-photon data simulation module is connected to the input end of the image reconstruction model training module; the output end of the image reconstruction model training module is connected to the input end of the single-photon imaging module; the output end of the single-photon imaging module is connected to the output high-quality single-photon image.

Claims

1. A single-photon image noise modeling enhancement method based on physical principles, characterized in that: The following steps are involved: Step 1: Based on various noise physics principles, establish a complete single-photon camera noise model and design a noise parameter calibration scheme; Firstly, the possible noise sources in the imaging process of the single-photon camera are identified as the incident photon shot noise and photon detection quantum efficiency related to the incident photon signal, as well as the dark count noise, after-pulse noise and crosstalk noise related to the device material of the single-photon camera itself; the probability of the time interval between two adjacent photon detection events is mathematically derived, and a noise model is formed; Then, a correction term is proposed based on the maximum frame number limit of a single shot in actual sampling to make the noise model conform to the actual situation; Finally, combining the image properties of the noise model, a scheme is designed to decouple and calibrate each noise source separately; In step 1, the probability that the output count of the nth frame is 1 after the output count of a certain pixel of the single-photon camera is 1 is: Among them, P nc (n) represents the probability that no avalanche current is detected in the nth frame. According to the statistical law obeyed by each noise source, the probability is described in detail as: Where η is the quantum efficiency of the pixel in detecting incident photons; P in represents the probability that the incident photon does not trigger avalanche breakdown, P dcr P represents the probability that dark counts do not trigger avalanche breakdown. ap (n) represents the probability of a post-pulse occurring in the nth frame after avalanche breakdown, P ct represents the probability of crosstalk; λ in is the incident photon rate, λ dcr represents the dark count rate, σ represents the neighborhood of the single photon sensor; p i represents the probability of crosstalk between the pixel and the i-th single-photon sensor in the neighborhood; λ i represents the sum of the dark count rate and the effective incident photon rate of the i-th single-photon sensor in the neighborhood; Δt represents the integration time; Substituting equation 2 into equation 1, we get the probability that the time interval between two adjacent photon detection events is exactly n: Equation 3 is the single photon noise model; The correction term is the ratio of the number of integration times that can produce a time interval of length n to the total number of time intervals: Where N is the total number of frames captured by the single-photon camera in a single shot; this correction term is added to Equation 3 as a multiplicative factor, and the logarithm of Equation 3 is taken to decouple the various parts of the multiplicative coupling, resulting in: In formula 5, the first term is a correction term, the third term is a linear term, and the second and fourth terms are both nonlinear terms; Step 2: Use a single-photon camera to capture noise data and perform data preprocessing. According to the real noise data after preprocessing, calibrate the parameters of each noise source; Use a single-photon camera to capture real noise data for subsequent noise calibration process; In the data preprocessing process, the histogram of the time interval between two adjacent photon detection events, the total number of counts of each sensor, and the number of simultaneous counts of each sensor and its neighboring sensors are extracted and saved from the real single photon data; After preprocessing the real single-photon data, the data required for noise calibration are obtained. Using these data and the noise calibration scheme, the calibration of the noise source parameters of the single-photon camera is completed. Step 3: Generate simulation data based on the noise parameter calibration results; First, the incident photon rate of each pixel in the scene is estimated using the RAW image. Then, the complete imaging process of the single-photon camera is simulated in combination with the calibrated noise parameters. Finally, single-photon simulation data with different brightness, different noise intensity, and different bit depth are generated. Step 4: Use the generated paired simulation data to train a deep learning image reconstruction model, and use the trained network for real single-photon image reconstruction tasks.

2. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 1, characterized in that: A calibration method is designed for the crosstalk probability. The crosstalk noise probability p is calculated without collecting additional data. i Independent calibration to achieve decoupling of crosstalk noise and non-homogeneous Poisson noise; Assume that the total number of count frames of single-photon sensors i and j in N frames are N i and N j , the number of simultaneous counting frames due to crosstalk noise is N ct At this time, the number of simultaneous counting frames N caused by the non-homogeneous Poisson process unrelated to crosstalk u and the number of frames where crosstalk occurs N ct Described as: According to formula 6, we get N u and N ct ; The number of simultaneous counting frames N due to crosstalk noise ct The expectation E{N ct }for: Among them, μ i and μ j represents the sum of the effective incident photon rate and the dark count rate of single-photon sensors i and j, respectively, P ct represents the crosstalk probability between single-photon sensors i and j; Solving equation 7 yields the crosstalk probability P between two single-photon sensors: ct .

3. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 1, characterized in that: In step 2, for the calibration of dark count noise, a linear fitting method based on RANSAC is used; for the calibration of post-pulse noise, a nonlinear fitting method is used; for the calibration of crosstalk noise, it is completed by solving transcendental equations.

4. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 1, characterized in that: Step 3 includes scene information simulation and simulation image generation; The incident photon rate of each pixel is predicted using each 2×2 pixel block of the Bayer array in the RAW image. Then, in the simulation image generation stage, the incident photon rate and the calibrated noise parameters are used to synthesize single-photon simulation data by simulating the complete imaging process of a single-photon camera. First, different illumination scaling factors are used to multiply the incident photon rate estimated in the scene information simulation stage to simulate scenes of different brightness; Then, different integration times are used to multiply the calibrated dark count rate to simulate different dark count noise intensities at different integration times; Finally, the scaled incident photon rate and dark count rate are added together to obtain the Poisson mean of the non-homogeneous Poisson process, and the after-pulse probability is calculated in each frame while considering the influence of crosstalk. By cyclically sampling each frame, single-photon data of different frame numbers are obtained.

5. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 1, characterized in that: In step 4, an end-to-end image reconstruction model is established based on the deep learning method and the Transformer module structure, wherein the spatial-channel Transformer block is used as its basic module and the structural feature enhancement module is used as the feed-forward layer of the Transformer block; The network is trained using simulated data; after the training is completed, the real noisy data is input into the image reconstruction model and the reconstructed result image is calculated.

6. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 5, characterized in that: After the channel features and spatial features enter the SCTB, a 1×1 convolution is first used to integrate the input features along the channel dimension, and then the feature map is evenly divided into two blocks along the channel dimension. The channel domain Transformer block and the spatial domain Transformer block are used to perform self-attention calculations in the spatial domain and channel domain for these two sub-feature maps respectively; the sliding window mechanism of Swin-Transformer is used in the spatial domain Transformer block; in the channel domain Transformer block, the channel self-attention mechanism that only performs global perception along the channel without caring about the specific position information is used; After passing through the channel domain Transformer and the spatial domain Transformer, the feature maps calculated by the two modules are concatenated along the channel dimension and gated fused using a 1×1 convolution.

7. The single photon image noise modeling enhancement method based on physical principles as claimed in claim 5, characterized in that: The structural feature enhancement module uses symmetrical dual-path channel-by-channel convolution to extract local structural information; The module first uses a 1×1 convolution to increase the channel dimension of the input feature by k times, then uses the GELU activation function to process the features after dimensionality increase, and uses a 1×1 convolution to perform preliminary feature extraction; then, two branches are used to perform different convolution methods respectively, one of which uses 3×3 channel-by-channel convolution to extract local features, and the other branch uses 3×3 channel-by-channel dilated convolution to extract feature information within a larger receptive field; in these two branches, the structural feature enhancement module uses two symmetrical channel-by-channel convolutions to extract features, and uses element-level multiplication to fuse the features; finally, the structural enhancement feature map that has passed the above two branches is spliced ​​along the channel dimension, and a 1×1 convolution is used to fuse features of different spatial scales, and the number of channels is restored to the number of channels of the input feature map, thereby obtaining the final calculation result.

8. The single photon image noise modeling enhancement method based on physical principle as claimed in claim 5, characterized in that: First, we introduce the logarithmic loss The loss is calculated in the logarithmic domain to increase the difference between different pixel values, that is: Among them, I r represents the reconstructed image, I gt Represents the true value image, W and H represent the width and height of the image respectively; Then, LPIPS loss and SSIM loss are introduced; LPIPS loss is used to utilize the perception ability of neural networks to ensure the similarity of high-level structures, and SSIM loss is used to ensure the similarity of local structures by calculating the statistical information of brightness, contrast and structure; edge loss is introduced The Sobel operators in the x-direction and y-direction are denoted as G x and G y , to extract the edge information of the reconstructed image and the real image, and calculate the loss for the extracted edge information to guide the model to pay more attention to the restoration of high-frequency edges; The marginal loss is calculated as: Combine the above loss functions to get the complete loss function Among them, λ1, λ2, λ3, λ4, and λ5 represent the weights of each loss respectively; It indicates Charbonnier loss; After the training is completed, the real noisy data taken by the single-photon camera is input into the image reconstruction model, and the reconstructed result image is calculated.

9. A single photon image noise modeling enhancement system based on physical principles for implementing the method of claim 1, characterized in that: It includes a single-photon data shooting module, a single-photon noise calibration module, a single-photon data simulation module, an image reconstruction model training module and a single-photon imaging module; The single-photon data capturing module is used to capture binary single-photon data using a single-photon camera, including noise data for calibrating noise and scene data for imaging; The single photon noise calibration module is used to complete the calibration process of each noise source parameter of the single photon camera according to the real single photon noise data captured; The single photon data simulation module is used to synthesize simulation data using RAW images according to the calibrated noise parameters; The image reconstruction model training module is used to complete the training process of the image reconstruction model using the synthesized simulation data; The single-photon imaging module is used to reconstruct the real scene data captured by the single-photon data capturing module using the trained image reconstruction model to remove noise and improve the image quality. The connection relationship between the above modules is: The output end of the single-photon data shooting module is connected to the input end of the single-photon noise calibration module and the single-photon imaging module; the output end of the single-photon noise calibration module is connected to the input end of the single-photon data simulation module; the output end of the single-photon data simulation module is connected to the input end of the image reconstruction model training module; the output end of the image reconstruction model training module is connected to the input end of the single-photon imaging module; the output end of the single-photon imaging module is connected to the output high-quality single-photon image.

Images