A few-photon imaging method based on deep learning

Abstract

The present application discloses a few-photon imaging method based on deep learning, which is a few-photon counting scattering reconstruction method. The training target can be sampled only once under high photon counting conditions, and then a simulation algorithm can be used to generate a training database under different few-photon counting conditions for the training of a deep learning neural network. After the training is completed, the network parameters are fixed, and this neural network can be used for scattering image reconstruction. The acquired new scattering image under few-photon counting is used as the input of the deep learning network, and the output is a clear image. This method simplifies the complex sampling process of the deep learning method in the field of low-light imaging, while reducing the requirements for the sensitivity of the sampling equipment, saving equipment costs and time costs. And for different few-photon counting restoration samples, this method does not need to re-sample to generate a training database, which increases the robustness of the deep learning method in the field of few-photon counting imaging.

Description

Technical Field

[0001] The present application relates to the field of computational imaging and image processing, and specifically to a few-photon imaging method based on deep learning. Background Art

[0002] Few-photon counting scattering imaging reconstruction has important applications in real life, such as nighttime detection, astronomical remote sensing, biomedical imaging, etc.

[0003] Traditional imaging equipment generally takes 10 12 The number of photons, averaged per pixel, is 10 for a high-quality array detector. 5 Photon count. However, in many actual low-light scenarios, the photon count value is difficult to meet this requirement. Especially in extremely low light and limited exposure time, the effective photon count can even reach only a few. This application focuses on such low-photon count scenarios.

[0004] When the number of photons is sufficient, the signal detected by ordinary detectors is an analog quantity. Its optical signal contains a large number of photons, and the superposition of these photons is the light intensity obtained by detection. The method of obtaining images by recording the light intensity at each position on the detection target is called the analog method. However, as the light intensity of the detection target decays, the light intensity gradually becomes a pulse signal. Especially when the light intensity decays to a single photon condition, the signal becomes a discrete pulse signal with a small number of pulses. Single photons are generally regarded as the limit of detection and the smallest unit of energy that cannot be further divided. Therefore, under the condition of few photon counting, the signal presents particle characteristics. In this case, the light is recorded as a single photon, and its spatial position is determined while detecting a single photon, and two-dimensional photon counting detection is performed. This is the basis of photon counting imaging.

[0005] A single-photon camera (SPC) is a two-dimensional array detector, each pixel of which is equivalent to an independent point detector. Its sensitivity can break through the shot noise limit, so it has been widely used for few-photon counting detection in recent years. In particular, this camera does not directly capture a grayscale image. Each pixel records the number of detected photons, and the randomness of photon emission and detection is the main source of noise in the imaging system. The noise distribution under the condition of few-photon counting conforms to the Poisson distribution. The required grayscale image can be obtained by normalizing the number of photons. However, the packaging process of single-photon detectors is difficult, the pixel spacing is large, and only low-resolution images can be obtained. But this is also equivalent to downsampling the acquired high-resolution image.

[0006] The combination of traditional physics research problems and deep learning has brought some advantages, especially in image processing. Traditional scattering imaging only focuses on the single problem of scattering, and does not pay attention to the complex situation where the two limitations of few photon counting and scattering exist at the same time. With the loss of information caused by the weakening of light intensity, imaging is more difficult in this case. In addition, the traditional scattering imaging method requires multiple scans of the target, and the algorithm depends on the detection model. Different processing methods are required for detection in different scenarios. In particular, it is difficult for traditional methods to restore images for slightly thick scattering media. The experimental operation and algorithm are complex, the detection is multiple times, and the image restoration quality is average. In recent years, deep learning methods have been widely used in the field of few photon counting scattering imaging, and good imaging results have been achieved. However, the deep learning methods in the prior art also have many defects, such as the need to sample a large amount of training data for pre-training in order to have the ability to image. However, the photon count of the training data needs to be consistent with the scattering pattern to be imaged, so that the deep learning network can correctly learn the characteristics of the scattering pattern. When the photon count of the scattering pattern to be imaged changes significantly, the deep learning methods in the prior art must resample the training database matching the photon count to maintain the learning ability of the neural network. Therefore, this places extremely high demands on the sensitivity of the sampling equipment and the long-term stability of the ambient light conditions, and is inefficient and not robust, which is not conducive to engineering. Therefore, technicians in this field are committed to developing a few-photon imaging method based on deep learning to solve the technical problems existing in the prior art. Summary of the invention

[0007] To achieve the above objectives, the present application provides a few-photon imaging method based on deep learning, which specifically includes the following steps:

[0008] Step 1: adjusting a scattering imaging device so that the scattering imaging device can obtain a scattering image;

[0009] Step 2: Adjust the light source intensity and exposure time to obtain a scattering pattern with high photon counts;

[0010] Step 3, generating a scattering map with low photon counts for training by using a Poisson algorithm;

[0011] Step 4, using the plurality of scattering images with low photon counts for training generated in step 3 as a training set for training a neural network, and after the training is completed, fixing the parameters to determine the neural network for scattering imaging reconstruction;

[0012] Step 5, adjusting the light source intensity and exposure time to obtain a low photon count scattering image to be restored under weak light;

[0013] Step 6: Using the neural network for scattering imaging reconstruction, reconstruct the low photon count scattering image to be restored obtained in step 5.

[0014] Furthermore, in step 1, the sampling formula of the scatter diagram satisfies the following formula:

[0015]

[0016] in, represents the average number of photons per unit time, and hν represents the energy of a photon. P(x, y) represents the scattering image sampled during the exposure time Δt. Possion[] represents the Poisson process of photon detection.

[0017] Furthermore, in step 1, the total number of photons n detected by each pixel in each scattering pattern dj It conforms to the Poisson random distribution probability.

[0018] Furthermore, the total number of photons n dj Satisfies the following formula:

[0019]

[0020] in, Represents the average number of photons at exposure time Δt.

[0021] Furthermore, in step 3, the exposure time Δt of each pixel and the total number of photons n dj , a simulation is performed to generate a scattering graph with the same photon count as the low photon count scattering graph to be restored, which becomes the scattering graph with the low photon count for training.

[0022] Furthermore, step 3 specifically includes the following steps:

[0023] Step 3.1: Select a series of images N i , i=1,2,3,...;

[0024] Step 3.2: For N i Each of them is sampled at high photon count after exposure time t to obtain the scattering pattern S i ;

[0025] Step 3.3: Scattering plot S i Each pixel in the ij ;

[0026] Step 3.4: Set the number of photons P ij As n dj , exposure time t as Δt, P ij / t as Substitute into the formula as claimed in claim 4;

[0027] Step 3.5: reducing Δt and randomly generating a plurality of random pixel values ​​according to the formula as claimed in claim 4;

[0028] Step 3.6: The matrix composed of all the random pixel values ​​constitutes the low photon count scattering map used for training.

[0029] Furthermore, in step 3.5, Δt is reduced to 0.001 times of the original value.

[0030] Furthermore, in step 1, the scattering imaging device includes a light source, a spatial light modulation device, a lens, frosted glass, a detector and a processor module. The modulated light is emitted from the light source, passes through the spatial light modulation device, the lens, the frosted glass in sequence, and reaches the detector. The detector is data connected to the processor module, so that the processor module performs imaging restoration on the data collected by the detector.

[0031] Furthermore, the spatial light modulation device is used to load a target image.

[0032] Furthermore, the detector is a single-photon camera for counting photons, the sensitivity of which can exceed the limit of shot noise.

[0033] Compared with the prior art, the technical solution of this application has at least the following technical advantages:

[0034] 1. Applicable to low photon count situations. When the light is weak enough to show particle characteristics, certain effective information can still be obtained through counting imaging;

[0035] 2. The algorithm simplifies the process of sampling training samples when deep learning methods are used for scattering imaging. Only a small number of high-photon count scattering images need to be sampled to quickly generate a large number of weak-light scattering images.

[0036] 3. The algorithm avoids the need for deep learning methods to resample the training set when the photon counts of the scattering image to be recovered change greatly, and is more robust;

[0037] 4. The algorithm reduces the requirements for camera sensitivity and the requirement for ambient light to remain stable for a long time when collecting low-photon count scatter imaging training samples;

[0038] 5. More training samples improve the quality of image reconstruction using deep learning methods for low photon count scattering imaging.

[0039] The concept, specific structure and technical effects of the present application will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1A schematic diagram of a scattering imaging system according to an embodiment of the present application;

[0041] Figure 2 This is one of the target original images in the training set in one embodiment of the present application, the pixel size is 32*32, and the high photon count condition is an average of 350 photons per pixel;

[0042] Figure 3 for Figure 2 The image shown is a scattering pattern obtained after scattering under high photon count conditions, with a pixel size of 32*32;

[0043] Figure 4 for Figure 3 The medium and high photon count scattering diagram is a low photon count scattering diagram generated by Poisson simulation using formula (2), with a pixel size of 32*32 and a low photon count condition of 0.3 photons per pixel on average;

[0044] Figure 5 for Figure 3 The medium and high photon count scattering diagram uses formula (2) to perform Poisson simulation to generate another low photon count scattering diagram, with a pixel size of 32*32 and a low photon count condition of 1.5 photons per pixel on average;

[0045] Figure 6 It is one of the target original images in the test set that is disjoint from the training set in one embodiment of the present application, and the pixel size is 32*32;

[0046] Figure 7 for Figure 6 The image shown is a scattering map obtained after scattering under low photon count conditions with an average of 0.3 photons per pixel, and the pixels are 32*32;

[0047] Figure 8 for Figure 6 The low photon count scattering image shown is a scattering image obtained after scattering under low photon count conditions with an average number of photons per pixel of 1.5, and the pixels are 32*32;

[0048] Fig. 9 for Figure 7 The low photon count scattering image shown is a clear image reconstructed by the trained neural network, with a pixel size of 32*32;

[0049] Fig.10 for Figure 8 The low photon count scattering image shown is a clear image reconstructed by the trained neural network, with a pixel size of 32*32. DETAILED DESCRIPTION

[0050] The following describes several preferred embodiments of the present application with reference to the drawings in the specification, making the technical content clearer and easier to understand. The present application can be embodied in many different forms of embodiments, and the protection scope of the present application is not limited to the embodiments mentioned in the text.

[0051] In the drawings, components with the same structure are indicated by the same numerical labels, and components with similar structures or functions are indicated by similar numerical labels. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0052] Embodiment 1

[0053] This embodiment provides a few-photon imaging method based on deep learning, which specifically includes the following steps:

[0054] Step 1: Adjust the scattering imaging device so that the scattering imaging device can obtain a scattering image.

[0055] Step 2: Adjust the light source intensity and exposure time to obtain a scattering pattern with high photon counts.

[0056] The scattering imaging device used in this embodiment is as follows Figure 1 As shown, it includes a light source 1, a spatial light modulation device 2, a lens 3, a frosted glass 4, a detector 5 and a processor module 6. The modulated light is emitted from the light source 1, passes through the spatial light modulation device 2, the lens 3, the frosted glass 4 in sequence, and reaches the detector 5. The lens 3 is used to converge the reflected light of the spatial light modulator 2, the frosted glass 4 is used to scatter the light converged by the lens 3, the detector 5 is used to detect the scattered light intensity and count the number of photons, and the processor module 6 is used to perform data processing and scattered imaging recovery on the data collected by the detector 5. Preferably, the detector 5 can collect light field intensity values ​​at the same frequency as the spatial light modulation device 2. Preferably, the light intensity of the light source 1 is adjustable. Preferably, the detector 5 is a single-photon camera for counting photons with a sensitivity that can break through the shot noise limit.

[0057] The specific steps of the scattering imaging device to obtain the scattering map are:

[0058] (1) The spatial light modulation device 2 is loaded with a target image, and the light emitted by the light source 1 is irradiated onto the target image of the spatial light modulator 2;

[0059] (2) lens 3 converges the reflected light of the target in spatial light modulator 2;

[0060] (3) frosted glass 4 scatters the light focused by lens 3;

[0061] (4) the processor module 6 collects the number of photons of the reflected light detected by the detector 5;

[0062] (5) A grayscale image is obtained by normalizing the number of photons obtained by processor 6.

[0063] In this embodiment, the light intensity of the light source is 97 nanowatts, the SPC exposure time is adjusted to 1500 microseconds, and 3000 original clear images and high photon count scattering images are obtained.

[0064] Step 3: Generate a scattering map with low photon counts for training using the Poisson algorithm.

[0065] The sampling process under the condition of low photon count is a Poisson process, and the distribution law of light intensity at each pixel conforms to Poisson's law. Although the detection under the condition of low photon count is random due to Poisson noise, the detection under the condition of sufficient photons is relatively stable. If any other noise is ignored, the sampling formula of the scattering image can be given by the following formula (1):

[0066]

[0067] in, represents the average number of photons per unit time, and hν represents the energy of a photon. P(x, y) represents the scattering image sampled during the exposure time Δt. Possion[] represents the Poisson process of photon detection.

[0068] If the total number of photons detected per pixel in each image is expressed as an integer n dj , then the average number of photons per pixel under exposure time Δt can be expressed as Because n dj It conforms to the Poisson random distribution probability, and its distribution formula is given by the following formula (2):

[0069]

[0070] Based on formula (2), once the average number of photons per unit time is known and exposure time Δt, any possible distribution of the photon number scattering pattern on the detector can be simulated. It is well known that in a higher illumination range, the variance introduced by photon shot noise accounts for a smaller percentage of the entire signal. Therefore, by using the measured high photon count scattering map, it is possible to approximate that the photon count value at each pixel of the scattering map is the average number of photons at that pixel. Based on this feature and the characteristics of deep learning neural networks (DL for short), there is no need to measure the real training samples at the same photon count level as the test set. A large number of different training data at low photon count levels can be simulated by measuring only one scattering map at high photon count for each training target.

[0071] In view of the problems existing in the prior art, this method provides a new idea for obtaining training data under low photon counts. During the training phase of the network, only a small number of high photon count spots can be sampled. Since the sampling time Δt of the SPC is known, the total number of photons n in the scattering pattern can also be easily calculated. dj . Therefore, Δt can be flexibly adjusted, and the Poisson simulation algorithm can be used to simulate a large number of scattering graphs with the same photon counts as the test data. This method can skip the complex process of obtaining a training set under low photon numbers, reduce the requirements for sampling equipment accuracy and sampling environment stability, and is more conducive to practical applications. On the other hand, after the scattering image of the target is detected by a single-photon camera, a detected weak light scattering image is directly reconstructed through the algorithm, which can reduce the complexity of the experimental operation. The algorithm does not depend on the model. For thin and thick scattering media, the same algorithm can be used to process them, which improves the generalization ability of post-processing.

[0072] In addition, it is worth pointing out that imaging after passing through a scattering medium is a relatively complex imaging scenario. For imaging in other common scenarios, there are mature physical models that can calculate images under high photon count conditions without the need for actual sampling. At this time, combined with the simulation algorithm of the present application, a large number of images under low photon counts can be generated for deep learning network training without actual sampling. This has very important engineering significance. Therefore, the technology of the present application has important significance in the fields of military reconnaissance, detection, remote sensing, and biomedical imaging.

[0073] In this embodiment, the total exposure time Δt and the total number of photons n of each pixel are known. dj , a scattering map with the same photon counts as a large number of scattering maps to be recovered can be simulated and generated as a training data set. The specific steps are as follows:

[0074] Step 3.1: Select a series of images N i , i=1,2,3,...;

[0075] Step 3.2: For N i Each of them is sampled at high photon count after exposure time t to obtain the scattering pattern S i ;

[0076] Step 3.3: Scattering plot S i Each pixel in the ij ;

[0077] Step 3.4: Set the number of photons P ij As n dj , exposure time t as Δt, P ij / t as Substitute into formula (2);

[0078] Step 3.5: Reduce Δt and randomly generate multiple random pixel values ​​according to formula (2);

[0079] Step 3.6: The matrix of all random pixel values ​​constitutes the low photon count scattering map used for training.

[0080] Specifically, in this embodiment, assuming that the number of photons in the target low-light image is 0.001 times that of the high-photon count, it is only necessary to reduce Δt in formula (2) to 0.001 times of the original value, and then randomly generate a value for each pixel according to the probability distribution in formula (2) as the pixel value generated by simulation. The matrix composed of all pixel values ​​is the corresponding low-photon count scattering map value generated by simulation. Since the value of each pixel point is randomly generated according to probability, the same high-photon count scattering map can be simulated multiple times to generate multiple low-photon count scattering maps, thereby avoiding the complicated repeated sampling process. In addition, the photon count value of the low-photon count generated by simulation can also be generated and changed by adjusting Δt, which increases the practicality and robustness of the method.

[0081] In this embodiment, in order to form a data set for neural network training, 3,000 high-photon count scattering images are collected, and 30,000 low-photon count scattering images are generated through Poisson algorithm simulation.

[0082] Step 4: Use the multiple scattering images with low photon counts for training generated in step 3 as training sets to train the neural network. After the training is completed, the parameters of the neural network used for scattering imaging reconstruction are fixed.

[0083] In this embodiment, preferably, the weak light scattering image is used as the input of the neural network, and the output is a clear image reconstructed by the model. The parameters are automatically adjusted by minimizing the mean square error between the clear image reconstructed by the model and the original clear image, thereby optimizing the neural network model.

[0084] Step 5: Adjust the light source intensity and exposure time to obtain a low photon count scattering image to be recovered under weak light.

[0085] Specifically, in this embodiment, the exposure time Δt is adjusted to 2 microseconds, and the intensity of the light emitted by the light source is reduced to obtain a low photon count scattering image to be restored under weak light.

[0086] Step 6: Use a neural network for scattering imaging reconstruction to reconstruct the low photon count scattering image to be restored obtained in step 5, and its output is the high-quality image required for reconstruction.

[0087] Figure 2-Figure 10These are some scattering diagrams for reconstructing a few-photon counting scattering image using the method of this embodiment, where:

[0088] Figure 2 This is one of the target original images in the training set of this embodiment, with a pixel size of 32*32 and a high photon count condition of 350 photons per pixel on average.

[0089] Figure 3 for Figure 2 The image shown is a scattering pattern obtained after scattering under high photon count conditions, with a pixel size of 32*32.

[0090] Figure 4 for Figure 3 The medium and high photon count scattering diagram is a low photon count scattering diagram generated by Poisson simulation using formula (2). The pixel size is 32*32, and the low photon count condition is an average of 0.3 photons per pixel.

[0091] Figure 5 for Figure 3 The medium and high photon count scattering diagram uses formula (2) to perform Poisson simulation to generate another low photon count scattering diagram. The pixel size is 32*32, and the low photon count condition is an average of 1.5 photons per pixel.

[0092] Figure 6 It is one of the target original images in the test set that does not intersect with the training set in this embodiment, and the pixel size is 32*32.

[0093] Figure 7 for Figure 6 The image shown is a scattering diagram obtained after scattering under low photon count conditions with an average of 0.3 photons per pixel, and the pixels are 32*32.

[0094] Figure 8 for Figure 6 The low photon count scattering image shown is a scattering image obtained after scattering under low photon count conditions with an average of 1.5 photons per pixel, and the pixels are 32*32.

[0095] Fig. 9 for Figure 7 The low photon count scattering image shown is a clear image reconstructed by the trained neural network, with a pixel size of 32*32.

[0096] Fig.10 for Figure 8 The low photon count scattering image shown is a clear image reconstructed by the trained neural network, with a pixel size of 32*32.

[0097] The preferred specific embodiments of the present application are described in detail above. It should be understood that ordinary technology in the field can make many modifications and changes according to the concept of the present application without creative work. Therefore, any technical solution that can be obtained by a technician in the technical field based on the concept of the present application through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A few-photon imaging method based on deep learning, characterized in that: The specific steps include: Step 1: adjusting the scattering imaging device so that the scattering imaging device can obtain a scattering image. The total number of photons n detected by each pixel in the scattering map dj It is consistent with the Poisson random distribution probability, And satisfy the following formula: in, represents the average number of photons under exposure time Δy; Step 2: Adjust the light source intensity and exposure time to obtain a scattering pattern with high photon counts; Step 3: Generate a scattering map with low photon counts for training using the Poisson algorithm. The exposure time Δt of each pixel and the total number of photons n dj , simulate and generate a scattering graph with the same photon count as the low photon count scattering graph to be restored, which becomes the scattering graph with the low photon count for training, The specific steps include: Step 3.1: Select a series of images N i , i=1,2,3,...; Step 3.2: For N i Each of them is sampled at high photon count after exposure time t to obtain the scattering pattern S i ; Step 3.3: Scattering plot S i Each pixel in the ij ; Step 3.4: Set the number of photons P ij As n dj , exposure time t as Δt, P ij / t as Substitute into the formula described in step 1; Step 3.5: Reduce Δt and randomly generate multiple random pixel values ​​according to the formula described in step 1; Step 3.6: The matrix composed of all the random pixel values ​​constitutes the scattering map of low photon counts for training; Step 4, using the plurality of scattering images with low photon counts for training generated in step 3 as a training set for training a neural network, and after the training is completed, fixing the parameters to determine the neural network for scattering imaging reconstruction; Step 5, adjusting the light source intensity and exposure time to obtain a low photon count scattering image to be restored under weak light; Step 6: Using the neural network for scattering imaging reconstruction, reconstruct the low photon count scattering image to be restored obtained in step 5.

2. The deep learning-based few-photon imaging method according to claim 1, characterized in that: In step 1, the sampling formula of the scatter diagram satisfies the following formula: in, represents the average number of photons per unit time, hν represents the energy of a photon, P(x,y) represents the scattering image sampled in the exposure time Δt, and Possion[] represents the Poisson process of photon detection.

3. The deep learning-based few-photon imaging method according to claim 2, characterized in that: In step 3.5, Δt is reduced to 0.001 times of the original value.

4. The deep learning-based few-photon imaging method according to claim 3, characterized in that: In step 1, the scattering imaging device includes a light source, a spatial light modulation device, a lens, frosted glass, a detector and a processor module. The modulated light is emitted from the light source, passes through the spatial light modulation device, the lens, the frosted glass in sequence, and reaches the detector. The detector and the processor module are data connected, so that the processor module performs imaging restoration on the data collected by the detector.

5. The deep learning-based few-photon imaging method according to claim 4, characterized in that: The spatial light modulation device is used to load a target image.

6. The deep learning-based few-photon imaging method according to claim 4, characterized in that: The detector is a single-photon camera for counting photons, the sensitivity of which can exceed the limit of shot noise.

Images