Deep learning based detector motion scatter imaging device and method in extremely weak light environment

Abstract

The application discloses a detector motion scattering imaging device and method in an extremely weak light environment based on deep learning, relates to the imaging field under target motion in an extremely weak light environment, and comprises the following: a light source for generating an extremely weak light source; at least one scattering light path for performing scattering processing on the focused extremely weak light; a detector arranged on an adjusting device, the adjusting device being configured to have a motion mode in a set measurement range, and the adjusting device driving the detector to collect the extremely weak light processed by the scattering light path in multiple position states; the adjusting device generating position information in the multiple position states; and a processor receiving the light signal of the detector and performing analysis, and generating neural network data samples of deep learning corresponding to the multiple positions, the data samples comprising image information and position information corresponding to the image information. The application uses multiple positions of the neural network to jointly recover the speckle, acquires more environmental information, and is beneficial to the imaging precision of detection in a complex motion state of a target.

Description

Technical Field

[0001] This invention relates to the field of imaging, and more particularly to a detector motion scattering imaging device and method based on deep learning in extremely low light environments. Background Technology

[0002] Scattering imaging recovery under low-photon conditions has important applications in real life, including night vision, biomedical imaging, and satellite observation.

[0003] Traditional imaging devices typically require 10¹² photons to acquire a high-quality image. On average, a high-quality array detector needs to capture 10⁵ photons per pixel. However, in practical applications, it's difficult to achieve this photon count. Especially in extremely low light conditions and with limited exposure time, the effective photon count may only reach a few.

[0004] When the number of photons is sufficient, ordinary detectors detect analog signals, which contain a large number of photons. The superposition of these photons constitutes the detected light intensity. The method of acquiring an image by recording the light intensity at each location on the target is called the analog method. However, as the light intensity at the target decays, it gradually transforms into a pulse signal. Especially when the light intensity weakens to the single-photon condition, the signal becomes a discrete pulse signal with a very small number of pulses. A single photon is generally considered the detection limit, the smallest unit of energy that cannot be further divided. Therefore, under low-photon counting conditions, the signal exhibits particle characteristics. In this case, recording light as a single photon, determining its spatial position while detecting a single photon, and performing two-dimensional photon counting detection is the basis of photon counting imaging.

[0005] A single-photon camera (SPC) is a two-dimensional array detector where each pixel has a single-photon avalanche diode (SPAD), essentially an independent point detector. In extremely low light conditions, photons break down the SPAD and are counted. Notably, the image produced by this camera is not a grayscale image; each pixel represents the number of photons falling on that pixel per unit time. The main source of noise in extremely low light is Poisson noise due to the particle nature of light. Normalizing the photon count yields the desired grayscale image. Furthermore, the resolution of single-photon detectors can be further improved by using the time-of-flight of photon sequences in SPC technology. Because its sensitivity can overcome the shot noise limit, and it possesses advantages such as good signal-to-noise ratio, low power consumption, high quantum efficiency, and small size, it is widely used in the field of extremely low light imaging.

[0006] The combination of traditional physics research problems and deep learning has yielded some advantages, especially in image processing. Traditional scattering imaging only focuses on the single problem of scattering, neglecting the complex situation where three constraints—few photon counts, scattering, and moving targets—simultaneously exist. In a few-photon environment, the detected speckle contains a large amount of shot noise, reducing the signal-to-noise ratio. Scattering imaging methods based on the wave nature of light struggle to recover speckle under few-photon detection. Furthermore, the transmission path changes when the target moves, and the scattered wavefront information it carries also differs. Imaging methods based on learning the scattering medium and modeling the transmission process cannot solve the dilemmas in these complex situations. Methods under strong light conditions require multiple scans and detections of the target, and the scattering process during motion and in the optical path further complicates imaging under scattering with relative motion between the detector and the target in low-photon conditions. In recent years, deep learning methods have been widely used in the field of extremely low-light imaging, achieving good imaging results. However, existing deep learning methods also have many drawbacks, such as the need to sample a large amount of training data for pre-training so that the neural network can learn the characteristics of the scattering patterns and thus acquire imaging capabilities. However, when the target is moving, the scattering medium area used is large, and the correlation between different positions is weak, making it impossible to learn effective information. Therefore, when the target's position changes significantly, existing deep learning methods require processing large amounts of data during training, and the learning ability of the neural network cannot be guaranteed. Therefore, those skilled in the art are dedicated to developing a motion detection imaging device and method based on deep learning in extremely low light environments to solve the problems existing in the prior art. Summary of the Invention

[0007] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is that traditional scattering imaging only focuses on the single problem of scattering, without considering the complex situation where three limitations of photon counting, scattering and detector motion coexist. Therefore, an apparatus and method for detection and imaging in extremely low light environment and under target motion are proposed to solve the above problems.

[0008] To achieve the above objectives, the present invention provides a detector motion scattering imaging device based on deep learning in extremely low light environments, characterized in that the device comprises:

[0009] Light source, used to generate extremely weak light sources;

[0010] A focusing component for focusing the optical path of the extremely weak light source;

[0011] At least one scattering optical path is used to perform scattering processing on the focused, extremely weak light;

[0012] A detector is mounted on an adjustment device, which is configured to move within a set measurement range, thereby driving the detector to collect extremely weak light after it has been processed by the scattered light path in multiple position states; the adjustment device generates position information in multiple position states.

[0013] The processor is configured to receive the optical signals from the detector and perform analysis, generating neural network data samples for deep learning at the multiple locations, wherein the data samples include image information and location information corresponding to the image information.

[0014] Furthermore, the detector and the light source acquire light field intensity values ​​at the same frequency.

[0015] Furthermore, the adjustment device is synchronized with the information acquisition time of the detector.

[0016] Furthermore, the at least one-stage scattering optical path includes a first frosted glass and a second frosted glass arranged sequentially along the optical path.

[0017] Furthermore, the detector is a single-photon camera.

[0018] This invention also discloses a method for motion scattering imaging of detectors in extremely low light environments based on deep learning, characterized in that the method includes:

[0019] Step 1: Construct an optical path with a scattering medium, adjust the light intensity and exposure time, and obtain detection imaging under extremely low light conditions;

[0020] Step 2: Obtain target scattering maps at N fixed locations within the detection range, and acquire scattering imaging maps and corresponding fixed location information;

[0021] Step 3: Train the neural network using the scattering image at each fixed position and the fixed position information respectively. After training, a neural network with the fixed position labels is formed.

[0022] Step 4: Use the above neural network to train and obtain scattering imaging images at other locations.

[0023] Furthermore,

[0024] The neural network in step 3 is optimized by using the weak light scattering image as the input of the neural network and outputting a clear image reconstructed by the model. The parameters are automatically adjusted by minimizing the mean square error between the clear image reconstructed by the model and the original clear image, thereby obtaining the neural network model.

[0025] Furthermore,

[0026] Step 4 specifically involves:

[0027] Step 4-1: Move the detector within the detection range to obtain a sampled image of the motion detection for testing and the displacement information of the guide rail where the detector is located;

[0028] Step 4-2: Assign the low-photon speckle to be recovered and the guide rail position pair from Step 4-1 to the area where the fixed position is located in Step 2;

[0029] Step 4-3: Using the neural network, reconstruct the speckle pattern of the pre-classified low-photon moving target to be recovered obtained in Step 4-2.

[0030] Furthermore, step 4-3 specifically includes:

[0031] Step 4-3-1: Use the sampling image obtained in step 4-1 and the location classification information obtained in step 4-2 through the neural network corresponding to step 3 to obtain the reconstruction target;

[0032] Step 4-3-2: Use the sampled image obtained in Step 4-1 through the single-location neural network obtained in Step 3 to obtain the reconstruction target;

[0033] Step 4-3-3: Compare and analyze the results of the neural network trained with the detector at a single location and the neural network that classifies and then recovers the detector at multiple locations, and output the results.

[0034] Furthermore, the exposure time in step 1 is 2 μs.

[0035] Technical effect

[0036] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention.

[0037] This invention first proposes a scattering imaging device and method for imaging in extremely weak light (low photon number) conditions, where the detector is in motion. It proposes pre-partitioning the detector location to reduce the number of training samples required for each training iteration, thus limiting subsequent training runs to a finite number of batches and reducing the overall training frequency. Using datasets from different locations for classification training effectively improves the learning of environmental information, increasing the training speed and overall training performance of the network within the same region.

[0038] Using low-photon speckle sampling maps as training data, a neural network is trained and stored for N specified locations corresponding to the original target images from the low-photon speckle sampling maps in extremely low light conditions. The neural network is a U-shaped network with additional skip connections. Skip connections are used between the corresponding layers of downsampling and upsampling in the neural network, and skip connections are also used between each layer in the convolution process to increase the correlation between layers. This reduces the number of samples required for training a single neural network. It avoids the process of resampling the speckle at the current location for deep learning when the detector moves, resulting in greater robustness. Overall, more training samples are used, improving the quality of scattering imaging reconstruction and tracking of deep learning methods in extremely low light environments. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the composition structure of an imaging device according to an embodiment of this application;

[0040] Figure 2 This is a schematic flowchart of an imaging method according to an embodiment of this application;

[0041] Figure 3 This is one of the target original images in the training set in one embodiment of this application, with a pixel size of 32*32;

[0042] Figure 4 for Figure 3 The image shown is in Figure 1 The scattering device and the speckle pattern obtained under extremely low light conditions have a pixel size of 32*32 and an average of 0.3 photons per pixel. The detector is located on the motion guide rail and sets the current position as the reference position.

[0043] Figure 5 for Figure 3 The image shown is in Figure 1 The scattering device and the speckle pattern obtained under extremely low light conditions have a pixel size of 32*32. The detector is located on the motion guide rail and the current position is 6mm away from the reference position.

[0044] Figure 6 for Figure 3 The image shown is in Figure 1 The speckle pattern, obtained under extremely low light conditions using a scattering device, has a pixel size of 32*32. The detector is located on a motion guide rail, with its current position 6mm away from the reference position, and the offset direction is... Figure 4 same;

[0045] Figure 7 for Figure 4 The image shown is a speckle reconstruction obtained by the detector at the reference position, with a pixel size of 32*32.

[0046] Figure 8 for Figure 5The image shown is a speckle reconstruction obtained by the detector at the reference position, with a pixel size of 32*32.

[0047] Figure 9 for Figure 6 The image shown is a speckle reconstruction obtained by the detector at the reference position, with a pixel size of 32*32.

[0048] Figure 10 This is the original target image in the test set in one embodiment of this application, with a pixel size of 32*32;

[0049] Figure 11 for Figure 10 The speckle pattern collected under the movement of the detector has a pixel size of 32*32.

[0050] Figure 12 for Figure 11 Images reconstructed by a neural network trained on a training set with the speckle center directly opposite the detector, each image being 32*32 pixels;

[0051] Figure 13 for Figure 11 The clear images are reconstructed by a neural network that has been trained, with each pixel being 32*32.

[0052] The reference numerals in the embodiments of the present invention are explained as follows:

[0053] The system consists of a light source 1, an attenuator 2, a spatial light modulator 3, a lens 4, a grating 5, a lens 6, a frosted glass 7, a frosted glass 8, a detector 9, a motion guide rail 10, and a processor module 11. Light is emitted from the light source 1 and passes sequentially through the attenuator 2, the spatial light modulator 3, the lens 4, the grating 5, the lens 6, the frosted glass 7, and the frosted glass 8 before reaching the detector 9. Detailed Implementation

[0054] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0055] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0056] The extremely weak light environment defined by this invention is specifically a photon count on the order of several orders of magnitude.

[0057] The scattering imaging device used in this embodiment is such as Figure 1 As shown, the optical path includes, in sequence, a light source 1, an attenuator 2, a spatial light modulator 3, a lens 4, a grating 5, a lens 6, a frosted glass 1, a frosted glass 2, a detector 9, a motion guide rail 10, and a processor module 11.

[0058] Light is emitted from light source 1 and passes sequentially through attenuator 2, spatial light modulator 3, lens 4, grating 5, lens 6, frosted glass 7, and frosted glass 8 before reaching detector 9. Lens 4 is used to converge the reflected light from spatial light modulator 3, grating 5 is used to filter out clutter in the light converged by lens 4, lens 6 is used to diverge the filtered modulated light, forming a double-lens system with lens 4, frosted glass 7 is used to scatter the light diverged by lens 6, and frosted glass 8 is used to scatter the light scattered by frosted glass 7. Detector 9 is used to detect and count the intensity of the scattered light. Detector 9 is fixed on motion guide rail 10 and has a data connection with processor module 11. The processor module can perform moving target imaging reconstruction on the data collected by detector 9.

[0059] Preferably, the detector 9 can acquire light field intensity values ​​at the same frequency as the spatial light modulation device 3. Preferably, the light intensity of the light source 1 is adjustable.

[0060] Preferably, detector 9 is a single-photon camera with sensitivity exceeding the shot noise limit for counting photons. Preferably, motion guide rail 10 is an electrically controlled rail capable of setting velocity and acceleration for one-dimensional reciprocating motion and recording position information. Preferably, motion guide rail 10 can store motion position information and synchronize it with the information acquisition time of detector 9.

[0061] The specific steps for a scattering imaging device to acquire a scattering map are as follows:

[0062] (1) Spatial light modulator 3 uses a lens to flip the target, and the light emitted by light source 1 is attenuated and hits the target of spatial light modulator 3.

[0063] (2) Lens 4 converges the reflected light from the target in spatial light modulator 3;

[0064] (3) The grating 5 filters out reflected stray light from the lens 4 and focuses the spatial light;

[0065] (4) Lens 6 appropriately diffuses the light filtered by grating 5, and together with lens 4, forms a double lens system;

[0066] (5) The light diffused by the frosted glass-7 scattering lens-6;

[0067] (6) Frosted glass 28 scatters light; Frosted glass 17 scatters light.

[0068] (7) The processor module 11 collects the number of photons of the reflected light detected by the detector 9;

[0069] (5) A grayscale image is obtained by normalizing the number of photons obtained by processor 11.

[0070] In this embodiment, the exposure time is 2μs, the average number of photons received by the single-photon detector is controlled to be 0.2, and 10400 scattering images corresponding to 2600 original clear targets are acquired at 4 different positions of the detector on the motion guide rail.

[0071] like Figure 2 As shown in the figure, this embodiment provides an imaging method for detector motion in extremely low light environments, specifically including the following steps:

[0072] Step 1: Construct an optical path with a scattering medium, and adjust the light intensity and exposure time so that the single-photon detector can obtain the scattering map of the target under weak light.

[0073] Step 2: Move the guide rail so that the detector obtains target scattering maps at N fixed positions within the detection range, obtains sampling maps of fixed positions for training, and obtains displacement information of the guide rail where the detector is located.

[0074] Step 3: Train the neural network using the sampled speckle at each fixed position. After training, fix the parameters to form a neural network for low-photon imaging reconstruction at fixed positions.

[0075] In this embodiment, the weak light scattering image is preferably used as the input of the neural network, and the output is the clear image reconstructed by the model. The parameters are automatically adjusted by minimizing the mean square error between the clear image reconstructed by the model and the original clear image, thereby optimizing the neural network model.

[0076] Step 4: Move the detector within the detection range to obtain a sampled image of the motion detection for testing and the displacement information of the guide rail where the detector is located.

[0077] Step 5: Assign the low-photon speckle to be recovered from Step 4 to the nearest area of ​​the fixed sampling point described in Step 2, based on the guide rail position.

[0078] Step 6: Using the neural network described above for low-photon fixed-position reconstruction, reconstruct the speckle pattern of the pre-classified low-photon moving target to be recovered obtained in Step 5.

[0079] Step 6.1: Use the sampling image obtained in step 4 and the location classification information obtained in step 5 to obtain the reconstruction target through the neural network corresponding to step 3.

[0080] Step 6.2: Use the sampled image obtained in step 4 through the single-location neural network obtained in step 3 to obtain the reconstruction target.

[0081] Step 6.3: Compare and analyze the results of the neural network trained with the detector at a single location and the neural network that classifies and then recovers the detector at multiple locations.

[0082] In this embodiment, preferably, step 6.2 selects a single position as the detector position where the speckle center is directly opposite the detector center.

[0083] Pre-partitioning the detector location reduces the number of training samples required for each training session, thus limiting subsequent training sessions to a limited number of batches and reducing the total number of training sessions.

[0084] Using datasets from different locations for classification training can effectively improve the learning of environmental information, and increase the training speed and overall training effect of networks in the same area. For example... Figure 3-13 The image shown is a specific imaging diagram of a particular embodiment performed according to the above-described apparatus and method.

[0085] Using low-photon speckle sampling maps as training data, a neural network is trained and stored for N specified locations corresponding to the original target images from the low-photon speckle sampling maps. The neural network is a U-shaped network with additional skip connections. Skip connections are used between the corresponding layers of downsampling and upsampling in the neural network, and skip connections are also used between each layer in the convolution process to increase the correlation between layers.

[0086] This reduces the number of training samples required for a single neural network. It avoids the need to resample the speckle pattern at the detector's location for deep learning when the detector moves, resulting in greater robustness. Overall, it utilizes more training samples, improving the quality of deep learning methods for scattering imaging recovery and tracking in extremely low-light environments.

[0087] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for motion scattering imaging of detectors in extremely low light environments based on deep learning, characterized in that, The method includes: Step 1: Construct an optical path with a scattering medium, adjust the light intensity and exposure time, and obtain detection imaging under extremely low light conditions; Step 2: Obtain target scattering maps at N fixed locations within the detection range, and acquire scattering imaging maps and corresponding fixed location information; Step 3: Train the neural network using the scattering image at each fixed position and the fixed position information respectively. After training, a neural network with the fixed position labels is formed. Step 4: Apply the above neural network training to obtain scattering imaging images at other locations; Step 4 specifically involves: Step 4-1: Move the detector within the detection range to obtain a sampled image of the motion detection for testing and the displacement information of the guide rail where the detector is located; Step 4-2: Assign the low-photon speckle to be recovered and the guide rail position pair from Step 4-1 to the area where the fixed position is located in Step 2; Step 4-3: Using the neural network, reconstruct the speckle pattern of the pre-classified low-photon moving target to be recovered obtained in Step 4-2; Step 4-3 specifically involves: Step 4-3-1: Use the sampling image obtained in step 4-1 and the location classification information obtained in step 4-2 through the neural network corresponding to step 3 to obtain the reconstruction target; Step 4-3-2: Use the sampled image obtained in Step 4-1 through the single-location neural network obtained in Step 3 to obtain the reconstruction target; Step 4-3-3: Compare and analyze the results of the neural network trained with the detector at a single location and the neural network that classifies and then recovers the detector at multiple locations, and output the results.

2. The method for motion scattering imaging of detectors in extremely low light environments based on deep learning as described in claim 1, characterized in that, The neural network in step 3 is optimized by using the weak light scattering image as the input of the neural network and outputting a clear image reconstructed by the model. The parameters are automatically adjusted by minimizing the mean square error between the clear image reconstructed by the model and the original clear image, thereby obtaining the neural network model.

3. The method for motion scattering imaging of detectors in extremely low light environments based on deep learning as described in claim 1, characterized in that, The exposure time in step 1 is 2 μs.

4. A detector motion scattering imaging device based on deep learning in extremely low light environments, characterized in that, For performing the deep learning-based detector motion scattering imaging method in extremely low light environments as described in any one of claims 1-3, the apparatus comprises: Light source (1, 2, 3) is used to generate extremely weak light sources; Focusing components (4, 5, 6) are used to perform optical path focusing on the extremely weak light source; At least one scattering optical path (7, 8), the at least one scattering optical path (7, 8) includes a first frosted glass (7) and a second frosted glass (8) arranged sequentially along the optical path, for performing scattering processing on the extremely weak light after focusing; The detector (9) is a single-photon camera, which is mounted on an adjustment device (10). The adjustment device (10) is configured to move within a set measurement range, thereby enabling the detector (9) to collect extremely weak light after being processed by the scattering light path in multiple position states. The adjustment device (10) generates position information in multiple position states. The detector (9) and the light source (1, 2, 3) collect light field intensity values ​​at the same frequency. The information collection time of the adjustment device (10) and the detector (9) is synchronized. The processor (11) is used to receive the light signal from the detector (9) and perform analysis to generate neural network data samples for deep learning at the multiple locations. The data samples include image information and the location information corresponding to the image information.

Images