Method and device for imaging moving targets in extremely weak light environment based on deep learning

Abstract

The application discloses a kind of based on deep learning's extremely weak light environment under moving target scattering imaging method and device, it is related to extremely weak light environment under moving target imaging field, for the speckle image obtained by detector, establish neural network to obtain position information and original image recovery information.The accurate position information of target is obtained using prior information and recovered speckle to carry out trajectory recovery.N fixed positions in the detection range are placed target, and the sampling graph of fixed position for training is obtained.N fixed positions of sample speckle are used to train neural network, and after training, fixed parameter becomes the neural network for low-light imaging position classification.Each fixed position of sample speckle is used to train neural network respectively, and after training, fixed parameter becomes the neural network for low-light imaging reconstruction in fixed position.The application uses multiple position neural networks to recover speckle together, obtains more environmental information, and is conducive to generating the accuracy of detection imaging under the complex motion state of target.

Description

A Deep Learning-Based Method and Device for Moving Target Scattering Imaging in Extremely Low Light Environments Technical Field

[0001] This invention relates to the field of imaging, and more particularly to a method and apparatus for imaging the scattering of moving targets in extremely low light environments based on deep learning. Background Technology

[0002] Scattering imaging recovery under low-photon conditions has important applications in real life, including night vision, biomedical imaging, and satellite observation.

[0003] Traditional imaging devices typically require 10¹² photons to acquire a high-quality image. On average, a high-quality array detector needs to capture 10⁵ photons per pixel. However, in practical applications, it's difficult to achieve this photon count. Especially in extremely low light conditions and with limited exposure time, the effective photon count may only reach a few.

[0004] When the number of photons is sufficient, ordinary detectors detect analog signals, which contain a large number of photons. The superposition of these photons constitutes the detected light intensity. The method of acquiring an image by recording the light intensity at each location on the target is called the analog method. However, as the light intensity at the target decays, it gradually transforms into a pulse signal. Especially when the light intensity weakens to the single-photon condition, the signal becomes a discrete pulse signal with a very small number of pulses. A single photon is generally considered the detection limit, the smallest unit of energy that cannot be further divided. Therefore, under low-photon counting conditions, the signal exhibits particle characteristics. In this case, recording light as a single photon, determining its spatial position while detecting a single photon, and performing two-dimensional photon counting detection is the basis of photon counting imaging.

[0005] A single-photon camera (SPC) is a two-dimensional array detector where each pixel has a single-photon avalanche diode (SPAD), essentially an independent point detector. In extremely low light conditions, photons break down the SPAD and are counted. Notably, the image produced by this camera is not a grayscale image; each pixel represents the number of photons falling on that pixel per unit time. The main source of noise in extremely low light is Poisson noise due to the particle nature of light. Normalizing the photon count yields the desired grayscale image. Furthermore, the resolution of single-photon detectors can be further improved by using the time-of-flight of photon sequences in SPC technology. Because its sensitivity can overcome the shot noise limit, and it possesses advantages such as good signal-to-noise ratio, low power consumption, high quantum efficiency, and small size, it is widely used in the field of extremely low light imaging.

[0006] The combination of traditional physics research problems and deep learning has yielded some advantages, especially in image processing. Traditional scattering imaging only focuses on the single problem of scattering, neglecting the complex situation where three constraints—few photon counts, scattering, and moving targets—simultaneously exist. In a few-photon environment, the detected speckle contains a large amount of shot noise, reducing the signal-to-noise ratio. Scattering imaging methods based on the wave nature of light struggle to recover speckle under few-photon detection. Moving targets not only alter the optical path but also introduce motion blur errors into traditional long-exposure imaging methods, further degrading image quality. Imaging methods based on learning the scattering medium and modeling the transmission process also fail to address the challenges in these complex situations. Methods under strong light conditions require multiple scans and detections of the target, and the motion and scattering processes in the optical path further complicate imaging under low-photon conditions with relative motion between the detector and the target. In recent years, deep learning methods have been widely applied in the field of extremely low-light imaging, achieving promising results. However, existing deep learning methods also have many drawbacks, such as the need for pre-training with large amounts of training data for the neural network to learn the characteristics of scattering patterns and thus acquire imaging capabilities. However, when the target is moving, the scattering medium area used is large, and the correlation between different positions is weak, making it impossible to learn effective information. Therefore, when the target's position changes significantly, existing deep learning methods require processing large amounts of data during training, and the learning ability of the neural network cannot be guaranteed. Therefore, those skilled in the art are dedicated to developing a deep learning-based method and device for scattering imaging of moving targets in extremely low light environments to solve the problems existing in the prior art. Summary of the Invention

[0007] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is that traditional scattering imaging only focuses on the single problem of scattering, without considering the complex situation where three limitations of photon counting, scattering and detector motion coexist. Therefore, an apparatus and method for detection and imaging in extremely low light environment and under target motion are proposed to solve the above problems.

[0008] To achieve the above objectives, this invention provides a deep learning-based method for imaging the scattering of moving targets in extremely low light environments, characterized in that the method includes:

[0009] Step 1: Construct an optical path with a scattering medium, adjust the light intensity and exposure time, and obtain detection imaging under extremely low light conditions;

[0010] Step 2: Place targets at N fixed locations within the detection range and obtain target scattering imaging sampling images at each fixed location;

[0011] Step 3: Train the neural network using the scattering imaging sampling map at each fixed position. After training, a neural network is formed. The neural network includes a first neural network for imaging position classification and a second neural network for reconstructing the original sampling map at the corresponding imaging position in extremely low light environment.

[0012] Step 4: Obtain a scattering imaging sample map of the moving target, and use the neural network obtained in step 3 to train and obtain the reconstructed image.

[0013] Furthermore,

[0014] The second neural network in step 3 is optimized by using the weak light scattering image as the input of the neural network and outputting a clear image reconstructed by the model. The training parameters are automatically adjusted by minimizing the mean square error between the clear image reconstructed by the model and the original clear image, thereby optimizing the neural network model.

[0015] Furthermore,

[0016] Step 4 specifically involves:

[0017] Step 4-1: The moving target scattering imaging sampling image is input through the first neural network obtained in step 3 to obtain the position classification information of the moving target;

[0018] Step 4-2: Correct the position classification information obtained in step 4-1. The correction is based on the prior information that the moving target cannot jump in non-adjacent intervals.

[0019] Step 4-3: Input the moving target scattering imaging sampling map and the position classification information obtained in step 4-2 into the second neural network obtained in step 3 to obtain the reconstructed moving target sequence.

[0020] Furthermore, the second neural network is a U-shaped network with additional skip connections, wherein the corresponding levels of downsampling and upsampling in the neural network are skip connections, and the convolutional layers of the neural network are skip connections.

[0021] Furthermore, the first neural network is a fully connected network containing an additional optimizer and randomly discarded connections.

[0022] This invention provides a deep learning-based scattering imaging device for moving targets in extremely low light environments, characterized in that the device comprises:

[0023] Light source, used to generate extremely weak light sources;

[0024] A spatial light modulator uses mirror flipping to load multiple targets; the light source emits light that is incident on the targets of the spatial light modulator.

[0025] A focusing component for focusing the target reflected light;

[0026] At least one scattering optical path is used to perform scattering processing on the focused, extremely weak light;

[0027] The detector collects extremely weak light after it has been processed by the scattered light path.

[0028] The processor module is used to receive the optical signal from the detector and perform analysis, and execute the deep learning-based motion target scattering imaging method in extremely low light environment as described above.

[0029] Furthermore, the detector and the light source acquire light field intensity values ​​at the same frequency.

[0030] Furthermore, the light intensity of the light source is adjustable.

[0031] Furthermore, the at least one-stage scattering optical path includes a first frosted glass and a second frosted glass arranged sequentially along the optical path.

[0032] Furthermore, the detector is a single-photon camera.

[0033] Technical effect

[0034] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention.

[0035] For the speckle image acquired by the detector, a neural network is built to obtain positional information and reconstruct information from the original image. Prior information and the reconstructed speckle are used to obtain the precise positional information of the target for trajectory recovery.

[0036] Targets are placed at N fixed locations within the detection range to obtain sampled images of these fixed locations for training. A neural network is trained using the speckle patterns from the N fixed locations. After training, the parameters are fixed to create a neural network for low-photon imaging location classification. A neural network is then trained separately using the speckle patterns from each fixed location. After training, the parameters are fixed to create a neural network for low-photon imaging location reconstruction at these fixed locations.

[0037] Considering the prior information that the moving target sequence cannot jump between non-adjacent intervals, the information obtained through the location classification network is corrected. Then, the reconstructed target sequence is obtained through the speckle recovery network at the corresponding location.

[0038] The cross-correlation between the reconstructed target sequence and the original target at different locations is calculated, and the maximum value of the cross-correlation is taken as the result of trajectory recovery.

[0039] Targets are placed at N fixed locations within the detection range to obtain sampled images of these fixed locations for training. A neural network is trained using the speckle patterns from these N fixed locations. After training, the parameters are fixed to create a fully connected network for low-photon imaging location classification, which includes an additional optimizer and randomly discarded connections. This classification network limits subsequent training iterations to a finite number of batches, reducing the number of training iterations.

[0040] The neural network is trained separately using sampled speckle data at each fixed location. After training, the parameters are fixed to form a neural network for low-photon imaging reconstruction at fixed locations. This neural network is a U-shaped network with additional skip connections. Skip connections are used between the corresponding layers of downsampling and upsampling in the neural network, and skip connections are also used between each layer in the convolution process to increase the correlation between layers.

[0041] A unique two-step neural network was developed to address this problem, performing location classification followed by speckle recovery for speckle images acquired by the detector. Both networks contain additional optimizers and fully connected layers with randomly discarded connections to increase the amount of information that can be learned. Attached Figure Description

[0042] Figure 1 is a schematic diagram of the composition structure of an imaging device according to an embodiment of the present invention;

[0043] Figure 2 is a schematic flowchart of an imaging method according to an embodiment of the present invention;

[0044] Figure 3 is one of the original target images in the training set in an embodiment of the present invention, with a pixel size of 32*32;

[0045] Figure 4 is a speckle pattern of the image shown in Figure 3 obtained under the scattering device in Figure 1 and under extremely low light conditions. In one embodiment of the present invention, the speckle sampling pattern of the same target original image in the training set at six different locations has a pixel size of 32*32 and the average number of photons received by the detector after scattering is 0.2.

[0046] Figure 5 is a speckle map of the image shown in Figure 4 obtained under the scattering device in Figure 1 and under extremely low light conditions. In one embodiment of the present invention, the speckle reconstruction map of the unified target original image of the training set at six different locations has a pixel size of 32*32.

[0047] Figure 6 is a diagram showing the relationship between a test trajectory of a moving target and the positions of sampling points in the trajectory in one embodiment of the present invention;

[0048] Figure 7 is a diagram showing the positional relationship between a trajectory of a test moving target and six positions in the training set in one embodiment of the present invention;

[0049] Figure 8 is a schematic diagram of the recovery of the test motion target under the motion sequence shown in Figure 6 in one embodiment of the present invention.

[0050] The reference numerals in the embodiments of the present invention are explained as follows:

[0051] 1-Light source, 2-Spatial light modulator, 3-Lens, 4-Grating, 5-Lens, 6-Frosted glass I, 7-Frosted glass II

[0052] 8-Detector, 9-Processor module. Detailed Implementation

[0053] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0054] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0055] The extremely weak light environment defined by this invention is specifically a photon count on the order of several orders of magnitude.

[0056] The scattering imaging device used in this embodiment is shown in Figure 1. From the optical path, it includes a light source 1, a spatial light modulator 2, a lens 3, a grating 4, a lens 5, a first frosted glass 6, a second frosted glass 7, a detector 8, and a processor module 9.

[0057] Light is emitted from light source 1 and passes sequentially through spatial light modulator 2, lens 3, grating 4, lens 5, frosted glass 6, and frosted glass 7 before reaching detector 8. Lens 3 is used to converge the reflected light from spatial light modulator 3, grating 4 is used to filter out clutter in the light converged by lens 3, and lens 5 is used to diverge the filtered modulated light, forming a double-lens system with lens 3. Frosted glass 6 is used to scatter the light diverged by lens 5, and frosted glass 7 is used to scatter the light scattered by frosted glass 6. Detector 8 is used to detect and count the intensity of the scattered light. Detector 8 has a data connection with processor module 9, which can perform moving target imaging and reconstruction on the data collected by detector 8.

[0058] Preferably, the detector 9 can acquire light field intensity values ​​at the same frequency as the spatial light modulation device 3. Preferably, the light intensity of the light source 1 is adjustable.

[0059] Preferably, detector 9 is a single-photon camera used for counting photons, whose sensitivity can break through the shot noise limit.

[0060] The specific steps for a scattering imaging device to acquire a scattering map are as follows:

[0061] (1) Spatial light modulator 2 uses a lens to flip the loaded target, and the light emitted by light source 1 hits the target of spatial light modulator 2;

[0062] (2) Lens 3 converges the reflected light from the target in spatial light modulator 2;

[0063] (3) The grating 4 filters out reflected stray light from the focused spatial light by the lens 3;

[0064] (4) Lens 5 appropriately diffuses the light filtered by grating 4, and together with lens 3, forms a double lens system;

[0065] (5) The light emitted by the frosted glass-6 scattering lens 5;

[0066] (6) Frosted glass 27 scatters light; Frosted glass 16 scatters light.

[0067] (7) The processor module 9 collects the number of photons of the reflected light detected by the detector 8;

[0068] (5) A grayscale image is obtained by normalizing the number of photons obtained by processor 11.

[0069] In this embodiment, the exposure time is 2μs, the average number of photons received by the single-photon detector is controlled to be 0.2, and 13,000 scattering images corresponding to 2,600 original clear targets are acquired for 6 different positions of the moving target.

[0070] As shown in Figure 2, this embodiment provides one implementation of a deep learning-based method for imaging moving targets in extremely low light environments, specifically including the following steps:

[0071] Step 1: Construct an optical path with a scattering medium, adjust the light intensity and exposure time, and obtain detection imaging under extremely low light conditions;

[0072] Step 2: Place targets at N fixed locations within the detection range and obtain target scattering imaging sampling images at each fixed location;

[0073] Step 3: Train the neural network using the scattering imaging sampling map of each fixed position. After training, the neural network is formed. The neural network includes a first neural network for imaging position classification and a second neural network for reconstructing the original sampling map of the corresponding imaging position in extremely low light environment.

[0074] Step 4: Obtain the scattering imaging sample map of the moving target, and use the neural network obtained in Step 3 to train and obtain the reconstructed image.

[0075] The second neural network in step 3 is optimized by taking the weak light scattering image as the input of the neural network and outputting the clear image reconstructed by the model. The training parameters are automatically adjusted by minimizing the mean square error between the clear image reconstructed by the model and the original clear image, thereby optimizing the neural network model.

[0076] Step 4 specifically involves:

[0077] Step 4-1: Input the moving target scattering imaging sampling image into the first neural network obtained in Step 3 to obtain the position classification information of the moving target;

[0078] Step 4-2: Correct the position classification information obtained in Step 4-1, and correct it to the prior information that the reference moving target cannot jump in non-adjacent intervals;

[0079] Step 4-3: Input the moving target scattering imaging sampling map and the position classification information obtained in step 4-2 into the second neural network obtained in step 3 to obtain the reconstructed moving target sequence.

[0080] Using datasets from different locations for classification training can effectively improve the learning of environmental information, increase the training speed of the network in the same area, and improve the overall training effect. Figures 3-8 show imaging images under a specific implementation of the above-described apparatus and method.

[0081] Traditional scattering imaging only focuses on the single problem of scattering, without considering the complex situation where three constraints exist simultaneously: low photon count, scattering, and moving targets. In such cases, trajectory recovery is extremely difficult.

[0082] For the speckle image acquired by the detector, a neural network is built to obtain positional information and reconstruct information from the original image. Prior information and the reconstructed speckle are used to obtain the precise positional information of the target for trajectory recovery.

[0083] A unique two-step neural network was developed to address this problem, performing location classification followed by speckle recovery for speckle images acquired by the detector. Both networks contain additional optimizers and fully connected layers with randomly discarded connections to increase the amount of information that can be learned.

[0084] As another embodiment of the present invention:

[0085] Step 1: Construct an optical path with a scattering medium, and adjust the light intensity and exposure time so that the single-photon detector can obtain the scattering map of the target under weak light.

[0086] Step 2: Place targets at N fixed locations within the detection range to obtain a sampling map of the fixed locations for training.

[0087] In this embodiment, the exposure time is 2μs, the average number of photons received by the single-photon detector is controlled to be 0.2, and 13,000 scattering images corresponding to 2,600 original clear targets are acquired at 6 different locations.

[0088] Step 3: Train the neural network using N fixed-position sampling speckle data. After training, fix the parameters to create a neural network for low-photon imaging position classification.

[0089] In this embodiment, preferably, the weak light scattering image is used as the input to the neural network, and the output is any one of the six location classifications.

[0090] Step 4: Train the neural network using the sampled speckle at each fixed position. After training, fix the parameters to form a neural network for low-photon imaging reconstruction at fixed positions.

[0091] In this embodiment, the weak light scattering image is preferably used as the input of the neural network, and the output is the clear image reconstructed by the model. The parameters are automatically adjusted by minimizing the mean square error between the clear image reconstructed by the model and the original clear image, thereby optimizing the neural network model.

[0092] Step 5: Move the target within the detection range along a predetermined trajectory to obtain a sampling map of the movement position for testing.

[0093] Step 6: Using the neural network described above for low-photon imaging position classification and fixed position reconstruction, the speckle pattern of the low-photon moving target to be recovered obtained in Step 5 is reconstructed.

[0094] Step 6.1: Pass the sampled image obtained in Step 5 through the neural network in Step 3 to obtain location classification information.

[0095] Step 6.2: Considering the prior information that the moving target sequence cannot jump between non-adjacent intervals, the position classification information obtained in Step 6.1 is corrected.

[0096] Step 6.3: Use the sampling image obtained in Step 5 and the location classification information obtained in Step 6.2 to obtain the reconstructed target sequence through the neural network corresponding to Step 4.

[0097] In summary, this invention is applicable to low photon count scenarios, and even when light exhibits particle characteristics, it can still acquire certain effective information through a single-photon detector. It is also applicable to moving targets, allowing for the acquisition of effective information even when the target is moving within a certain range. The algorithm reduces the number of samples required for training a single neural network. It avoids the need to resample the speckle pattern at the target's location for deep learning when the target moves, thus exhibiting greater robustness. Overall, the algorithm utilizes more training samples, improving the quality of scattering imaging recovery and tracking in extremely low-light environments using deep learning methods.

[0098] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A deep learning-based method for imaging the scattering of moving targets in extremely low light conditions, characterized in that, The method includes: Step 1, constructing an optical path with a scattering medium, adjusting the light intensity and exposure time to obtain detection imaging under extremely low light conditions; Step 2, placing targets at N fixed positions within the detection range, and obtaining target scattering imaging sampling images at each fixed position; Step 3, training a neural network using the scattering imaging sampling images at each fixed position, forming a neural network after training. The neural network includes a first neural network and a second neural network. The first neural network is a fully connected network with an additional optimizer and randomly discarded connections, used for imaging position classification; the second neural network is a U-shaped network with additional skip connections, where the corresponding levels of downsampling and upsampling in the U-shaped network are skipped, and the convolutional layers of the U-shaped network are skipped, used for reconstructing the original sampling images at the corresponding imaging positions under extremely low light conditions. The second neural network is optimized to reduce the intensity of the scattering imaging sampling images in low light conditions. The scattering image is used as input to the neural network, and the output is a clear image reconstructed by the model. The training parameters are automatically adjusted by minimizing the mean square error between the clear image reconstructed by the model and the original clear image, thereby optimizing the neural network model. Step 4: Obtain a scattering imaging sampling map of the moving target, and use the neural network obtained in step 3 to train and obtain the reconstructed image. Specifically, this includes: Step 4-1: Input the scattering imaging sampling map of the moving target into the first neural network obtained in step 3 to obtain the position classification information of the moving target; Step 4-2: Correct the position classification information obtained in step 4-1, wherein the correction is based on the prior information that the moving target cannot jump in non-adjacent intervals; Step 4-3: Input the scattering imaging sampling map of the moving target and the position classification information obtained in step 4-2 into the second neural network obtained in step 3 to obtain the reconstructed moving target sequence.

2. A deep learning-based imaging device for scattering light from moving targets in extremely low light conditions, characterized in that, The device includes: a light source (1) for generating extremely weak light; a spatial light modulator (2) for loading multiple targets using lens flipping; light emitted by the light source (1) is incident on the target of the spatial light modulator (2); focusing components (3, 4, 5) for focusing the reflected light from the target; at least one scattering optical path (6, 7) for scattering the focused extremely weak light, the at least one scattering optical path (6, 7) including a first frosted glass (6) and a second frosted glass (7) arranged sequentially along the optical path; a detector (8) for collecting the extremely weak light after processing by the scattering optical path, the detector (8) being a single-photon camera and collecting light field intensity values ​​at the same frequency as the light source (1); and a processor module (9) for receiving the light signal from the detector (8) and performing analysis, executing the deep learning-based moving target scattering imaging method in extremely weak light environment as described in claim 1.

3. The deep learning-based moving target scattering imaging device in extremely low light environments as described in claim 2, characterized in that, The light intensity of the light source (1) is adjustable.

Images