Motion-distinguishable lensless dynamic imaging system and method

Abstract

The invention provides a lens-free dynamic imaging system and a lens-free dynamic imaging method with distinguishable motion. The system comprises a laser light source, a binary optical mask, an image sensor and image processing equipment, coherent light emitted by the laser light source is coded and modulated by the binary optical mask and then propagated to the image sensor, the image sensor collects multiple frames of diffraction images of a moving sample and transmits the collected images to the image processing device, and the image processing device reconstructs the images through a neural network method. According to the image reconstruction method, physical priori knowledge of a lensless diffraction propagation model is fused into implicit neural representation, and a space-time priori loss and bidirectional memory training method is designed for network training, so that artifacts and blur generated by moving target imaging can be effectively reduced, and a high-resolution and high-quality imaging result can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of microscopic imaging technology, and in particular to a motion-resolvable lensless dynamic imaging system and method. Background Art

[0002] Lensless microscopy is a microscopic imaging technology that does not require any optical lenses to achieve complex amplitude imaging of samples. It uses a digital image sensor to directly collect the diffraction intensity image of the object, and calculates the inverse solution through the corresponding physical optical model to reconstruct the complex amplitude image information of the object. This technology uses advanced computational imaging methods to replace the traditional optical lens system, achieving microscopic imaging with a large field of view (FOV) and high resolution. It can simultaneously record the amplitude and phase information of the light field, and its optical system has a simple structure and simple operation, which greatly reduces the hardware cost. With the further development of lensless imaging technology, it is expected to support the development of high-resolution, economical, small, and easy-to-operate microscopic detection equipment for clinical bedside testing, improve the early diagnosis rate of diseases, and promote the improvement of medical and health levels.

[0003] At present, existing lensless microscopy technology needs to collect multiple images with high signal-to-noise ratios to reconstruct high-quality images. It has poor robustness to noise and can only image static targets. However, in real scenarios, it is often necessary to observe dynamic targets. Once targets such as cells and microorganisms undergo slight movements, phase blur and motion artifacts will often appear in the reconstruction results, resulting in reduced imaging quality, which limits the application of lensless imaging technology.

[0004] Traditional iterative reconstruction algorithms, such as the GS algorithm, often require the acquisition of multiple diffraction images with specific diversity (including axial height diversity, illumination wavelength diversity, and illumination angle diversity) for lensless imaging in order to achieve high-quality image reconstruction, so as to fuse and reconstruct high-quality complex amplitude information of the sample. However, for lensless dynamic reconstruction, since the sample target is dynamic, the problem of multiple reconstructions is transformed into a one-to-one single reconstruction, resulting in reduced reconstruction resolution and motion blur. Implicit Neural Representation (INR) is a novel deep learning method for parameterizing signals. The key idea is to represent the signal object as a function approximated by a neural network using a multilayer perceptron (MLP), which maps coordinates to values ​​corresponding to the signal. Compared with the commonly used display representation method, INR can better learn high-frequency signals in images through compression coding. Summary of the invention

[0005] In view of the defects existing in the above prior art, the object of the present invention is to provide a motion-resolvable lensless dynamic imaging system and method, which can achieve high-resolution and high-quality lensless dynamic imaging.

[0006] The technical solution adopted by the system of the present invention is as follows:

[0007] A motion-resolvable lensless dynamic imaging system comprises a laser light source, a binary optical mask, an image sensor and an image processing device. The coherent light emitted by the laser light source is modulated by the binary optical mask coding and then propagates to the image sensor. The image sensor collects multiple frames of diffraction images of motion samples and transmits the collected images to the image processing device. The image processing device reconstructs the images using a reconstruction algorithm.

[0008] The present invention also provides a method for image reconstruction using the above-mentioned motion-resolvable lensless dynamic imaging system, the method comprising the following steps: using the lensless dynamic imaging system to collect multiple frames of dynamic images of a motion sample; performing focus search and image registration in the xy direction on the collected dynamic images; and using a reconstruction algorithm to perform high-resolution dynamic multi-frame image reconstruction on the registered images.

[0009] Compared with the prior art, the invention has the following innovations and advantages:

[0010] (1) The present invention applies INR to lensless microscopic imaging, which can better learn the high-frequency components in the image and achieve higher resolution and higher quality single lensless image reconstruction. The present invention incorporates the physical prior knowledge of the lensless diffraction propagation model into the implicit neural representation INR, and trains it through the designed spatiotemporal prior loss and bidirectional memory network training method. While improving the reconstruction stability and accelerating the training, it can effectively reduce the motion blur and reconstruction artifacts caused by lensless imaging of dynamic samples, and achieve high-quality lensless dynamic imaging. In addition, the spatiotemporal prior loss designed by the present invention can better ensure the continuity of the motion sample, prevent sudden motion changes between frames, and achieve higher frame rate (FPS) reconstruction.

[0011] (2) The present invention provides a lensless imaging system for capturing dynamic sample images, which is used to continuously capture multiple frames of sample diffraction images and provide sample diffraction intensity information required for dynamic image reconstruction. Compared with existing multi-height lensless imaging systems, which capture multiple images per frame by changing the distance between the camera and the sample, the lensless system of the present invention does not require multi-height capture (only one image per frame needs to be captured for reconstruction), and uses an optical mask to encode and modulate the coherent light beam to achieve higher resolution when reconstructing the image. The binary optical mask in the system of the present invention has a smaller feature size and can achieve a stronger encoding and modulation effect on the coherent light beam. The system of the present invention is simple and easy to operate, does not require cumbersome diversity capture, can quickly obtain multiple frames of dynamic sample image information, and provides the required diffraction intensity information for the image reconstruction method of the present invention, thereby achieving motion-resolvable high-quality lensless dynamic imaging.

[0012] (3) The present invention uses multiple frames of diffraction images collected by the imaging system as input images for image reconstruction, which can effectively improve image resolution, reduce motion blur, reduce the impact of noise on imaging quality, and achieve high-quality lensless dynamic imaging with discernible motion. Through simulation tests, compared with traditional iterative reconstruction algorithms, the method of the present invention can reconstruct images with higher resolution and cleaner background, and can better reduce motion blur and artifacts caused by dynamic target imaging, and has higher noise robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a system block diagram of the lensless dynamic imaging system of the present invention;

[0014] Figure 2 Schematic diagram of the optical structure of the lensless dynamic imaging system of the present invention;

[0015] Figure 3 A flowchart of a dynamic sample image reconstruction algorithm according to an embodiment of the present invention;

[0016] Figure 4 The imaging results of the USAF-1951 resolution plate obtained by simulating the imaging method of the embodiment of the present invention and the traditional iterative reconstruction algorithm. (a1) is the amplitude image reconstructed by the traditional iterative reconstruction algorithm, (a2) is the local magnified image of (a1), (b1) is the amplitude image reconstructed by the PBMINR algorithm of the present invention, and (b2) is the local magnified image of (b1).

[0017] Figure 5The imaging results of plant samples obtained by simulating the imaging method of the embodiment of the present invention and the traditional iterative reconstruction algorithm. (a1) and (a2) are the amplitude and phase images reconstructed by the traditional iterative reconstruction algorithm, and (b1) and (b2) are the amplitude and phase images reconstructed by the PBMINR algorithm of the present invention. DETAILED DESCRIPTION

[0018] Embodiments of the present invention are described in detail below with reference to the accompanying drawings, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0019] Reference Figure 1 As shown, the system of the present invention includes a laser light source, an optical mask, an image sensor and an image processing device. It is a lens-free dynamic imaging system. The coherent light emitted by the laser light source is modulated by a binary optical mask code, and then propagates a certain distance to reach the image sensor. The image sensor collects multiple frames of diffraction images of the moving sample and provides the sample diffraction intensity information required for dynamic image reconstruction. The image sensor transmits the collected image to the image processing device for image reconstruction.

[0020] Specifically, the laser light source provides illumination; the optical mask provides a coding modulation effect on the light beam; the image sensor records the diffraction image information of each frame of the dynamic sample; and the image processing device reconstructs the diffraction image by using a neural network. Since the distance from the optical mask to the sample and the distance from the sample to the image sensor cannot be accurately determined by the system in actual experiments, the actual distance from the optical mask to the sample and the distance from the sample to the image sensor is obtained by focusing search in the imaging method.

[0021] Figure 2The optical structure diagram of a motion-resolvable lensless dynamic imaging system of the present invention. Specifically, the laser light source emits visible light with a wavelength of 400-700nm. In this embodiment, the MBL-FN-473 light source of Changchun New Industry Optoelectronic Technology Co., Ltd. is used to provide coherent light with a wavelength of 473nm vertically incident along the optical axis. The binary optical mask is made by Wuxi Beihui Optoelectronic Materials Co., Ltd. on a silicon dioxide substrate using laser direct writing technology. A thin layer of chromium coating is plated on the substrate to form a random binary pattern with a characteristic size of 10-20um×10-20um and a size of 4-6mm×4-6mm. In this embodiment, the characteristic size is 14.8um×14.8um and the size is 4mm×4mm. The image sensor uses an Image Source DMM DMK 33UX226 USB 3.0 monochrome camera with a pixel size of 1.85um and a captured image size of 2000(W)*2000(H). The image processing device is a computer, and the reconstruction calculation of the neural network is realized by NVIDIA RTX2080Ti GPU. The neural network adopts the classic MLP network structure.

[0022] Based on the above-mentioned imaging system, this embodiment proposes a motion-resolvable lensless dynamic imaging method, which mainly uses a physical-constrained bidirectional memory implicit neural representation algorithm (Physical-inspired Bidirectional Memory Implicit Neural Representation, hereinafter referred to as "PBMINR") to reconstruct the original dynamic image sequence. The PBMINR algorithm of the present invention is based on the INR technology and the coded modulation lensless imaging physical model (optical propagation diffraction model), and adopts a bidirectional memory training method. It is an application improvement of INR technology in lensless imaging technology. Specifically, the dynamic imaging method of this embodiment includes the following steps: focusing search and image registration in the xy direction of multiple frames of dynamic images collected by the lensless dynamic imaging system; using the PBMINR algorithm to perform high-resolution dynamic multi-frame image reconstruction on the registered images.

[0023] In the actual imaging experiment, the system cannot collect a diffraction image that completely corresponds to the produced binary optical mask. If the produced binary optical mask and the collected image are directly substituted into the PBMINR algorithm for solution, the quality of the reconstructed image will be significantly reduced. Therefore, before the algorithm reconstruction, the imaging method of the present invention uses the existing algorithm to perform focus search and image registration in the xy direction on the collected image, and calibrate the binary optical mask to ensure the accuracy of the information of the binary optical mask coding modulation.

[0024] The PBMINR algorithm of the present invention takes as input the multi-frame diffraction images collected by the imaging system of the present invention, and outputs the reconstructed dynamic sample image sequence. The physical prior knowledge of the lensless diffraction propagation model is integrated into the implicit neural representation, and is trained by the designed spatiotemporal prior loss and bidirectional memory network training method. During the reconstruction iteration, the Adam optimizer is used to update the weight parameters of the MLP network based on the INR method. Specifically, refer to Figure 3 As shown, the PBMINR algorithm of this embodiment includes the following steps:

[0025] Step 1, initialization: generate the initial image coordinates r(x,y) and the MLP network weights θ;

[0026] Step 2, position encoding: Apply the following position encoding (PE) to the input coordinates, with an encoding depth i of 10. The value of i can be adjusted according to different usage scenarios.

[0027] PE(x,y)={cos(2πsσ i x),sin(2πsσ i x)}

[0028] ∪{cos(2πsσ i y),sin(2πsσ i y)},i∈{1,2,...,10}

[0029] where σ i ~N(0,1) follows random sampling from a standard Gaussian distribution, and s is a scale value selected according to the scenario.

[0030] Step 3: Implicit mapping: The MLP network with the current weight θ realizes the mapping from the encoded coordinate r to the reconstructed image amplitude A and phase φ: g θ (r):(x,y)→(A,φ), that is, reconstructed image O=A·e jφ , j is a complex unit;

[0031] Step 4, forward propagation: Using the angular spectrum propagation theory, forward propagate the image O(k) reconstructed by the current k-th frame to obtain the 2D intensity image I′(k) of the motion sample on the image sensor plane;

[0032] Step 5, reverse gradient update: According to the spatiotemporal prior loss function L designed below, the relative loss between the intensity image I′(k) obtained in step 4 and the real collected intensity image I(k) is calculated, and the weight θ of the MLP network is updated using the Adam optimizer; more specifically, the learning rate of the optimizer in this embodiment adopts a cyclic attenuation method, which is initially 0.001 and decays by 50% every 300 iterations.

[0033] L=||I′(k)-I(k)|| 2 +[1-SSIM(I′ k ,I k )]+(ρ s ||D xy O(k,k-1)|| 1 +ρ t ||D t O(k,k-1)||)

[0034] Where k = 1, 2, ..., K, K is the total number of frames in the dynamic image sequence, structural similarity (SSIM) is a commonly used evaluation index for image quality, and D xy =(D x ,D y ),D t denote the finite difference operators along the three dimensions of space (x, y) and time t, respectively, and O(k, k-1) denotes the application of the difference operator to reconstruct the complex amplitude image of the kth and k-1th frames; specifically,

[0035]

[0036] Where m is the length and width of the reconstructed image, and ρ S and ρ t is the regularization parameter, and its size is set to 10 in this embodiment. -12 , 10 -11 By performing a differential operation on the current training reconstructed frame O(k) and the previous training reconstructed frame O(k-1), the temporal and spatial continuity loss is introduced to reduce the blur caused by moving targets.

[0037] Step 6, bidirectional memory training: Adopt the method of updating the MLP network parameters bidirectionally frame by frame according to the time axis, save the training parameters of the current kth frame, inherit the training from the k+1 frame during forward training, and inherit the training from the k-1 frame during reverse training, and stop in advance dynamically according to the minimum loss of forward training. If it is not stopped in advance, the reverse training will be stopped directly when the set maximum number of iterations is reached. This training method can improve the stability of reconstruction and reduce training time, and finally reconstruct a high-quality dynamic image sequence.

[0038] Repeat steps 2 to 6 above. In this embodiment, 700 cycles are used for each frame in the forward training. After all frames are trained, reverse training is performed. During reverse training, a maximum of 1500 cycles are set for each frame (the number of training rounds can be adjusted according to the specific usage scenario). When half of the minimum loss of the forward training is reached, the training is stopped in advance (or directly stopped when the set maximum number of iterations is reached). In this way, the MLP network weight θ is continuously updated to obtain the reconstructed image sequence O(K) of the final dynamic sample.

[0039] Traditional iterative reconstruction algorithms, such as the GS algorithm, are sensitive to noise, and once the observed sample moves, the reconstructed image will be blurred and the resolution will be reduced. The present invention is based on the INR optimization design reconstruction algorithm, which incorporates the physical prior knowledge of the lensless diffraction propagation model, and designs a time-space prior loss and a bidirectional memory network training method for training, which can effectively solve the problems of motion blur and image quality degradation caused by reconstructing dynamic targets.

[0040] Reference Figure 4 As shown, the embodiment of the present invention simulates the acquisition of the moving USAF-1951 resolution plate diffraction pattern information, and uses the traditional iterative reconstruction algorithm and the PBMINR algorithm proposed in the present invention to reconstruct the image. The embodiment simulates the movement of the sample through image processing, and sets the sample movement parameters as follows: horizontal movement of 1 pixel per frame, clockwise rotation of 0.5 degrees per frame, and a total of 10 frames of sample diffraction image information are collected. The amplitude image sequence reconstructed by the traditional iterative reconstruction algorithm and the PBMINR algorithm proposed in the present invention is shown in FIG. Figure 4 (Limited space, the embodiment only shows some frame effects). As shown in (a1), (b1) and the corresponding local enlarged images (a2), (b2), k represents the reconstructed k-th frame image. The reconstructed image obtained by the traditional iterative reconstruction algorithm has more obvious motion blur and resolution loss. The PBMINR algorithm proposed in the present invention can effectively reduce the motion blur caused by dynamic sample imaging, improve the resolution, and achieve higher quality motion-resolvable lensless dynamic imaging.

[0041] Reference Figure 5 As shown, the embodiment of the present invention simulates and collects the diffraction pattern information of the real plant sample in motion, and uses the traditional iterative reconstruction algorithm and the PBMINR algorithm proposed in the present invention to reconstruct the image. The embodiment collects the plant sample images required for simulation through real microscopic experiments, which have different amplitude and phase information. The embodiment sets the simulation motion parameters as follows: horizontal movement of 1 pixel per frame, clockwise rotation of 0.5 degrees per frame, and a total of 10 frames of sample diffraction image information are collected. The amplitude and phase image sequences reconstructed by the traditional iterative reconstruction algorithm and the PBMINR algorithm proposed in the present invention are shown in FIG. Figure 5 (Due to limited space, the embodiment only shows some frame effects.) As shown in (a1), (b1) and the corresponding phase images (a2), (b2), the reconstructed amplitude image obtained by the traditional iterative reconstruction algorithm loses detail information, has low resolution and phase ambiguity, while the PBMINR algorithm proposed in the present invention can achieve higher resolution and better phase recovery effect.

Claims

1. A motion-resolvable lensless dynamic imaging system, characterized in that: The system includes a laser light source, a binary optical mask, an image sensor and an image processing device; the coherent light emitted by the laser light source is coded and modulated by the binary optical mask, and then propagates to the image sensor, which collects multiple frames of diffraction images of the moving samples and transmits the collected images to the image processing device, which reconstructs the images using a reconstruction algorithm.

2. The motion-resolvable lensless dynamic imaging system according to claim 1, characterized in that: The wavelength of the laser light source is 400-700nm, and the incident light is vertically incident along the optical axis.

3. The motion-resolvable lensless dynamic imaging system according to claim 1, characterized in that: The binary optical mask is a random binary pattern with a feature size of 10-20um×10-20um and a size of 4-6mm×4-6mm.

4. A method of using a motion-resolvable lensless dynamic imaging system as claimed in claim 1, characterized in that: The method comprises the following steps: Using the lensless dynamic imaging system to collect multiple frames of dynamic images of the motion sample; Perform focus search and image registration in the xy direction on the acquired dynamic images; The reconstruction algorithm is used to perform high-resolution dynamic multi-frame image reconstruction on the registered images.

5. The method according to claim 4, characterized in that The reconstruction algorithm specifically comprises the following steps: Step 1, initialization: generate the initial image coordinates r(x,y) and the MLP network weights θ; Step 2, position encoding: Apply position encoding to the input coordinates; Step 3: Implicit mapping: The MLP network with the current weight θ realizes the mapping from the encoded coordinate r to the reconstructed image amplitude A and phase φ: g θ (r):(x,y)→(A,φ), that is, reconstructed image O=A·e jφ , j is a complex unit; Step 4, forward propagation: Using the angular spectrum propagation theory, forward propagate the image O(k) reconstructed by the current k-th frame to obtain the 2D intensity image I′(k) of the motion sample on the image sensor plane; Step 5, reverse gradient update: According to the spatiotemporal prior loss function L, the relative loss between the intensity image I′(k) and the real collected intensity image I(k) is calculated, and the weight θ of the MLP network is updated; Step 6, bidirectional memory training: Adopt the method of bidirectionally updating the MLP network parameters frame by frame according to the time axis, save the training parameters of the current k-th frame, inherit the training from the k+1 frame during forward training, and inherit the training from the k-1 frame during reverse training, and stop in advance dynamically according to the minimum loss of forward training. If it is not stopped in advance, the reverse training will be stopped directly when the set maximum number of iterations is reached; Step 7: Repeat steps 2 to 6 above, continuously update the weight θ of the MLP network, and obtain the final reconstructed image sequence O(K) of the dynamic sample.

6. The method according to claim 5, characterized in that In step 5, the spatiotemporal prior loss function L is: L=||I′(k)-I(k)|| 2 +[1-SSIM(I′ k ,I k )]+(ρ s ||D xy O(k,k-1)||1+ρ t ||D t O(k,k-1)||) Where k = 1, 2, ..., K, K is the total number of frames in the dynamic image sequence, SSIM represents structural similarity, and D xy =(D x ,D y ) and D t denote the finite difference operators along the three dimensions of space (x, y) and time t, respectively, ρ S and ρ t is the regularization parameter, and O(k,k-1) represents the complex amplitude image reconstructed by the difference operator applied to the kth and k-1th frames.

7. The method according to claim 5, characterized in that In step six, during reverse training, the dynamic early stopping strategy is: when the minimum loss of forward training is reached, reverse training is stopped early.

Images