Lensless microscopy system and method based on self-supervised deep learning

Abstract

The present invention proposes a lensless microscopic imaging system and method based on self-supervised deep learning. The system includes a multi-height imaging module and a phase recovery module based on self-supervised deep learning. The multi-height imaging module is used to collect diffraction images of an observed object at multiple heights that do not require calibration, including a light source, an observed object and an image sensor placed in sequence, wherein the image sensor can move axially to change the height of the diffraction image; the phase recovery module based on self-supervised deep learning is used to instantly reconstruct the amplitude and phase information of the observed object according to the diffraction image and simultaneously jointly optimize the imaging model parameters. The present invention introduces self-supervised deep learning into the existing lensless imaging system, collects multi-height diffraction image information of the sample, and uses self-supervised deep learning to reconstruct the sample image and jointly optimize the imaging model parameters, thereby achieving label-free, unconstrained, high-performance, and high-quality imaging of the sample.

Description

Technical Field

[0001] The present invention relates to the field of microscopic imaging technology, and in particular to a lensless microscopic imaging system and method based on self-supervised deep learning. Background Art

[0002] With the rapid growth of optofluidic on-chip imaging, wavefront sensing, cell counting, fluorescence imaging, digital pathology, and endoscopy applications, lensless imaging has recently become one of the most promising microscopy technologies for full-field observation. In a lensless microscopy system, the target object or sample is placed very close to the image sensor, illuminated by coherent (or partially coherent) light, and the imaging sensor directly records the diffraction pattern generated by the object. The focused amplitude intensity and phase information of the target object are then recovered from the diffraction measurement using a phase recovery algorithm. Since the lensless microscopy system does not require any optical lenses, but directly records the diffraction measurement for imaging processing, it breaks through the restrictive relationship between imaging resolution and field of view in traditional optical microscopes and improves imaging capabilities.

[0003] However, the existing phase recovery methods still have poor phase reconstruction performance in the complex domain. One reason is that the physical parameters of the imaging process are not accurately estimated, and small imaging parameter errors have a huge impact on the accuracy of phase imaging. Another reason is that the complex domain recovery at any given amplitude (0-1) and phase (0-2π) scale lacks an effective and reliable physical description of imaging, so even with accurate imaging model parameters, existing solutions cannot provide robust reconstruction performance.

[0004] In recent years, deep learning technology has flourished and has been widely used in many fields. Introducing deep learning into computational imaging problems has shown great potential. Deep learning has excellent performance in optical tomography, computational ghost imaging, digital holography, scatter imaging, phase imaging and other fields. The current mainstream supervised deep learning methods rely on a large number of labeled data sets. However, in the microscopic field, it is generally difficult to provide a large number of training samples for deep learning. Therefore, researchers are trying to explore the possibility of using self-supervised learning. Summary of the invention

[0005] The purpose of the present invention is to propose a lensless microscopic imaging system and method based on self-supervised deep learning, which can achieve label-free, unconstrained, high-performance, and high-quality imaging of samples.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A lensless microscopic imaging system based on self-supervised deep learning includes a multi-height imaging module and a phase recovery module based on self-supervised deep learning. The multi-height imaging module is used to collect diffraction images of an observed object at multiple heights that do not require calibration, and includes a light source, an observed object and an image sensor placed in sequence, wherein the image sensor can move axially to change the height of the diffraction image; the phase recovery module based on self-supervised deep learning is used to instantly reconstruct the amplitude and phase information of the observed object according to the diffraction image and simultaneously jointly optimize the imaging model parameters.

[0008] Furthermore, the light source is a coherent or partially coherent light source.

[0009] The present invention also provides a lensless microscopic imaging method based on self-supervised deep learning, comprising the following steps: a light source emits coherent or partially coherent light, which is incident on an observed object, and then an image sensor collects scattered light of the observed object, and obtains diffraction images of the observed object at multiple different heights by axially moving the image sensor; based on the collected diffraction images at multiple different heights, self-supervised deep learning is used to perform image reconstruction, and simultaneously jointly optimizes and obtains accurate imaging model parameters.

[0010] Furthermore, the specific implementation steps of the self-supervised deep learning are as follows:

[0011] Step 1: construct a multi-layer perceptron neural network and initialize the network model. Its input is the two-dimensional spatial coordinates, and its output is the reconstructed amplitude A and phase φ images. Initialize the imaging model parameters.

[0012] Step 2: According to the reconstructed amplitude A and phase φ images, and the point spread function PSF, N reconstructed diffraction images at different heights can be obtained, and the sum of the L1 norms of the differences between the reconstructed diffraction images and the real diffraction images collected at the corresponding heights is used as the loss function;

[0013] Step 3: Use BP back propagation algorithm to update network parameters and imaging model parameters;

[0014] Step 4: According to step 3, the network is iteratively updated using N diffraction images reconstructed at different heights for self-supervision, and then M rounds are repeated to obtain the reconstructed amplitude and phase images and the jointly optimized imaging model parameters.

[0015] Furthermore, in the step 1, the spatial two-dimensional coordinates of the object to be measured are mapped to a high-dimensional Fourier space using position coding, and then used as input of the neural network to enhance the neural network's ability to represent high-frequency information.

[0016] Furthermore, in the step 4, after the first joint optimization is completed, the weights and bias parameters of the network are randomly reinitialized, and a second joint optimization is performed in combination with the microscopic imaging parameters obtained by the first joint optimization to optimize more accurate microscopic imaging parameters.

[0017] The phase reconstruction performance of existing traditional lensless microscopy in the complex domain is still poor. Research on lensless imaging based on deep learning mainly focuses on supervised learning. However, since it is very difficult to construct a data set with a large number of training samples in the microscopic field, this method cannot provide a lensless imaging solution that is unlabeled and requires only a small amount of data. The present invention uses deep learning technology to solve the inverse problem in the microscopic imaging process, namely the phase recovery problem, and uses a neural network with an embedded physical imaging process to establish an implicit mapping from the spatial coordinates of the target object to its amplitude and phase information; by adopting a self-supervised learning method, only a small number of random diffraction snapshots of different heights are collected for the microscopic sample, and the complex domain information of the target object can be accurately restored; the physical parameters of the imaging model and the implicit mapping function are jointly optimized to obtain more accurate imaging model parameters and better amplitude intensity and phase information. In this way, a solution that is simpler and faster to use, has a higher quality of recovery effect, and requires very little training data can be provided for the lensless microscopic imaging problem.

[0018] The innovation and advantages of the present invention are:

[0019] (1) The present invention introduces self-supervised deep learning into the lensless imaging scheme. By only collecting a small number of random height diffraction snapshots of the microscopic sample, the complex domain information of the target object can be accurately restored, thus achieving label-free, unconstrained, high-quality imaging.

[0020] (2) The present invention collects only a few diffraction images of the sample at random heights each time it is imaged, and uses self-supervised deep learning to reconstruct the amplitude and phase images of the collected diffraction images. This algorithm has good potential for obtaining fast imaging and efficient computational imaging. At the same time, the imaging model parameters are jointly optimized, and accurate imaging model parameters can be obtained without a complex calibration process, which can make the lensless imaging system easier to operate and improve efficiency.

[0021] (3) The imaging performance of the lensless microscopic imaging scheme of the present invention is superior to that of the traditional lensless imaging method and can be used in practical microscopic imaging applications. In addition, when the physical parameter estimation of the imaging process is inaccurate, the traditional lensless imaging method cannot reconstruct the amplitude and phase images of the observed object, or the reconstructed image has obvious reconstruction artifacts. However, the present invention is based on self-supervised deep learning and joint training, which can reconstruct high-quality amplitude and phase images, and can also optimize and obtain accurate imaging model parameter estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a structural block diagram of the lensless microscopic imaging system based on self-supervised deep learning of the present invention;

[0023] Figure 2 This is a flow chart of the phase recovery algorithm based on self-supervised deep learning of the present invention;

[0024] Figure 3 Imaging results of the precision plate obtained by the imaging method of the embodiment of the present invention, (a) self-supervised deep learning algorithm, (b) GS algorithm + optimized height, (c) GS algorithm + random height;

[0025] Figure 4 This is the imaging result of biological cells obtained by the imaging method of an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Embodiments of the present invention are described in detail below with reference to the accompanying drawings, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0027] Reference Figure 1 As shown, the system of the present invention includes a multi-height imaging module and a phase recovery module based on self-supervised deep learning, which introduces deep learning into the lensless imaging scheme, and can achieve label-free, unconstrained, high-performance, and high-quality imaging of samples. Specifically, the multi-height imaging module includes a light source, an observed object, and an image sensor placed in sequence, the light source emits coherent or partially coherent light with a wavelength of 638nm, which is incident on the sample, and then the image sensor collects the scattered light of the sample. The image sensor is mounted on a motor for axial movement, and is used to collect diffraction images of the observed object at multiple random heights; according to the collected multiple sample diffraction images, the phase recovery module based on self-supervised deep learning uses self-supervised deep learning to instantly reconstruct amplitude and phase images, and simultaneously jointly optimizes and obtains accurate imaging model parameters, such as diffraction image height.

[0028] Figure 2 This is the self-supervised deep learning algorithm flow of the present invention. The algorithm can well mine imaging information and obtain high-quality lens-free imaging. Among them, the network model adopts a multi-layer perceptron, including 8 hidden layers, 256 hidden units in each layer, and the activation function is the ReLU function. In order to reconstruct more details in the amplitude and phase images, this embodiment also first position encodes the spatial coordinates of the sample and then inputs them into the network. Specifically, the implementation method is as follows:

[0029] (1) Building a physics-inspired self-supervised neural network

[0030] like Figure 2 As shown, this embodiment uses a multi-layer perceptron network to establish an implicit mapping f from the spatial coordinates (x, y) of the target sample to its amplitude A and phase φ θ to achieve phase recovery, where θ represents the network parameters; then the predicted amplitude and phase (A, φ) are combined with the point spread function PSF z The reconstructed diffraction image can be obtained, where z represents the height of the diffraction image; finally, the reconstructed diffraction image and the real diffraction image I z The sum of the L1 norms of the differences between them is used to construct the following loss function:

[0031]

[0032] (2) Joint optimization of microscopy parameters and phase retrieval

[0033] By jointly optimizing the microscopic imaging parameters and the implicit mapping between the input spatial coordinates and the output amplitude and phase, a very small amount of diffraction measurements can be used to accurately describe the physical process of imaging and optimally reconstruct the amplitude and phase information of the target object. In actual implementation, the optimization accuracy of the microscopic imaging parameters is very important. If the convergence value of the jointly optimized microscopic imaging parameters is not accurate enough, the neural network will fall into the local minimum and phase reconstruction distortion problem. Therefore, this embodiment re-randomly initializes the weights and bias parameters of the network after the first joint optimization, and combines the microscopic imaging parameters obtained by the first joint optimization to perform a second joint optimization, so that more accurate microscopic imaging parameters can be optimized.

[0034] (3) Using position encoding to improve the network's ability to fit high-frequency information

[0035] Since neural networks tend to learn low-frequency functions and perform poorly in fitting high-frequency functions, in order to better fit the mapping of spatial coordinates to the high-frequency parts of the amplitude and phase images, the spatial coordinates are mapped to a high-dimensional Fourier space using position encoding before being input into the network:

[0036] PE(x,y)={cos(2πsσ i x), sin(2πsσ i x)}∪{cos(2πsσ i y), sin(2πsσ i y)},

[0037] i={1J2 ， …, 10}

[0038] Where s is a scaling factor, σ i Obeys normal distribution.

[0039] Reference Figure 3 As shown in FIG. 1 , the system of this embodiment collects 8 diffraction images of the precision plate at random heights, and reconstructs the amplitude image using the phase recovery method based on self-supervised deep learning proposed in the present invention. The reconstructed image obtained in this embodiment is shown in FIG. Figure 3 In Figure (a), the precision plate can be clearly observed, and the image is high-contrast and low-noise. Therefore, the phase recovery method based on self-supervised deep learning in the present invention improves the imaging quality of the lensless system. However, in the case of highly random and uncalibrated diffraction in the imaging process, the traditional GS (Gerchberg-Saxton) phase recovery algorithm cannot accurately reconstruct the amplitude and phase images of the observed object, or the reconstructed image has obvious reconstruction artifacts, such as Figure 3 However, the self-supervised deep learning phase recovery method of the present invention can still reconstruct high-quality amplitude and phase images under random diffraction height conditions, and simultaneously obtain accurate diffraction height estimation using joint optimization. Figure 3 As shown in Figure (b), the quality of the reconstructed image of the traditional GS algorithm is significantly improved when the diffraction height estimation obtained by the optimization of the present invention is used, indicating that the diffraction height estimation obtained by the optimization of the present invention is accurate.

[0040] Reference Figure 4 As shown in FIG. 1 , the system of this embodiment collects 8 random height diffraction images of biological cells, and uses the phase recovery method based on self-supervised deep learning proposed in the present invention to reconstruct the amplitude and phase images. The reconstructed image obtained in this embodiment is as follows: Figure 4 In the case of the diffraction height estimation optimized by the present invention, the traditional GS phase recovery algorithm cannot reconstruct the complete amplitude and phase image of the observed object with high quality, such as Figure 4 However, the self-supervised deep learning phase recovery method of the present invention can still reconstruct high-quality, low-noise, complete amplitude and phase images under random diffraction height conditions, and simultaneously obtain accurate diffraction height estimation using joint optimization.

[0041] The present invention embeds the physical model of lensless microscopy into the network model, does not require a large amount of labeled data, can self-supervise the training and optimization of the network, and only needs to collect a small amount of diffraction images for training to ultimately generate high-quality amplitude and phase images of the object to be measured.

Claims

1. A lensless microscopic imaging system based on self-supervised deep learning, comprising a multi-height imaging module and a phase recovery module based on self-supervised deep learning, characterized in that: The multi-height imaging module is used to collect diffraction images of the observed object at multiple heights that do not require calibration, and includes a light source, an observed object, and an image sensor placed in sequence, wherein the image sensor can perform axial movement to change the height of the diffraction image; the phase recovery module based on self-supervised deep learning is used to instantly reconstruct the amplitude and phase information of the observed object according to the diffraction image and simultaneously jointly optimize the imaging model parameters; The specific implementation steps of the self-supervised deep learning are as follows: Step 1: construct a multi-layer perceptron neural network and initialize the network model. Its input is the two-dimensional spatial coordinates, and its output is the reconstructed amplitude A and phase φ images. Initialize the imaging model parameters. Step 2: According to the reconstructed amplitude A and phase φ images, and the point spread function PSF, N reconstructed diffraction images at different heights can be obtained, and the sum of the L1 norms of the differences between the reconstructed diffraction images and the real diffraction images collected at the corresponding heights is used as the loss function; Step 3: Use BP back propagation algorithm to update network parameters and imaging model parameters; Step 4: According to step 3, the network is iteratively updated using N diffraction images reconstructed at different heights for self-supervision, and then M rounds are repeated to obtain the reconstructed amplitude and phase images and the jointly optimized imaging model parameters.

2. The lensless microscopy system based on self-supervised deep learning according to claim 1, characterized in that: The light source is a coherent or partially coherent light source.

3. A lensless microscopy imaging method based on self-supervised deep learning, characterized in that: The method comprises the following steps: a light source emits coherent or partially coherent light, which is incident on an observed object; an image sensor collects scattered light of the observed object; and the image sensor is moved axially to obtain diffraction images of the observed object at multiple different heights; self-supervised deep learning is used to perform image reconstruction based on the collected diffraction images at multiple different heights, and simultaneously jointly optimizes and obtains accurate imaging model parameters; the specific implementation steps of the self-supervised deep learning are as follows: Step 1: construct a multi-layer perceptron neural network and initialize the network model. Its input is the two-dimensional spatial coordinates, and its output is the reconstructed amplitude A and phase φ images. Initialize the imaging model parameters. Step 2: According to the reconstructed amplitude A and phase φ images, and the point spread function PSF, N reconstructed diffraction images at different heights can be obtained, and the sum of the L1 norms of the differences between the reconstructed diffraction images and the real diffraction images collected at the corresponding heights is used as the loss function; Step 3: Use BP back propagation algorithm to update network parameters and imaging model parameters; Step 4: According to step 3, the network is iteratively updated using N diffraction images reconstructed at different heights for self-supervision, and then M rounds are repeated to obtain the reconstructed amplitude and phase images and the jointly optimized imaging model parameters.

4. The lensless microscopy imaging method based on self-supervised deep learning according to claim 3, characterized in that: In the step 1, the spatial two-dimensional coordinates of the object to be measured are mapped to a high-dimensional Fourier space using position coding, and then used as the input of the neural network to enhance the neural network's ability to represent high-frequency information.

5. The lensless microscopy imaging method based on self-supervised deep learning according to claim 3, characterized in that: In the step 4, after the first joint optimization is completed, the weights and bias parameters of the network are randomly re-initialized, and the second joint optimization is performed in combination with the microscopic imaging parameters obtained by the first joint optimization to optimize and obtain more accurate microscopic imaging parameters.

Images