A method for reconstructing a lensless microscopic image based on a physically driven neural network

Abstract

The application discloses a lens-free microscopic image reconstruction method based on a physical driving neural network, collects diffraction original images of multiple samples and corresponding displacement coordinates; based on a lens-free microscopic forward imaging model, a physical driving neural network is established: sample weights and coding layer weights are calculated from the collected diffraction original images and are initialized; iteration training parameters are set, the iteration training makes the loss function value tend to be stable and reach the minimum, then the sample weights and the coding layer weights in the physical driving neural network are taken out, the amplitude and the phase of the taken-out sample weights are reconstructed to obtain an intensity map and a phase map of a high-resolution microscopic image, and the method has the advantages that the lens-free microscopic forward imaging model is modeled by the physical driving neural network, has the ability to process complex nonlinear relationships, can well cope with common problems such as scattering and noise in the optical imaging process, and does not need to make a large number of adjustments and calibrations on the imaging system.

Description

Technical Field

[0001] This invention relates to a method for reconstructing microscopic images, and more particularly to a method for reconstructing lensless microscopic images based on a physical-driven neural network. Background Technology

[0002] Traditional lens-based microscopy systems rely on the principles of light refraction and focusing, using a combination of objective and eyepiece magnification to observe microscopic structures. However, their resolution is limited by the diffraction limit, and they also suffer from short depth of field, system complexity, and dependence on high-quality optical components. The need for high-quality optical materials and complex manufacturing processes results in expensive and bulky equipment, unsuitable for miniaturization or portable applications.

[0003] Lensless microscopy, by removing the limitations of traditional optical elements, offers the capabilities of lightweight, high-resolution, large field-of-view, and label-free imaging. Although it still requires complex computation and algorithmic support, it holds broad application prospects in fields such as microbiology, materials science, and medical testing. Lensless microscopy image reconstruction methods are advanced computational imaging techniques designed to significantly improve the resolution, field of view, and other imaging performance of the imaging system by spatially encoding the light field. During lensless microscopy, the system acquires multiple encoded images, each containing information about different aspects of the sample. Using reconstruction algorithms, the high-resolution original image can be recovered. This process not only improves image quality but also makes the system more adaptable to various imaging conditions.

[0004] However, traditional methods for reconstructing lensless microscopic images suffer from significant bottlenecks in application. First, reconstruction efficiency is limited by the complexity of iterative algorithms. Traditional methods rely on iterative reconstruction algorithms, requiring multiple iterations to solve the inverse problem. When faced with high-resolution, large-field-of-view samples, the computational complexity of such algorithms increases exponentially with the amount of data, leading to a sharp decline in reconstruction speed. Second, traditional coding patterns and reconstruction algorithm design rely on human experience. Finally, when solving the inverse problem, traditional methods often sacrifice data consistency due to the inaccuracy of the forward imaging model. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for reconstructing lensless microscopic images based on a physical driving neural network that can quickly and efficiently reconstruct images and effectively improve the quality of image reconstruction.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a method for reconstructing lensless microscopic images based on a physical driving neural network, comprising the following steps:

[0007] Step 1: Acquire the original diffraction images of multiple samples and their corresponding displacement coordinates;

[0008] Step 2: Based on the lensless forward imaging model of microscopy, establish a physical driving neural network that includes the predicted diffraction image.

[0009] Step 3: Calculate and initialize the sample weights and coding layer weights in the physical driving neural network from the acquired original diffraction image;

[0010] Step 4: Set the iterative training parameters, input the multiple original diffraction images and their corresponding displacement coordinates into the physical driving neural network in groups, and iteratively train to make the loss function value stabilize and reach a minimum.

[0011] Step 5: When the loss function value tends to stabilize, extract the sample weights and coding layer weights from the physical driving neural network, and reconstruct the amplitude and phase of the sample weights extracted from the physical driving neural network to obtain the intensity map and phase map of the high-resolution microscopic image.

[0012] Compared with existing technologies, the advantages of this invention are that it models the lensless microscopy forward imaging model with a physical driving neural network. Compared with traditional linear reconstruction methods, it has the ability to handle complex nonlinear relationships and can better cope with common problems such as scattering and noise in optical imaging. The physical driving neural network can adaptively process complex light field information in phase retrieval through learning, enhance the detail representation of the image, and obtain better imaging results. Through the training model of the physical driving neural network, it can adapt well to different imaging scenarios and devices without the need for a lot of adjustments and calibrations to the imaging system.

[0013] The method of this invention can significantly accelerate reconstruction speed when dealing with large amounts of data. The physics-driven neural network learns complex image features and encoding rules from massive amounts of data, avoiding the shortcomings of manually designed complex encoding patterns and reconstruction algorithms in traditional methods. By learning data-driven mapping relationships, the physics-driven neural network can optimize the encoding-decoding process and improve image reconstruction quality. Furthermore, the physics-driven neural network can improve the accuracy of the imaging system, helping the network better understand the relationship between input data and output, making the reconstructed microscopic images more consistent with the real physical scene.

[0014] This invention overcomes the shortcomings of existing technologies while achieving high-precision quantitative phase imaging. This is of great significance for detecting minute changes in samples and assessing their biological characteristics or physical state, and has broad application potential in multiple fields. For example, in disease detection, it can be used to detect pathogens; in environmental monitoring, it can be used for water and soil analysis; and in pathological diagnosis, it can be applied to observe changes in cell and biological tissue structures. Lens-free microscopy provides large field-of-view, high-resolution images, while also offering flexibility and adaptability, which can strongly promote research and applications in various fields.

[0015] The specific method for step 2 is as follows:

[0016] Step 2.1: In the physical driving neural network, the samples and the encoding layer are uniformly modeled as learnable sample weights and encoding layer weights, and the real and imaginary channels corresponding to the complex light field of the lensless microscopy forward imaging model are set in the physical driving neural network respectively.

[0017] Step 2.2: Translate the sample weights according to the displacement coordinates corresponding to the original diffraction image collected in Step 1, and propagate the translated sample weights forward to the coding layer plane in the physical driving neural network to obtain the wavefront weights mapped to the coding layer plane.

[0018] Step 2.3: In the physical-driven neural network, the wavefront weights of the coding layer plane are multiplied by the corresponding coding layer weights, and then propagated to the image plane according to the lensless microscopy forward imaging model to obtain the wavefront weights of the image plane.

[0019] Step 2.4: Take the modulus of the wavefront weights of the image plane and perform a square operation to obtain the predicted diffraction image corresponding to the image plane diffraction image in the lensless forward imaging model, and establish a physical driving neural network containing the predicted diffraction image.

[0020] The specific initialization in step 3 is as follows: the sample weights and coding layer weights are initialized based on the original diffraction image and its corresponding displacement coordinates acquired in step 1.

[0021] The specific method of step 4 is as follows: set the learning rate, decay rate and number of iterations of the physical driving neural network, input the acquired original diffraction images and their corresponding displacement coordinates into the physical driving neural network in groups, and iterate the training until the loss function value tends to stabilize and reach the minimum.

[0022] The specific method of step 5 is as follows: when the iterative training is completed and the loss function value tends to stabilize, the sample weights and coding layer weights in the neural network are extracted, the magnitude of the sample weights is taken and squared to obtain the intensity map of the high-resolution microscopic image; the phase value of the sample weights is taken to obtain the phase map of the high-resolution microscopic image. Attached Figure Description

[0023] Figure 1 This is a flowchart of the lensless microscopic image reconstruction method based on a physical-driven neural network according to the present invention.

[0024] Figure 2 Images of the original images of samples from embodiments of the present invention;

[0025] Figure 3 These are images of the original diffraction images of the samples acquired using the method of this invention;

[0026] Figure 4 These are images of intensity maps of high-resolution microscopic images reconstructed using the method of this invention:

[0027] Figure 5 These are images of phase maps of high-resolution microscopic images reconstructed using the method of this invention. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] Example: A lensless microscopic image reconstruction method based on a physical-driven neural network, as follows Figure 1 As shown, it includes the following steps:

[0030] Step 1, data collection Original diffraction image of Zhang sample and their corresponding displacement coordinates subscript , This represents the total number of original diffraction images, in this embodiment... The sample used in this embodiment is a Wright-stained blood smear.

[0031] Step 2: Based on the lensless forward imaging model, establish a physical driving neural network that includes the predicted diffraction image, specifically as follows:

[0032] Step 2.1: In the physical-driven neural network, the samples and the encoding layer are modeled as learnable sample weights. With coding layer weights ; weight the samples The modeling is set to the following two channels: , , the weights of the coding layer The modeling is set to the following two channels: , ,in, and They represent the real part and the imaginary part, respectively.

[0033] Step 2.2: Weight the samples According to displacement coordinates Perform translation and then apply the translated sample weights to the physical-driven neural network. Forward propagation Distance to the coding layer plane yields the wavefront weights of the coding layer plane. : ,in This indicates the distance that the light wave traveled in free space. ;

[0034] Step 2.3: In the physical-driven neural network, the wavefront weights of the encoding layer plane are... With the corresponding coding layer weights Multiply, and continue propagating forward according to the lensless forward imaging model. Distance to the image plane is used to obtain the wavefront weights of the image plane. , ;

[0035] Step 2.4: Wavefront weights for the image plane By taking the modulus and squaring, the predicted diffraction image corresponding to the image plane diffraction image in the lensless forward microscopy model is obtained. , ,

[0036] Predict the diffraction image As a mathematical model of the diffraction image at the image plane during the imaging process, a physical driving neural network containing the prediction of the diffraction image is established.

[0037] Step 3: Weight the samples and coding layer weights Initialize them as follows: and In this embodiment, N =500.

[0038] Step 4: Set the iterative training parameters and iteratively train the physical driving neural network until the loss function value stabilizes and reaches its minimum: Set the learning rate of the physical driving neural network to 0.01, the decay rate to 0.01, and the number of iterations to 100. Then, collect the data... Zhang diffraction original image and their corresponding displacement coordinates The input is fed into the physical-driven neural network to begin iterative training until the loss function value stabilizes and reaches its minimum. The loss function is expressed as:

[0039] ,

[0040] Here, subscript k The range is The serial number, Less than or equal to , representing the number of original diffraction images input in the iterative process, in this embodiment , Indicates the first k Predicted diffraction images With the k Original diffraction image The square of the L2 norm between them.

[0041] Step 5: During iterative training, the loss function value continuously decreases. When training is complete, the loss function value tends to stabilize and reach its minimum. At this point, the sample weights in the physical driving neural network are extracted. and coding layer weights , is represented as:

[0042] , ,

[0043] Take sample weights The amplitude and square operation is as follows: , to obtain Figure 4 Intensity map of the high-resolution microscopic image shown; sample weights are taken. phase value , to obtain Figure 5 The phase map of the high-resolution microscopic image shown, in which To obtain the amplitude function, To obtain the phase function.

[0044] like Figure 2 and Figure 4 and Figure 5 As shown, the high-resolution microscopic image reconstructed using the method of this embodiment has a significant quality improvement over the original image of the sample, and can also recover phase information, indicating that the method of this embodiment is effective.

Claims

1. A method for reconstructing a lensless microscopic image based on a physically-driven neural network, comprising the following steps: Step 1, collecting a plurality of diffraction original images of samples and corresponding displacement coordinates; Step 2, establishing a physically-driven neural network containing a predicted diffraction image based on a lensless microscopic forward imaging model; Step 3, calculating and initializing sample weights and encoding layer weights in the physically-driven neural network from the collected diffraction original images; Step 4, setting iterative training parameters, inputting the collected plurality of diffraction original images and corresponding displacement coordinates into the physically-driven neural network in groups, and iteratively training until the loss function value tends to be stable and reaches a minimum; Step 5, under the condition that the loss function tends to be stable and reaches a minimum, taking out the sample weights and encoding layer weights in the physically-driven neural network, reconstructing the amplitude and phase of the sample weights in the physically-driven neural network, and obtaining an intensity map and a phase map of a high-resolution microscopic image. The specific method of step 2 is as follows: Step 2.1, in the physically-driven neural network, modeling the sample and the encoding layer as learnable sample weights and encoding layer weights, and respectively modeling and setting the real part channel and the imaginary part channel corresponding to the complex light field of the lensless microscopic forward imaging model in the physically-driven neural network; Step 2.2, translating the sample weights according to the displacement coordinates corresponding to the diffraction original images collected in step 1, and propagating the translated sample weights to the encoding layer plane in the physically-driven neural network to obtain the wavefront weight of the encoding layer plane; Step 2.3, multiplying the wavefront weight of the encoding layer plane with the corresponding encoding layer weight in the physically-driven neural network, and continuing to propagate to the image plane according to the lensless microscopic forward imaging model to obtain the wavefront weight of the image plane; Step 2.4, taking the modulus and squaring the wavefront weight of the image plane to obtain a predicted diffraction image corresponding to the image plane diffraction image in the lensless microscopic forward imaging model, and establishing a physically-driven neural network containing a predicted diffraction image.

2. The method of claim 1, wherein the physical-drive neural network-based reconstruction of a lensless microscopic image is based on a physics-driven neural network. The specific content of the initialization in step 3 is that the sample weights and the encoding layer weights are respectively initialized based on the diffraction original images and corresponding displacement coordinates collected in step 1.

3. The method of claim 1, wherein the physical-drive neural network-based reconstruction of a lensless microscopic image is based on a physics-driven neural network. The specific method of step 4 is to set the learning rate, decay rate and iteration number of the physically-driven neural network, input the collected plurality of diffraction original images and corresponding displacement coordinates into the physically-driven neural network in groups, and iteratively train until the loss function value tends to be stable and reaches a minimum.

4. The method of claim 1, wherein the physical-drive neural network-based reconstruction of a lensless microscopic image is based on a physics-driven neural network. The specific method of step 5 is that when the iterative training is completed and the loss function value tends to be stable and reaches a minimum, the sample weights and the encoding layer weights in the physically-driven neural network are taken out, the amplitude of the sample weights is taken and squared to obtain an intensity map of a high-resolution microscopic image, and the phase value of the sample weights is taken to obtain a phase map of a high-resolution microscopic image.

Images