Three-dimensional reconstruction method based on Fourier laminated imaging and reconstruction system thereof

Abstract

The invention belongs to the related technical field of three-dimensional reconstruction, and discloses a three-dimensional reconstruction method and system based on Fourier laminated imaging, and the method comprises the steps: S1, constructing a two-dimensional and three-dimensional explicit feature body with the feature size smaller than a target feature size; s2, performing feature extraction on each feature body by adopting a multi-layer sensing network to obtain phase two-dimensional distribution of system optical aberration and real part and imaginary part three-dimensional distribution of sample refractive index distribution; performing an interpolation up-sampling operation during the period to recover the feature size; s3, reconstructing a two-dimensional light intensity graph of the sample through the forward imaging model; s4, calculating the difference loss between the collected two-dimensional light intensity image and the reconstructed two-dimensional light intensity image, and reversely adjusting the feature body and the multi-layer sensing network; and circularly executing the steps S2 to S4 until loss convergence, and outputting real part three-dimensional distribution and imaginary part three-dimensional distribution of the current refractive index. By means of the method, efficient and high-quality three-dimensional reconstruction can be achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to three-dimensional reconstruction, and more specifically, relates to a three-dimensional reconstruction method based on Fourier stack imaging and a reconstruction system thereof. Background Art

[0002] Fourier ptychographic microscopy (FPM) is a newly developed imaging method that is used in optical microscopy, biomedicine, life sciences and other fields to obtain large-field-of-view, high-resolution microscopic images. However, in the FPM imaging system, the pupil-image aberration and incoherent light imaging of the optical elements used significantly reduce the quality of the reconstruction results. In order to suppress the reconstruction artifacts caused by image noise and system aberrations, traditional methods often use iterative algorithms with "space-frequency domain" constraints to splice the two-dimensional scattering image information into the three-dimensional spectrum space by gradient backprojection to obtain high-quality three-dimensional reconstruction results of the sample. However, these schemes usually require a large number of illumination angles and multiple regularization constraints, and require a large number of iterations, which greatly limits the reconstruction efficiency.

[0003] With the development of deep learning technology, there are also studies on the use of deep learning technology to achieve three-dimensional reconstruction, such as Chinese patent CN111260556B-A Fourier stack microscopy reconstruction method based on deep convolutional neural network, which first trains the Fourier stack reconstruction network and then uses the trained network to reconstruct the complex amplitude information. Although this method can improve the reconstruction efficiency to a certain extent, it focuses on training a generalized network, and the trained network can reconstruct different samples. Therefore, a large number of samples are required to form a training data set in the early stage of training to train the network in large quantities, which also limits the reconstruction efficiency. When the training samples are insufficient, it is difficult to obtain a reconstruction network with good performance.

[0004] Therefore, how to achieve efficient and high-quality three-dimensional reconstruction based on images of the Fourier stack system is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a three-dimensional reconstruction method based on Fourier stack imaging and a reconstruction system thereof, the purpose of which is to achieve efficient and high-quality three-dimensional reconstruction based on images of the Fourier stack system.

[0006] To achieve the above object, the present invention provides a three-dimensional reconstruction method based on Fourier stack imaging, which comprises:

[0007] Step S1: Construct a two-dimensional explicit feature volume and a three-dimensional explicit feature volume with feature sizes smaller than the target feature size respectively, and initialize the values of the features of each pixel in the feature volumes. The target feature size is the size of the pixel map to be reconstructed. The two dimensions correspond to width and height, and the three dimensions correspond to width, height, and depth.

[0008] Step S2: Use a first multi-layer perceptron network to extract features from the two-dimensional explicit feature volume to obtain the two-dimensional phase distribution of the system optical aberration, and use a second multi-layer perceptron network to extract features from the three-dimensional explicit feature volume to obtain the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the sample refractive index distribution. Among them, an interpolation upsampling operation is performed between adjacent two layers in each multi-layer perceptron network to restore the feature size of the extracted features to the target feature size.

[0009] Step S3: Input the two-dimensional phase distribution, the three-dimensional real part distribution, and the three-dimensional imaginary part distribution into the forward imaging model to reconstruct the two-dimensional intensity maps of the sample at different illumination angles.

[0010] Step S4: Calculate the difference loss between the two-dimensional intensity map of the Fourier ptychography imaging system collected at the same illumination angle and the reconstructed two-dimensional intensity map, and reversely adjust the values of the features of each pixel in the two-dimensional explicit feature volume and the three-dimensional explicit feature volume and the network parameters of each multi-layer perceptron network based on the difference loss. Loop and execute Steps S2 to S4 until the obtained difference loss converges and then end the loop.

[0011] Step S5: Output the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the current refractive index distribution to obtain the three-dimensional complex refractive index distribution of the sample.

[0012] Optionally, the feature sizes of the two-dimensional explicit feature volume and the three-dimensional explicit feature volume are 0.5 to 0.7 times the system imaging feature size.

[0013] Optionally, the feature of each pixel in the two-dimensional explicit feature volume and the three-dimensional explicit feature volume is an N-dimensional tensor, where 5 ≤ N ≤ 8.

[0014] Optionally, both the first multi-layer perceptron network and the second multi-layer perceptron network have M layers. The interpolation upsampling operation includes performing an upsampling operation on the features output by the (M - 1)-th layer, and then inputting the upsampling result restored to the target feature size into the last layer. The output channel of the last layer of the first multi-layer perceptron network is 1, and the two-dimensional phase distribution is output. The output channels of the last layer of the second multi-layer perceptron network are 2, and the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the sample refractive index distribution are output respectively. 4 ≤ M ≤ 7.

[0015] Optionally, in each multi-layer perceptron network, except for the last layer, the number of output channels of other layers is 16.

[0016] Optionally, the difference loss includes two parts, namely the loss of the image and the loss of the high-frequency part obtained after the wavelet transform of the image.

[0017] Optionally, the loss is the L2 loss.

[0018] The present invention also provides a three-dimensional reconstruction system based on Fourier ptychography, which includes:

[0019] A feature volume construction unit for respectively constructing a two-dimensional explicit feature volume and a three-dimensional explicit feature volume with a feature size smaller than the target feature size and initializing the feature values of each pixel point of the feature volume, where the target feature size is the size of the pixel map to be reconstructed, the two dimensions correspond to width and height, and the three dimensions correspond to width, height, and depth;

[0020] A feature extraction unit for extracting features from the two-dimensional explicit feature volume by using a first multi-layer perceptron network to obtain the two-dimensional phase distribution of the system optical aberration, and extracting features from the three-dimensional explicit feature volume by using a second multi-layer perceptron network to obtain the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the sample refractive index distribution; wherein, an interpolation upsampling operation is performed between adjacent two layers of networks in each multi-layer perceptron network to restore the feature size of the extracted features to the target feature size;

[0021] A two-dimensional reconstruction unit for inputting the two-dimensional phase distribution, the three-dimensional real part distribution, and the three-dimensional imaginary part distribution into a forward imaging model to reconstruct the two-dimensional intensity maps of the sample at different illumination angles;

[0022] An adjustment unit for calculating the difference loss between the two-dimensional intensity map of the Fourier ptychography system collected at the same illumination angle and the reconstructed two-dimensional intensity map, and reversely adjusting the feature values of each pixel point of the two-dimensional explicit feature volume and the three-dimensional explicit feature volume and the network parameters of each multi-layer perceptron network based on the difference loss;

[0023] An output unit for outputting the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the current refractive index distribution when the obtained difference loss converges to obtain the three-dimensional complex refractive index distribution of the sample.

[0024] The present invention also provides a computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the above.

[0025] The present invention also provides a computer program product, including a computer program or instruction, wherein the computer program or instruction, when executed by a processor, implements the steps of the method described in any one of the above.

[0026] Generally speaking, compared with the prior art through the above technical solutions conceived by the present invention, the present invention mainly has the following beneficial effects:

[0027] 1. The present invention belongs to a self-supervised training method. When performing three-dimensional reconstruction for each sample, the above steps S1 to S5 need to be executed. During self-supervised training, only the image acquisition of the sample to be reconstructed is involved. By minimizing the composite loss function of the reconstructed two-dimensional intensity map and the actually acquired two-dimensional intensity map, the update of the feature volume and the multi-layer perceptron network is realized, and then the three-dimensional refractive index distribution and system aberration of the sample are reconstructed to obtain the three-dimensional complex refractive index distribution of the sample. Since this self-supervised training method only needs to collect the images of the target sample as loss labels, the data acquisition work is simple and does not require a large number of training samples, and the three-dimensional reconstruction of the target sample can be very easily realized.

[0028] 2. In the present invention, a feature volume that can be updated is set, which has pixel-level frequency representation. Compared with the traditional INR network, the present invention introduces a feature volume that can be updated, so that the network has no frequency bias, can greatly improve the fitting of high-frequency information, can improve the high-frequency fitting efficiency of the network, and combines interpolation upsampling to retain the continuity of the network, thereby improving the training efficiency of the network and quickly realizing the three-dimensional reconstruction of the sample.

[0029] 3. The present invention sets two branches. On the one hand, the combination of the two-dimensional feature volume and the MLP is used to realize the fitting process of the system pupil function and embed it into the forward imaging model. Since the phase part of the pupil function can represent the optical aberration of the system, this solution can realize the decoupling of the system aberration and eliminate its influence on the reconstruction result, improving the reconstruction quality; on the other hand, the combination of the three-dimensional feature volume and the MLP is used to realize the reconstruction of the real and imaginary parts of the refractive index. The real and imaginary parts correspond to the scattering and absorption abilities of the sample to light. Therefore, the present invention can be compatible with the reconstruction of transmissive and absorptive samples.

[0030] 4. In one embodiment, each pixel feature is characterized as a multi-dimensional tensor, so that the subsequent network can more accurately capture and represent the subtle changes in space.

[0031] 5. In one embodiment, wavelet transform is introduced into the loss. The reconstructed image and the actual image are subjected to wavelet transform, and the loss of the high-frequency information in the wavelet transforms of the two is calculated. Since this loss focuses on the high-frequency part, it can enhance the network's ability to fit high-frequency information, significantly reduce the number of iterations on the basis of ensuring the reconstruction quality, and greatly improve the reconstruction efficiency of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flowchart of the steps of the three-dimensional reconstruction method in an embodiment of the present invention;

[0033] Figure 2 It is the data flow diagram of the 3D reconstruction method in an embodiment of the present invention.

[0034] Figure 3 It is the comparison diagram of the reconstruction results between the 3D reconstruction algorithm in an embodiment of the present invention and the traditional algorithm on simulated data;

[0035] Figure 4 It is the comparison diagram of the results between the 3D reconstruction algorithm in an embodiment of the present invention and the traditional algorithm on measured data (Caenorhabditis elegans);

[0036] Figure 5 It is the reconstruction result diagram of the 3D reconstruction algorithm in an embodiment of the present invention for absorption-type samples (blood smear) and transmission-type samples (Caenorhabditis elegans). Detailed implementation manners

[0037] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] Embodiment 1

[0039] The present invention proposes a 3D reconstruction method based on Fourier ptychography, as Figure 1 shown is the step flowchart of the 3D reconstruction method in an embodiment of the present invention, as Figure 2 shown is the data flow diagram of the 3D reconstruction method in an embodiment of the present invention. This method includes multiple execution steps, and each of the steps will be introduced in detail below.

[0040] Step S1: Construct a two-dimensional explicit feature volume and a three-dimensional explicit feature volume with feature sizes smaller than the target feature size respectively, and initialize the feature values of each pixel point of the feature volume.

[0041] Among them, the target feature size is the size of the pixel map to be reconstructed. In two dimensions, it corresponds to width and height, and in three dimensions, it corresponds to width, height and depth.

[0042] Specifically, the target feature size refers to the number of pixels to be finally reconstructed and output, including the number of pixels in the width direction, the number of pixels in the height direction, and the number of pixels in the depth direction, that is, the width, height, and depth dimensions. This target feature size can be set according to the parameters of the imaging system. A size reduction ratio can be set. The two-dimensional explicit feature volume has width and height, which are proportionally reduced from the width and height of the target feature size. The three-dimensional explicit feature volume has width, height and depth, which are proportionally reduced from the width, height and depth of the target feature size.

[0043] In one embodiment, the downscaling ratio is 0.5 to 0.7. Limiting the downscaling ratio within this range can, on the one hand, reduce the number of computational parameters of the network and improve the network training speed; on the other hand, in combination with the subsequent upsampling operation, it can introduce a smoothness prior and suppress the generation of high-frequency noise.

[0044] Taking the downscaling ratio of 0.6 as an example, assuming the set target feature size is 1024 (width) × 1024 (height) × 120 (depth), then the feature size of the two-dimensional explicit feature volume is 614 (width) × 614 (height), and the feature size of the three-dimensional explicit feature volume is 614 (width) × 614 (height) × 72 (depth).

[0045] In the present invention, both the two-dimensional explicit feature volume and the three-dimensional explicit feature volume can be updated during iteration, which respectively represent the network input parameters for the system optical aberration and the target three-dimensional refractive index distribution.

[0046] In one embodiment, considering that the target refractive index distribution and the system aberration have information representations with higher dimensions than the spatial coordinates, each pixel feature in the constructed feature volume is an N-dimensional tensor, where 5 ≤ N ≤ 8. For example, N = 6. Representing each pixel feature as a multi-dimensional tensor can enable the subsequent network to more accurately capture and represent the subtle changes in space.

[0047] Step S2: Use the first multi-layer perceptron network to extract features from the two-dimensional explicit feature volume to obtain the two-dimensional phase distribution of the system optical aberration, and use the second multi-layer perceptron network to extract features from the three-dimensional explicit feature volume to obtain the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the sample refractive index distribution; wherein, an interpolation upsampling operation is performed between adjacent two layers of the network in each multi-layer perceptron network to restore the feature size of the extracted features to the target feature size.

[0048] Specifically, a multi-layer perceptron (MLP) is a feedforward neural network that includes an input layer, hidden layers, and an output layer, and its hidden layers can be multiple layers. By using a multi-layer perceptron, due to the smoothness prior of the network, high-frequency noise and artifacts in the reconstruction result can be effectively suppressed, and the three-dimensional reconstruction quality under sparse illumination angles can be improved.

[0049] The present invention sets two branches, namely the first branch and the second branch. On the first branch, use the first multi-layer perceptron network to extract features from the two-dimensional explicit feature volume to obtain the two-dimensional phase distribution of the system optical aberration. On the second branch, use the second multi-layer perceptron network to extract features from the three-dimensional explicit feature volume to obtain the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the sample refractive index distribution.

[0050] Since the size of the constructed feature volume is smaller than the target feature size, which is equivalent to performing a downsampling operation on the target pixel map, therefore, in the present invention, an interpolation upsampling operation is set in the multi-layer perceptron network of each branch to restore the feature size extracted by each branch to the target feature size.

[0051] Experiments have found that if no downsampling is performed, it will result in still having relatively large reconstruction noise and artifacts. In the present invention, a downsampled feature volume is first constructed, and then the size is restored by upsampling during feature extraction, which is equivalent to setting a blurred feature volume at the front end, and then making the feature volume exhibit local continuity through interpolation upsampling, thereby suppressing the noise of the reconstruction result and improving the reconstruction quality.

[0052] In a specific embodiment, both the first multi-layer perceptron network and the second multi-layer perceptron network have M layers. Performing the interpolation upsampling operation includes performing an upsampling operation on the features output by the (M - 1)-th layer, and then inputting the upsampling result restored to the target feature size into the last layer. The output channel of the last layer of the first multi-layer perceptron network is 1, and the output is the two-dimensional phase distribution (i.e., the pupil function). The output channel of the last layer of the second multi-layer perceptron network is 2, and respectively outputs the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the sample refractive index distribution. Specifically, the network data satisfies 4 ≤ M ≤ 7. Among them, specifically, "trilinear" interpolation upsampling can be selected to enlarge its size to the target size.

[0053] In a specific embodiment, in each multi-layer perceptron network, except for the last layer, the number of output channels of other layers is 16. Except for the last layer, a ReLU non-linear activation layer is connected in series after each layer of MLP to ensure the non-linear fitting ability of the network and at the same time control the computational cost.

[0054] For example, the multi-layer perceptron network can be 5 layers, and the number of output channels of the first 4 layers is 16. To enhance the spatial continuity, the output of the 4th layer is upsampled and interpolated to the target size, and the result is output through the last layer of MLP. The output of the second branch represents the real part and the imaginary part of the reconstructed target refractive index distribution, corresponding to the scattering and absorption capabilities of the sample to light. Furthermore, this solution can be compatible with the reconstruction of transmissive and absorptive samples.

[0055] Specifically, this solution realizes the mapping from the high-dimensional feature volume vector to the target refractive index distribution and the system aberration based on the MLP. A mapping network is constructed using a 5-layer MLP. For the two branches of reconstructing the pupil function and the refractive index distribution, except for the last layer of the MLP, the number of output channels of each layer of the MLP is 16, and a non-linear activation function is connected in series. The ReLU function is used as the non-linear activation function to balance the network performance and the non-linear expression ability. The output channel of the last layer of the MLP for the branch of reconstructing the pupil function is 1 (the phase of the system optical aberration); the number of output channels for the branch of reconstructing the refractive index distribution is 2, which respectively output the real part and the imaginary part of the sample refractive index distribution. Among them, the real part and the imaginary part of the refractive index distribution can be used to characterize the scattering and absorption ability of the sample to light. Furthermore, this method can be compatible with the reconstruction of transmissive and absorptive samples.

[0056] Step S3: Input the two-dimensional phase distribution, the three-dimensional real part distribution, and the three-dimensional imaginary part distribution into the forward imaging model to reconstruct the two-dimensional intensity map of the sample at different illumination angles.

[0057] Specifically, the Fourier ptychography system has multiple LED light sources, and different LEDs have different illumination angles.

[0058] Specifically, a multi-layer Born approximation model can be selected to construct a forward physical model from the refractive index distribution of the sample under each LED illumination (corresponding to different illumination angles) to the imaging result of the camera. Input the refractive index distribution output by the MLP into the model, and combine the two-dimensional phase distribution output by the network (i.e., the phase part of the pupil function) to obtain the simulated imaging data at different illumination angles, that is, reconstruct the two-dimensional intensity map of the sample at different illumination angles. Since the phase part of the pupil function can characterize the aberration of the system, this method can realize the aberration-free reconstruction of the three-dimensional refractive index function of the sample.

[0059] Step S4: Calculate the difference loss between the two-dimensional intensity map of the Fourier ptychography imaging system collected at the same illumination angle and the reconstructed two-dimensional intensity map, and inversely adjust the values of the features of each pixel point of the two-dimensional explicit feature volume and the three-dimensional explicit feature volume and the network parameters of each multi-layer perceptron network based on the difference loss; loop and execute steps S2 to S4 until the obtained difference loss converges and then end the loop.

[0060] Using the multi-Born diffraction model as the forward imaging model, project the three-dimensional refractive index distribution reconstructed by the network layer by layer through scattering to obtain the two-dimensional intensity map at each illumination angle, so as to construct a loss function with the actually collected intensity image and form a training closed-loop. At the same time, the pupil function generated by this solution is used as the pupil phase part in the model. By updating it, the aberration-free reconstruction of the target refractive index can be realized.

[0061] Understandably, before the algorithm runs, two-dimensional intensity maps of the Fourier ptychography system under different LED illuminations are collected first, serving as the labels used to calculate the loss during each iteration.

[0062] For example, 50 illumination angles are selected, and two-dimensional intensity maps at each illumination angle are collected respectively, obtaining 50 two-dimensional intensity maps as the reference terms for calculating the loss. Therefore, in step S3, it is also necessary to reconstruct the two-dimensional intensity maps at these 50 illumination angles as the output terms for calculating the loss. When calculating the loss between the output terms and the reference terms for calculation, the difference loss between the collected two-dimensional intensity maps and the reconstructed two-dimensional intensity maps at the same illumination angle can be calculated respectively, and then the difference losses at all illumination angles are integrated to obtain the final difference loss.

[0063] In a specific operation, the difference loss includes two parts: 1) the L2 loss between the generated image and the actually collected image; 2) the L2 loss of the high-frequency part after the generated image and the actually collected image are subjected to wavelet transform. In this embodiment, wavelet transform is introduced when calculating the loss, the reconstructed simulated imaging data and the actually captured data are subjected to wavelet transform, and the L2 loss of the high-frequency information in the wavelet transforms of the two is calculated. Since this loss focuses on the high-frequency part, it can enhance the network's fitting ability for high-frequency information, significantly reduce the number of iterations on the basis of ensuring the reconstruction quality, and greatly improve the reconstruction efficiency of the algorithm.

[0064] In a specific operation, by minimizing the difference between the simulated imaging data and the actually collected pictures, the Adam optimization algorithm is used to update the network parameters, including updating the network parameters of the feature volume and the MLP.

[0065] Step S5: Output the real part three-dimensional distribution and the imaginary part three-dimensional distribution of the current refractive index distribution to obtain the three-dimensional complex refractive index distribution of the sample.

[0066] Understandably, determining the three-dimensional complex refractive index distribution of the sample actually means determining the three-dimensional structure of the sample, where the value of each pixel represents the refractive index.

[0067] The present invention belongs to a self-supervised training method. When performing three-dimensional reconstruction for each sample, the above steps S1 to S5 need to be executed. During self-supervised training, only the image acquisition of the sample to be reconstructed is involved. By minimizing the composite loss function between the reconstructed two-dimensional intensity map and the actually collected two-dimensional intensity map, the update of the feature volume and the multi-layer perception network is realized, and then the reconstruction of the three-dimensional refractive index distribution and system aberration of the sample is realized to obtain the three-dimensional complex refractive index distribution of the sample. Since this self-supervised training method only needs to collect the images of the target sample as loss labels, the data acquisition work is simple and does not require a large number of training samples. For different samples, it is very easy to realize the three-dimensional reconstruction of the sample using this method.

[0068] The input of the traditional implicit neural radiance representation (INR) network is spatial coordinates, and different frequency grid representations are introduced through positional encoding. However, it has an obvious frequency bias itself, with a higher proportion of low frequencies in the network, and the network tends to reconstruct low-frequency information first. In the present invention, the grid representation in the INR is replaced by an updatable feature volume, which has pixel-level frequency representation. Compared with the traditional INR network, the present invention introduces an updatable feature volume, so that the network has no frequency bias, can greatly improve the fitting of high-frequency information, can improve the high-frequency fitting efficiency of the network, combines interpolation upsampling to retain the continuity of the network, and then improves the training efficiency of the network and quickly realizes the three-dimensional reconstruction of the sample.

[0069] Moreover, the present invention sets two branches. On the one hand, it combines a two-dimensional feature volume and an MLP to implement the fitting process of the system pupil function and embeds it into the forward imaging model. Since the phase part of the pupil function can represent the optical aberration of the system, this solution can decouple the system aberration and eliminate its influence on the reconstruction result. On the other hand, it combines a three-dimensional feature volume and an MLP to implement the reconstruction of the real and imaginary parts of the refractive index. The real and imaginary parts correspond to the scattering and absorption abilities of the sample to light. Therefore, the present invention can be compatible with the reconstruction of transmissive and absorptive samples.

[0070] The following verifies the effect of this solution through experiments. Specifically, an iterative algorithm with "spatial-frequency domain" constraints is used as the traditional algorithm for comparison.

[0071] As Figure 3 shown is the comparison diagram of the reconstruction results of this solution and the traditional algorithm when performing three-dimensional reconstruction on a simulated sample. Among them, the two pictures in (a) respectively represent the two-dimensional images collected by the Fourier ptychography system at different illumination angles; the simulated aberration in (b) is the optical aberration added during the imaging of the Fourier ptychography system, and the reconstructed aberration is the optical aberration (phase two-dimensional distribution) output by the network of this solution; (c) is the view of the three-dimensional structure reconstructed by this solution on the X-Y plane and the Y-Z plane, as well as the view of the three-dimensional structure reconstructed by the traditional method on the X-Y plane and the Y-Z plane; (d) is the real three-dimensional structure of the simulated sample and its view on the X-Y plane and the Y-Z plane. Through Figure 3 it can be seen that the reconstruction algorithm proposed by this solution can reconstruct the sample details more clearly than the traditional algorithm. At the same time, from the Y-Z view, it can be seen that the reconstruction algorithm of this solution has fewer reconstruction artifacts and is closer to the true value.

[0072] Figure 4The figure shows the result comparison diagram between the proposed scheme and traditional algorithms when performing three-dimensional reconstruction on Caenorhabditis elegans. It can be seen from the figure that compared with the traditional reconstruction algorithm, the proposed algorithm has more reconstruction details and fewer artifacts under multiple view numbers (the number of LED illuminations). At the same time, as the number of views decreases, the reconstruction quality robustness of the proposed algorithm is higher.

[0073] Figure 5 The figure shows the reconstruction result diagrams of absorption samples (blood smear) and transmission samples (Caenorhabditis elegans) using the proposed scheme. It can be seen from the figure that by reconstructing the complex refractive index of the sample, the proposed algorithm can achieve high reconstruction quality for both absorption and transmission samples.

[0074] Example 2

[0075] The present invention also relates to a three-dimensional reconstruction system based on Fourier ptychography, which includes:

[0076] A feature volume construction unit for respectively constructing a two-dimensional explicit feature volume and a three-dimensional explicit feature volume with feature sizes smaller than the target feature size and initializing the feature values of each pixel point in the feature volume. The target feature size is the size of the pixel map to be reconstructed, where two-dimensional corresponds to width and height, and three-dimensional corresponds to width, height, and depth;

[0077] A feature extraction unit for using a first multi-layer perceptron network to extract features from the two-dimensional explicit feature volume to obtain the two-dimensional phase distribution of the system optical aberration, and using a second multi-layer perceptron network to extract features from the three-dimensional explicit feature volume to obtain the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the sample refractive index distribution. Among them, an interpolation upsampling operation is performed between adjacent two layers in each multi-layer perceptron network to restore the feature size of the extracted features to the target feature size;

[0078] A two-dimensional reconstruction unit for inputting the two-dimensional phase distribution, the three-dimensional real part distribution, and the three-dimensional imaginary part distribution into a forward imaging model to reconstruct the two-dimensional intensity maps of the sample at different illumination angles;

[0079] An adjustment unit for calculating the difference loss between the two-dimensional intensity map of the Fourier ptychography system collected at the same illumination angle and the reconstructed two-dimensional intensity map, and reversely adjusting the feature values of each pixel point in the two-dimensional explicit feature volume and the three-dimensional explicit feature volume as well as the network parameters of each multi-layer perceptron network based on the difference loss;

[0080] An output unit for outputting the three-dimensional real part distribution and the three-dimensional imaginary part distribution of the current refractive index distribution when the obtained difference loss converges, and obtaining the three-dimensional complex refractive index distribution of the sample.

[0081] Understandably, the above system can be used to implement the reconstruction method in Embodiment 1, where each unit can be used to implement the corresponding steps in the reconstruction method. For details, reference can be made to the above introduction and will not be elaborated here.

[0082] Embodiment 3

[0083] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0084] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0085] Embodiment 4

[0086] The embodiments of the present invention provide a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of the method in the above embodiments of the present invention.

[0087] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification. It should be noted that "in one embodiment", "for example", "again, for example", etc. of the present invention are intended to illustrate the present invention and are not used to limit the present invention.

[0088] The above embodiments only represent several implementation manners of the present invention, and the description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the patent application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all belong to the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method based on Fourier stack imaging, characterized in that: include: Step S1, respectively constructing a two-dimensional explicit feature body and a three-dimensional explicit feature body whose feature size is smaller than a target feature size and initializing the value of each pixel feature of the feature body, wherein the target feature size is the size of the pixel image to be reconstructed, the two-dimensional corresponds to width and height, and the three-dimensional corresponds to width, height and depth; Step S2, using a first multi-layer perception network to perform feature extraction on the two-dimensional explicit feature body to obtain a two-dimensional phase distribution of the system optical aberration, and using a second multi-layer perception network to perform feature extraction on the three-dimensional explicit feature body to obtain a real three-dimensional distribution and an imaginary three-dimensional distribution of the sample refractive index distribution; wherein an interpolation upsampling operation is performed between two adjacent layers of the network in each multi-layer perception network to restore the feature size of the extracted feature to the target feature size; Step S3, inputting the two-dimensional phase distribution, the three-dimensional real part distribution and the three-dimensional imaginary part distribution into a forward imaging model to reconstruct a two-dimensional light intensity map of the sample at different illumination angles; Step S4, calculating the difference loss between the two-dimensional light intensity map of the Fourier stack imaging system collected at the same illumination angle and the reconstructed two-dimensional light intensity map, and reversely adjusting the values ​​of the features of each pixel point of the two-dimensional explicit feature body and the three-dimensional explicit feature body and the network parameters of each multi-layer perception network based on the difference loss; looping through steps S2 to S4 until the obtained difference loss converges and the loop ends; Step S5, outputting the real three-dimensional distribution and the imaginary three-dimensional distribution of the current refractive index distribution to obtain the three-dimensional complex refractive index distribution of the sample.

2. The three-dimensional reconstruction method based on Fourier stack imaging according to claim 1, characterized in that: The characteristic sizes of the two-dimensional explicit feature body and the three-dimensional explicit feature body are 0.5 to 0.7 times of the characteristic size of system imaging.

3. The three-dimensional reconstruction method based on Fourier stack imaging according to claim 1, characterized in that: Each pixel feature in the two-dimensional explicit feature volume and the three-dimensional explicit feature volume is an N-dimensional tensor, 5≤N≤8.

4. The three-dimensional reconstruction method based on Fourier stack imaging according to claim 1, characterized in that: The first multi-layer perception network and the second multi-layer perception network both have M layers. The interpolation upsampling operation includes performing an upsampling operation on the features output by the M-1th layer, and then inputting the upsampling results restored to the target feature size into the last layer. The output channel of the last layer of the first multi-layer perception network is 1, and the two-dimensional distribution of phase is output. The output channel of the last layer of the second multi-layer perception network is 2, and the real three-dimensional distribution and the imaginary three-dimensional distribution of the sample refractive index distribution are output respectively; 4≤M≤7.

5. The three-dimensional reconstruction method based on Fourier stack imaging according to claim 4, characterized in that: In each multi-layer perceptron network, except for the last layer, the number of output channels of other layers is 16.

6. The three-dimensional reconstruction method based on Fourier stack imaging according to claim 1, characterized in that: The difference loss includes two parts, namely, the loss of the image and the loss of the high-frequency part of the image after wavelet transformation.

7. The three-dimensional reconstruction method based on Fourier stack imaging according to claim 6, characterized in that: The loss is L2 loss.

8. A three-dimensional reconstruction system based on Fourier stack imaging, characterized in that: include: A feature volume construction unit is used to respectively construct a two-dimensional explicit feature volume and a three-dimensional explicit feature volume whose feature size is smaller than a target feature size and initialize the value of each pixel feature of the feature volume, wherein the target feature size is the size of the pixel image to be reconstructed, the two-dimensional corresponds to width and height, and the three-dimensional corresponds to width, height and depth; A feature extraction unit is used to extract features from the two-dimensional explicit feature body using a first multi-layer perception network to obtain a two-dimensional phase distribution of the system optical aberration, and to extract features from the three-dimensional explicit feature body using a second multi-layer perception network to obtain a three-dimensional distribution of the real part and a three-dimensional distribution of the imaginary part of the sample refractive index distribution; wherein an interpolation upsampling operation is performed between two adjacent layers of the network in each multi-layer perception network to restore the feature size of the extracted feature to the target feature size; A two-dimensional reconstruction unit, used for inputting the two-dimensional phase distribution, the three-dimensional real part distribution and the three-dimensional imaginary part distribution into a forward imaging model to reconstruct a two-dimensional light intensity map of the sample at different illumination angles; An adjustment unit, used to calculate the difference loss between the two-dimensional light intensity map of the Fourier stack imaging system collected at the same illumination angle and the reconstructed two-dimensional light intensity map, and reversely adjust the values ​​of the features of each pixel point of the two-dimensional explicit feature body and the three-dimensional explicit feature body and the network parameters of each multi-layer perception network based on the difference loss; The output unit is used to output the real three-dimensional distribution and the imaginary three-dimensional distribution of the current refractive index distribution when the obtained difference loss converges, so as to obtain the three-dimensional complex refractive index distribution of the sample.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images