A method and device for rapid scanning three-dimensional imaging of large-volume scattering samples

Abstract

The present application proposes a method and device for rapid scanning of large-volume scattering samples in three-dimensional imaging, wherein the method includes: imaging the sample, obtaining light field images of the sample in different directions, rearranging the light field images to obtain sub-aperture images at different angles, obtaining the point spread function of the light field imaging system by computational simulation, and performing three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm based on the point spread function and the sub-aperture image. By adopting the above method, the present invention can quickly perform in-situ rapid three-dimensional imaging of large-volume samples, obtain the three-dimensional refractive index distribution of the sample by multi-angle aberration estimation, and finally reconstruct a three-dimensional reconstruction result with phase difference removed.

Description

Technical Field

[0001] The present invention relates to the technical field of optical imaging systems, and in particular to a method and device for rapid scanning three-dimensional imaging of a large-volume scattering sample. Background Art

[0002] Light field imaging is a fast 3D imaging method that records 4D light field information by adding a microlens array in the optical path. Since the light field records both spatial and angular information, the 3D information of the scene can be obtained by processing a single or multiple light field images.

[0003] Computed tomography is a three-dimensional imaging method that uses detectors to image the imaging object from multiple angles to obtain spatial and angular information, and then performs back-projection reconstruction through algorithms to obtain high-quality three-dimensional information of the detected object.

[0004] In the process of multi-angle sampling, multiple lenses are often required, which makes the system complex, or the object needs to be rotated, which makes it impossible to perform real-time imaging in situ. In macroscopic imaging scenarios, optical components such as lenses will introduce aberrations; in microscopic imaging, the scattering and refraction of large samples will cause non-negligible aberrations, and the aberration space at different positions in the sample is inconsistent. These aberrations will lead to inaccurate reconstruction results. In practical applications, it is necessary to extract aberration information and correct it during the reconstruction process. (In addition, since the point spread function actually collected is different from the simulation, it is generally necessary to calibrate the point spread function of the system, making the reconstruction process more complicated.) Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] In order to solve the deficiencies in the prior art, the purpose of the present invention is to combine computed tomography and light field imaging methods to perform in-situ rapid multi-angle scanning imaging of large-volume fluorescent samples, and perform phase difference correction and accurate three-dimensional reconstruction.

[0007] To this end, the first purpose of the present invention is to propose a method for rapid scanning three-dimensional imaging of large-volume scattering samples, which combines computed tomography and light field imaging methods to perform in-situ rapid multi-angle scanning imaging of large-volume fluorescent samples, and perform phase difference correction and accurate three-dimensional reconstruction.

[0008] The second object of the present invention is to provide a device for rapid scanning three-dimensional imaging of large-volume scattering samples.

[0009] To achieve the above-mentioned purpose, a first embodiment of the present invention provides a method for rapid scanning three-dimensional imaging of a large volume scattering sample, comprising:

[0010] S1, imaging the sample to obtain light field images of the sample in different directions;

[0011] S2, rearranging the light field image to obtain sub-aperture images at different angles;

[0012] S3, obtaining the point spread function of the light field imaging system through computational simulation;

[0013] S4, registering the image pair and calculating the coordinate transformation relationship of the image pair;

[0014] S5, performing three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm according to the point spread function and based on the sub-aperture image.

[0015] In addition, the large volume scattering sample fast scanning three-dimensional imaging method according to the above embodiment of the present invention may also have the following additional technical features:

[0016] Furthermore, in one embodiment of the present invention, imaging the sample to obtain light field images of the sample in different directions includes:

[0017] The sample is imaged from different angles, with multiple angles imaged at intervals of a certain angle each time it is rotated, and imaging is performed after each rotation is completed.

[0018] Furthermore, in one embodiment of the present invention, obtaining the point spread function of the light field imaging system by computational simulation includes:

[0019] The forward propagation process of the optical path is simulated, the complex light field is calculated, and the light intensity distribution is obtained through phase modulation through the forward propagation process of the optical path to obtain the sub-aperture point spread function corresponding to each depth.

[0020] Furthermore, in one embodiment of the present invention, the phase matrix and the three-dimensional reconstruction result are alternately optimized based on the alternating direction multiplier method.

[0021] Furthermore, in one embodiment of the present invention, the phase matrix and the intensity matrix are initialized to all zeros.

[0022] Furthermore, in one embodiment of the present invention, the intensity matrix and the phase matrix are optimized, wherein:

[0023] A rotation imaging angle is selected, and a phase correction matrix is ​​obtained by order accumulation. The phase correction matrix forward projects the intensity matrix, and is combined with the phase correction matrix for correction to obtain a forward projection result. An error matrix between the forward projection result and the true value of the current sub-aperture is calculated to obtain a calculation result. Based on the calculation result, a gradient descent algorithm is executed on the intensity matrix and the phase matrix respectively for optimization.

[0024] Furthermore, in one embodiment of the present invention, the optimizing the intensity matrix further includes:

[0025] The error matrix is ​​projected onto a volume in conjunction with the phase correction matrix for back-projection to optimize the intensity matrix.

[0026] Furthermore, in one embodiment of the present invention, the intensity matrix and the phase matrix are optimized using sub-apertures of all rotation angles sequentially or randomly.

[0027] Furthermore, in one embodiment of the present invention, the intensity matrix and the phase matrix are repeatedly optimized until they converge, and the intensity matrix is ​​the three-dimensional reconstruction result after phase correction.

[0028] The large-volume scattering sample fast scanning three-dimensional imaging method of the embodiment of the present invention obtains light field images of the sample in different directions by imaging the sample, rearranges the light field images to obtain sub-aperture images at different angles, obtains the point spread function of the light field imaging system by computational simulation, and performs three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm based on the point spread function and the sub-aperture images. By adopting the above method, the present invention can quickly perform in-situ fast three-dimensional imaging of large-volume samples, obtain the three-dimensional refractive index distribution of the sample by multi-angle aberration estimation, and finally reconstruct a three-dimensional reconstruction result with phase difference removed.

[0029] To achieve the above-mentioned purpose, a second embodiment of the present invention provides a large-volume scattering sample fast scanning three-dimensional imaging device, comprising:

[0030] An imaging module, used to image the sample and obtain light field images of the sample in different directions;

[0031] A rearrangement module, used for rearranging the light field image to obtain sub-aperture images at different angles;

[0032] A calculation module, used for obtaining a point spread function of a light field imaging system through calculation simulation;

[0033] A reconstruction module is used to perform three-dimensional reconstruction according to the point spread function and based on the sub-aperture image using a phase difference estimation synchronous reconstruction algorithm.

[0034] The large-volume scattering sample fast scanning three-dimensional imaging device of the embodiment of the present invention obtains light field images of the sample in different directions by imaging the sample, rearranges the light field images to obtain sub-aperture images at different angles, obtains the point spread function of the light field imaging system by computational simulation, and performs three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm based on the point spread function and the sub-aperture images. By adopting the above method, the present invention can quickly perform in-situ fast three-dimensional imaging of large-volume samples, obtain the three-dimensional refractive index distribution of the sample by multi-angle aberration estimation, and finally reconstruct a three-dimensional reconstruction result with phase difference removed.

[0035] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A flowchart of a method for fast scanning three-dimensional imaging of a large volume scattering sample according to an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the structure of an example of a rotating light field imaging system according to an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of an optical path of a light field imaging system according to an embodiment of the present invention;

[0040] Figure 4 is a schematic diagram of a single light field rearrangement according to an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of a light field diagram and a rearranged sub-aperture image according to an embodiment of the present invention;

[0042] Figure 6 A flow chart of a phase difference estimation and synchronous reconstruction algorithm according to an embodiment of the present invention;

[0043] Figure 7 Schematic diagram of the structure of a device for rapid scanning of a large volume scattering sample in three-dimensional imaging according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0045] The following describes a method and apparatus for rapid scanning three-dimensional imaging of a large volume scattering sample according to an embodiment of the present invention with reference to the accompanying drawings.

[0046] The large-volume scattered sample fast scanning three-dimensional imaging method of the embodiment of the present application provides a large-volume scattered sample fast scanning three-dimensional imaging method, through a self-designed rotating mirror group, a single objective lens is used to quickly collect fixed samples at multiple angles, and for each angle collected by the objective lens, the angle and position information of the sample fluorescence signal are recorded simultaneously using the light field. Afterwards, the light field data of multiple angles are rearranged to obtain sub-aperture images, and the aberration of the sample at each angle is estimated based on the intensity and position information of multiple angles through the convex optimization algorithm ADMM (augmented Lagrangian penalty method), thereby obtaining the sample refractive index three-dimensional distribution. In the reconstruction process, the refractive index distribution matrix is ​​used to correct the three-dimensional intensity distribution of the fluorescent sample, thereby obtaining a high-precision three-dimensional reconstruction result.

[0047] Figure 1 The present invention provides a flowchart of a method for rapid scanning three-dimensional imaging of a large volume scattering sample according to an embodiment of the present invention.

[0048] like Figure 1 As shown, the large volume scattering sample rapid scanning three-dimensional imaging method comprises:

[0049] Step S1, imaging the sample to obtain light field images of the sample in different directions.

[0050] Specifically, the sample is imaged by a light field rotation imaging system to obtain light field images in different directions. Figure 2 FIG. 4 is a schematic structural diagram of an example of a rotating light field imaging system according to an embodiment of the present invention.

[0051] 1) First, embed the sample into a transparent container with a smooth surface and fix it in front of the rotating mirror group. Figure 2 As shown, during the imaging process, the rotating stage drives the mirror group to move, so that the objective lens can image the sample from different angles, each time rotating at a certain angle interval to provide multi-angle imaging.

[0052] 2) The acquisition module adopts a light field imaging system and performs imaging after each rotation is completed. Figure 3In the example, the point light source is a point source, L1 and L2 are lenses, a microlens array ML is placed on the back focal plane of L2, and an image sensor is placed on the sensor plane of the back focal plane of the microlens to collect the light field. In the figure, f1 is the focal length of L1, f2 is the focal length of L2, and fml is the focal length of ML. The collected information is 4D information, including spatial information (2 dimensions) and angle information (2 dimensions). For example, in a microscopic scene, the light source is a fluorescent sample, and the excitation light path is not constrained. L1 and L2 are the objective lens and the tube lens, respectively.

[0053] Step S2, rearrange the light field image to obtain sub-aperture images at different angles.

[0054] It can be understood that the light field image is rearranged to obtain the sub-aperture image, Figure 4 This is a schematic diagram of the rearrangement of a single light field. The specific image size and rearrangement order depend on the actual situation. Figure 3 For simplicity, each microlens corresponds to 3*3 sensor pixels. Figure 4 (a) is an example of a light field diagram. Figure 4 (b) and (c) are obtained by taking the corresponding positions in the light field image and stitching them together to obtain the images of each sub-aperture. The figure shows two of the sub-aperture images. In this example, a total of 3*3 sub-aperture images can be obtained. For this light field image, it should be interpolated to the original light field image size of 9*9. For the light field image obtained by scanning the light field, it also needs to be rearranged. The basic process remains unchanged, and the rearrangement order needs to be determined according to the actual situation. Figure 5 is an example of a light field image and a rearranged sub-aperture image. Figure 5 As shown in FIG. 1 , the sub-aperture image obtained from the light field image contains information at different angles of the object. Figure 5 In the example, there are 13*13 sub-apertures, corresponding to 13*13 sub-aperture images. Define the sub-aperture image corresponding to the sub-aperture (u, v) as I(u, v). Figure 4 (a) is the original light field image, Figure 4 (b) is an example of sub-aperture images corresponding to four sub-apertures, and the corresponding apertures are (2,9), (3,9), (7,7), and (1,1).

[0055] Step S3, obtaining the point spread function of the light field imaging system through calculation simulation.

[0056] It is understandable that before reconstruction, the point spread function (PSF) of the light field imaging system needs to be obtained first. Specifically, the forward propagation process of the light path is simulated by computer, and the simulated sub-aperture point spread function PSF is calculated. For example Figure 3Schematic diagram of the medium light field system. Starting from a point source, the forward propagation process of the light path is simulated by a computer to calculate the complex light field on the front surface of the microlens array ML. Through the phase modulation of the microlens array, after a propagation process, the light intensity distribution of the point source on the sensor plane can be obtained. From this, the sub-aperture point spread function corresponding to each depth can be calculated. Figure 3 Where L1 and L2 are lenses, f1 is the focal length of L1, f2 is the focal length of L2, and fml is the focal length of the microlens.

[0057] Step S4, performing three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm according to the point spread function and based on the sub-aperture image.

[0058] It can be understood that after preprocessing the light field image and obtaining the point spread function (PSF) of the light field imaging system, the phase difference estimation synchronous reconstruction algorithm is used to simultaneously estimate the phase matrix from multiple angles and eliminate the phase difference multi-angle 3D reconstruction. Under the continuity assumption, this problem is a convex problem, so the present invention alternately optimizes the phase matrix and the 3D reconstruction result based on the alternating direction multiplier method (ADMM). Figure 6 The flowchart of the phase difference estimation synchronous reconstruction algorithm according to an embodiment of the present invention is as shown in the figure:

[0059] c. For the range of 3D reconstruction, the phase matrix and intensity matrix are initialized to all zeros.

[0060] d. To optimize the intensity matrix, first select a rotation imaging angle, first accumulate the phase matrix in order according to this direction, and obtain the phase difference correction matrix in this direction. Select a sub-aperture image at this angle, which forward projects the intensity matrix through the point spread function (PSF) of the light field imaging system, and corrects it in combination with the phase difference correction matrix in this direction to obtain the forward projection result. Calculate the error matrix between the forward projection result and the true value of the current sub-aperture, and perform a gradient descent algorithm on the intensity matrix based on it for optimization. (And perform back projection in combination with the phase difference correction matrix in this direction, project the error matrix onto the volume, and optimize the intensity matrix.) Then use the sub-apertures of all rotation angles sequentially or randomly to optimize the intensity matrix.

[0061] e. To optimize the phase matrix, first select a rotation imaging angle, first accumulate the phase matrix in order according to this direction, and obtain the phase difference correction matrix in this direction. Select a sub-aperture image at this angle, which forward projects the intensity matrix through the point spread function (PSF) of the light field imaging system, and corrects it in combination with the phase difference correction matrix in this direction to obtain the forward projection result. Calculate the error matrix between the forward projection result and the true value of the current sub-aperture, and perform the gradient descent algorithm on the phase matrix based on it for optimization. Then, use the sub-apertures of all rotation angles sequentially or randomly to optimize the phase matrix.

[0062] f. Repeat steps d and e until they converge, and the intensity matrix is ​​the phase-corrected 3D reconstruction result.

[0063] According to the method for fast scanning three-dimensional imaging of a large volume scattering sample provided by an embodiment of the present invention, by imaging the sample, light field images of the sample in different directions are obtained, the light field images are rearranged to obtain sub-aperture images at different angles, the point spread function of the light field imaging system is obtained by computational simulation, and three-dimensional reconstruction is performed using a phase difference estimation synchronous reconstruction algorithm based on the point spread function and the sub-aperture image. By adopting the above method, the present invention can quickly perform in-situ fast three-dimensional imaging on a large volume sample, obtain the three-dimensional refractive index distribution of the sample by multi-angle aberration estimation, and finally reconstruct a three-dimensional reconstruction result with phase difference removed.

[0064] Figure 7 It is a schematic structural diagram of a device for rapid scanning of a large volume scattering sample in three-dimensional imaging according to an embodiment of the present invention.

[0065] like Figure 7 As shown, the large volume scattering sample fast scanning three-dimensional imaging device 10 comprises:

[0066] An imaging module 100 , a rearrangement module 200 , a calculation module 300 and a reconstruction module 400 .

[0067] An imaging module 100 is used to image the sample and obtain light field images of the sample in different directions;

[0068] A rearrangement module 200, used to rearrange the light field image to obtain sub-aperture images at different angles;

[0069] A calculation module 300 is used to obtain a point spread function of a light field imaging system through calculation simulation;

[0070] The reconstruction module 400 is used to perform three-dimensional reconstruction according to the point spread function and based on the sub-aperture image using a phase difference estimation synchronous reconstruction algorithm.

[0071] According to the large-volume scattering sample fast scanning three-dimensional imaging device proposed in the embodiment of the present invention, by imaging the sample, the light field image of the sample in different directions is obtained, the light field image is rearranged to obtain sub-aperture images at different angles, the point spread function of the light field imaging system is obtained by computational simulation, and three-dimensional reconstruction is performed using a phase difference estimation synchronous reconstruction algorithm based on the point spread function and the sub-aperture image. By adopting the above method, the present invention can quickly perform in-situ fast three-dimensional imaging on a large-volume sample, and the three-dimensional refractive index distribution of the sample can be obtained by multi-angle aberration estimation, and finally reconstruct a three-dimensional reconstruction result with phase difference removed.

[0072] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0073] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0074] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for rapid scanning three-dimensional imaging of a large volume scattering sample, characterized in that: include: Imaging the sample to obtain light field images of the sample in different directions; Rearranging the light field image to obtain sub-aperture images at different angles; The point spread function of the light field imaging system is obtained through computational simulation; According to the point spread function and based on the sub-aperture image, a phase difference estimation synchronous reconstruction algorithm is used to perform three-dimensional reconstruction; Based on the alternating direction multiplication method, the phase matrix and the three-dimensional reconstruction results are alternately optimized; Performing all-zero initialization on the phase matrix and the intensity matrix; The intensity matrix and the phase matrix are optimized, wherein: A rotation imaging angle is selected, and a phase correction matrix is ​​obtained by order accumulation. The phase correction matrix forward projects the intensity matrix, and is combined with the phase correction matrix for correction to obtain a forward projection result. An error matrix between the forward projection result and the true value of the current sub-aperture is calculated to obtain a calculation result. Based on the calculation result, a gradient descent algorithm is executed on the intensity matrix and the phase matrix respectively for optimization.

2. The method for rapid scanning three-dimensional imaging of a large volume scattering sample according to claim 1, characterized in that: The imaging of the sample to obtain light field images of the sample in different directions includes: The sample is imaged from different angles, with multiple angles imaged at intervals of a certain angle each time it is rotated, and imaging is performed after each rotation is completed.

3. The method for rapid scanning three-dimensional imaging of a large volume scattering sample according to claim 1, characterized in that: The point spread function of the light field imaging system is obtained by computational simulation, including: The forward propagation process of the optical path is simulated, the complex light field is calculated, and the light intensity distribution is obtained through phase modulation through the forward propagation process of the optical path to obtain the sub-aperture point spread function corresponding to each depth.

4. The method for rapid scanning three-dimensional imaging of a large volume scattering sample according to claim 1, characterized in that: The optimization of the intensity matrix further includes: The error matrix is ​​projected onto a volume in conjunction with the phase correction matrix for back-projection to optimize the intensity matrix.

5. The method for rapid scanning three-dimensional imaging of a large volume scattering sample according to claim 4, characterized in that: The intensity matrix and the phase matrix are optimized using subapertures of all rotation angles sequentially or out of order.

6. The method for rapid scanning three-dimensional imaging of a large volume scattering sample according to claim 5, characterized in that: The intensity matrix and the phase matrix are optimized repeatedly until they converge, and the intensity matrix is ​​the three-dimensional reconstruction result after phase correction.

7. A device for rapid scanning three-dimensional imaging of a large volume scattering sample, comprising: An imaging module, used to image the sample and obtain light field images of the sample in different directions; A rearrangement module, used for rearranging the light field image to obtain sub-aperture images at different angles; A calculation module, used for obtaining a point spread function of a light field imaging system through calculation simulation; A reconstruction module, configured to perform three-dimensional reconstruction using a phase difference estimation synchronous reconstruction algorithm according to the point spread function and based on the sub-aperture image; The reconstruction module is further used for: Based on the alternating direction multiplication method, the phase matrix and the three-dimensional reconstruction results are alternately optimized; Performing all-zero initialization on the phase matrix and the intensity matrix; The intensity matrix and the phase matrix are optimized, wherein: A rotation imaging angle is selected, and a phase correction matrix is ​​obtained by order accumulation. The phase correction matrix forward projects the intensity matrix, and is combined with the phase correction matrix for correction to obtain a forward projection result. An error matrix between the forward projection result and the true value of the current sub-aperture is calculated to obtain a calculation result. Based on the calculation result, a gradient descent algorithm is executed on the intensity matrix and the phase matrix respectively for optimization.

Images