A three-dimensional microscopic imaging method based on multi-angle projection

Abstract

The present application relates to the technical field of three-dimensional microscopic imaging, and discloses a three-dimensional microscopic imaging method based on multi-angle projection, comprising the following steps: S1: modulating incident light by using a spatial light modulator, so that the light emitted from the spatial light modulator generates a V-shaped light beam at the focal point of an objective lens, and the sample is projected in at least two angle directions, thereby obtaining a two-dimensional projection image; S2: performing three-dimensional reconstruction on the two-dimensional projection image by using a three-dimensional reconstruction algorithm, thereby forming a three-dimensional reconstruction image; and S3: fusing the three-dimensional reconstruction images of different angles, thereby generating a final fused three-dimensional image. In the present application, the V-shaped light beam modulated by the spatial light modulator is projected on the sample in at least two angle directions, without the need to rotate the sample or to perform 180° / 360° projection imaging on the sample, thereby reducing the difficulty of sample processing and placement, shortening the scanning time of sample imaging, and reducing the influence of light bleaching / light toxicity on the sample.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional microscopic imaging, and in particular to a three-dimensional microscopic imaging method based on multi-angle projection. BACKGROUND

[0002] Common three-dimensional microscopic imaging methods can be divided into spatial point scanning three-dimensional imaging and projection reconstruction three-dimensional imaging. In the imaging method based on spatial point scanning in the prior art, the spot position of the objective lens in the microscopic system is changed by using a scanning galvanometer to perform point scanning on the target sample, and the data obtained after scanning can be directly imaged. Defects: due to the size limitation of the spot, the time required for three-dimensional scanning is proportional to the cube of the sample size, so the scanning time required for a large sample is very long, and at the same time, the long scanning time limits the time resolution of imaging. In addition, long scanning time also increases the risk of light bleaching / light toxicity of the sample.

[0003] In the optical projection tomography (OPT) method based on computed tomography (CT) algorithm in the prior art, the three-dimensional information of the sample is obtained by using the two-dimensional projection of the sample for filtered back-projection (FBP) reconstruction. OPT can use a point array detector such as a CCD, so the imaging speed is faster than that of the point scanning method. Defects: since the FBP algorithm needs to scan and project the sample at 180° / 360°, the sample needs to be embedded in a medium such as agrose gel and needs to be fixed on a rotating shaft for projection at different angles, so the preparation process of the sample is relatively complex. In addition, if fast imaging is required, the sample needs to be rotated quickly, and the fast rotation will cause spatial deviation, affecting the reconstruction effect. Finally, rotation may also interfere with the sample of the living body, affecting the physiological activity of the sample itself.

[0004] In summary, the spatial point scanning three-dimensional imaging needs a long imaging time, and the scanning time is proportional to the cube of the sample size. The projection reconstruction three-dimensional imaging needs to fix the sample on a rotating shaft for 180° / 360° rotation, so the sample needs to be embedded or fixed for processing. SUMMARY

[0005] The present application aims to provide a three-dimensional microscopic imaging method based on multi-angle projection, which aims to solve the problem of long three-dimensional imaging time or the need to rotate the sample in the prior art.

[0006] The present application is implemented as follows: a three-dimensional microscopic imaging method based on multi-angle projection, comprising the following steps:

[0007] S1: modulating the incident light by a spatial light modulator, so that the light emitted from the spatial light modulator generates a V-shaped light beam at the focal point of the objective lens, and the sample is projected in at least two angle directions, thereby obtaining a two-dimensional projection image;

[0008] S2: performing three-dimensional reconstruction on the two-dimensional projection image by using a three-dimensional reconstruction algorithm to form a three-dimensional reconstruction image;

[0009] S3: fusing the three-dimensional reconstruction images of different angles to generate a final fused three-dimensional image.

[0010] Optionally, in step S1, the phase modulation of the spatial light modulator causes the phase change of the light field incident to the back pupil of the objective lens, thereby generating a point spread function with a long axial length, which covers a certain axial area in a single transverse scan.

[0011] Optionally, the V-shaped light beam is realized by controlling the incident light to reach different positions of the objective lens through the spatial light modulator.

[0012] Optionally, the axial length of the point spread function is greater than the transverse length.

[0013] Optionally, the aspect ratio of the point spread function is usually ≥ 10:1.

[0014] Optionally, the point spread function has an asymmetric side lobe in the axial direction, and the side lobe is used for axial encoding in three-dimensional reconstruction.

[0015] Optionally, the V-shaped light beam is projected at angles that are orthogonal to each other.

[0016] Optionally, in step S2, a neural network-based algorithm is used to perform three-dimensional reconstruction on the two-dimensional projection image, and the input two-dimensional projection image is from a projection image at a certain angle, with a size of X*Y, and the output three-dimensional reconstruction image has a size of X*Y*Z, corresponding to the three-dimensional space information of the output image.

[0017] Optionally, in step S3, at least two three-dimensional reconstruction images are mapped in the same coordinate system, and the at least two three-dimensional reconstruction images are fused by supervised learning of a neural network to generate a final fused three-dimensional image.

[0018] Optionally, the mapping manner includes but is not limited to rotation, translation, shearing, and scaling.

[0019] Compared with the prior art, the three-dimensional microscopic imaging method based on multi-angle projection provided by the application modulates the incident light through a spatial light modulator, generates a V-shaped light beam at the focal point of the objective lens, and projects the sample in at least two angle directions, without rotating the sample or performing 180° / 360° projection imaging on the sample, thereby reducing the difficulty of sample processing and placement. The axial projection is performed through a lengthened point spread function, thereby shortening the scanning time of sample imaging and reducing the influence of light bleaching / light toxicity on the sample. The projection images at different angles are fused to realize the isotropy of the reconstructed image resolution, thereby improving the quality of the fused three-dimensional image. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of the three-dimensional microscopic imaging method based on multi-angle projection provided by the application;

[0021] Figure 2 is a light path schematic diagram of the three-dimensional microscopic imaging method based on multi-angle projection provided by the application;

[0022] Figure 3 is an axial schematic diagram of the point spread function of the three-dimensional microscopic imaging method based on multi-angle projection provided by the application;

[0023] Figure 4 is a transverse schematic diagram of the point spread function of the three-dimensional microscopic imaging method based on multi-angle projection provided by the application;

[0024] Figure 5 is an axial schematic diagram of the V-shaped light beam constituted by the point spread function in the three-dimensional microscopic imaging method based on multi-angle projection provided by the application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0026] The implementation of the application is described in detail below in combination with specific examples.

[0027] The same or similar reference numerals in the drawings of the embodiments correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation of the present patent, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0028] Referring to Figures 1-4 The preferred embodiment provided by the present application is shown.

[0029] A three-dimensional microscopic imaging method based on multi-angle projection is disclosed in the present application, which projects a sample using a V-shaped light beam and reconstructs the three-dimensional information of the sample by combining a reconstruction algorithm. Without additional processing of the sample, the three-dimensional reconstruction is performed using the projection of the V-shaped light beam, reducing the projection angle and imaging time.

[0030] Referring to Figures 1-2 A three-dimensional microscopic imaging method based on multi-angle projection, comprising the following steps:

[0031] S1: A spatial light modulator 1 is used to modulate the incident light, so that the light emitted from the spatial light modulator 1 generates a V-shaped light beam 3 at the focal point of an objective lens 2, and the sample 4 is projected in at least two angle directions, and a two-dimensional projection image is obtained correspondingly;

[0032] S2: A three-dimensional reconstruction algorithm is used to perform three-dimensional reconstruction on the two-dimensional projection image to form a three-dimensional reconstruction image;

[0033] S3: The three-dimensional reconstruction images of different angles are fused to generate a final fused three-dimensional image.

[0034] The three-dimensional microscopic imaging method based on multi-angle projection provided by the present embodiment modulates the incident light by a spatial light modulator 1, generates a V-shaped light beam 3 at the focal point of an objective lens 2, and projects a sample 4 in at least two angle directions, without rotating the sample 4, without performing 180° / 360° projection imaging on the sample 4, reducing the difficulty of sample processing and placement, shortening the scanning time of sample imaging, reducing the influence of light bleaching / light toxicity on the sample. By fusing the projection images of different angles, the isotropy of the reconstruction image resolution is realized, and the quality of the fused three-dimensional image is improved.

[0035] Different from the OPT imaging technology based on rotating axis, the embodiment realizes the multi-angle projection of the sample 4 through the spatial light modulator 1 and the objective lens 2. In comparison, generally, the OPT projection needs to use a low numerical aperture objective lens, resulting in the resolution reduction, and the three-dimensional microscopic imaging method based on the multi-angle projection provided by the embodiment can use a high numerical aperture objective lens for projection, thereby avoiding the resolution loss.

[0036] The application does not need to additionally add an objective lens on the three-dimensional microscopic imaging system, and does not need to obtain samples at different angles by rotating the sample, and is fully compatible with the existing microscopic imaging system. Figure 2 In the schematic diagram, the collection device and the optical path of the two-dimensional projection image are not shown, and the sensor such as a photomultiplier tube (PMT) or a CCD can be generally used to collect image information in the existing microscopic imaging system.

[0037] The existing spatial scanning type microscopic imaging method can be compatible, and the spatial light modulator 1 is added to the existing three-dimensional microscopic imaging system to form the multi-angle microscopic imaging system of the application, which is used to generate different angle projections to realize. In the multi-angle microscopic imaging system, the incident light is modulated by the spatial light modulator 1 and then incident into the three-dimensional microscopic imaging system, and a V-shaped light beam 3 is formed at the focal point of the objective lens 2.

[0038] The spatial light modulator 1 contains many independent units, which are arranged in a one-dimensional or two-dimensional array in space, each unit can independently receive optical signal or electrical signal control, and change its optical properties according to the signal, thereby modulating the light wave on which the illumination is modulated. The spatial light modulator 1 can change the amplitude or intensity, phase, polarization state and wavelength of the spatial light distribution under the control of the time-varying electrical driving signal or other signals, or convert incoherent light into coherent light, write certain information into the light wave, and achieve the purpose of light wave modulation. Due to its properties, it can be used as a construction unit or a key device in real-time optical information processing, optical computing and optical neural network systems. In addition, if the phase does not need to be dynamically adjusted during the imaging process, a phase mask or the like can be used to realize the modulation of the light wave.

[0039] Specifically, in step S1, the phase modulation of the spatial light modulator 1 causes the phase change of the light field incident to the back pupil of the objective lens 2, and a point spread function with a long axial length is generated, which covers a certain depth of the axial region in a single transverse scan.

[0040] The axial length of the point spread function used in the embodiment is greater than the transverse length. For example, the point spread function with a long axial length is generated as shown in Figures 3-4As shown in the figure, in the XZ direction, a feature of a relatively long axial length is presented; in the XY direction, a feature of a relatively short transverse length is presented, which is longer in the Z direction than in the X direction.

[0041] The axial length of the incident light beam can be adjusted according to the volume of the sample, a longer axial length can project a larger axial range, reduce the axial scanning time, and thus ensure efficient axial sampling and reduce the time of three-dimensional microscopic imaging.

[0042] The aspect ratio of the point spread function is ≥ 10:1, which is verified by simulation experiments. Such a point spread function with a relatively long axial length can greatly reduce the axial scanning time and significantly improve the efficiency of the overall three-dimensional microscopic imaging.

[0043] For example, as shown in the figure, Figures 3-4 The point spread function used in the present application has an asymmetric side lobe in the axial direction, so it can play an axial coding effect, which is very convenient for three-dimensional reconstruction. Other point spread functions with axial characteristics can also be used in the implementation of the present application. The morphological changes of the point spread function used in the implementation depend on the modulation of spatial light, which can be adjusted according to the needs of the sample or algorithm.

[0044] As shown in the figure, Figure 5 The axial schematic diagram of the V-shaped light beam composed of the point spread function provided by the present application is shown in the figure. Through the projection of two-angle light beams, the point spread function of each angle of light beam is an axial length greater than a transverse length, which can realize isotropic three-dimensional image resolution.

[0045] In the embodiment, the phase modulation of the spatial light modulator 1 causes the phase change of the light field incident to the back pupil of the objective lens, which can produce a point spread function different from the Gaussian spot used in the common microscopic system. The present embodiment adopts a point spread function with a relatively long axial length, which can cover a deeper axial area in a single transverse scan. The corresponding three-dimensional information can be obtained by algorithm reconstruction.

[0046] Specifically, the V-shaped light beam 3 passes through the spatial light modulator 1, and controls the different positions of the incident light to the objective lens 2 to achieve.

[0047] The three-dimensional microscopic imaging method based on multi-angle projection provided in the embodiment is different from the algorithm based on FBP, which does not need to project at a projection angle of 180° / 360°. In the embodiment, only two angles are needed to reconstruct the three-dimensional information of the sample 4.

[0048] Preferably, when only two angles are used for projection to reconstruct the three-dimensional image, two angles orthogonal to each other are used for projection, such as Figure 5As shown, the axial schematic diagram of the V-shaped light beam composed of the point spread function, by two mutually orthogonal angle light beam projection, can realize the isotropy of the resolution of the reconstructed three-dimensional image, and the simulation experiment verifies that the reconstructed image is good.

[0049] In addition, when the spatial light modulator 1 modulates the incident light, a plurality of angle V-shaped light beams 3 can be generated at the focal point of the objective lens 2, not limited to two angles, and more angle projection can improve the quality of the reconstructed image, and of course the calculation amount of three-dimensional reconstruction will increase.

[0050] Multi-angle projection can provide complementary information of the sample from different angles, and has the characteristics of high spatial resolution compared with single angle observation. In the embodiment, the spatial light modulator 1 modulates the multi-angle V-shaped light beam 3 to project on the sample, so that the blocked object can also be observed from different angles at the same time, so as to obtain the complete information of the sample.

[0051] Three-dimensional reconstruction generally includes three parts, first, using a sensor to scan the target to be reconstructed from multiple angles, then extracting and matching the features of the scanned multiple frames of pictures, and finally completing the mapping of two-dimensional pixels to three-dimensional coordinate points through stereo vision technology to obtain the final reconstructed model.

[0052] Specifically, in step S2, a neural network-based algorithm is used to reconstruct three-dimensional images from two-dimensional projection images. The input two-dimensional projection image comes from a projection image at a certain angle, with a size of X*Y, and the output three-dimensional reconstruction image has a size of X*Y*Z, corresponding to the three-dimensional space information of the output image. In the neural network-based algorithm, the input data in the training data can be collected by a multi-angle microscopic imaging system, and the output target image can be collected by a traditional point scanning microscopic imaging system.

[0053] In step S3, after reconstruction, the left and right two reconstructed images are mapped in the same coordinate system. Common transformation mapping includes rotation, translation, shearing, scaling, etc. The specific transformation relationship depends on the relationship between the incident light corresponding coordinate and the target coordinate system.

[0054] Through supervised learning of the neural network, the left and right two images are fused. Similar to step S2, the target of supervised learning can be obtained by a traditional three-dimensional point scanning.

[0055] In the process of finally fusing the multi-angle two-dimensional image to obtain a three-dimensional image, a convolutional neural network can be used for feature extraction and fusion to obtain high-dimensional features. Preferably, in the present application, an artificial neural network is used in the supervised learning process to fuse the multi-angle projection images.

[0056] The artificial neural network has a three-layer structure: an input layer, a hidden layer, and an output layer. In the input layer, input variables of the neural network are inputted; in the hidden layer and the output layer, calculations are performed and outputs are outputted. On the hidden layer of the neural network, there are "neurons" that operate by relying on an Activation function.

[0057] The training method of the artificial neural network is a backpropagation algorithm. The backpropagation algorithm mainly includes two steps: forward calculation to obtain an error function and reverse derivation gradient descent.

[0058] In the training process of the artificial neural network, firstly, a defined error function is minimized. Secondly, the self-variables of the neural network are updated, so that the error function of the neural network is reduced under the updated self-variables; the gradient descent method is adopted; for example, the partial derivative of the error function with respect to the self-variables of the neural network is calculated. The neural network is updated by the gradient descent method, and the principle is similar to the Newton method and the least square method. Then the above steps are repeatedly performed until the error function is reduced to an acceptable range, and the learning of the neural network on the training data is completed.

[0059] In the present application, a deep learning algorithm is introduced into the traditional three-dimensional reconstruction, and the efficiency and image quality of three-dimensional reconstruction are optimized.

[0060] In the simulation experiment, through the simulation data of two-dimensional projection images to three-dimensional images, an end-to-end neural network is used to directly generate three-dimensional information of an object represented by a grid from a single color image, and good results are obtained.

[0061] The three-dimensional microscopic imaging method based on multi-angle projection provided by the present application has the following beneficial effects:

[0062] 1. The three-dimensional microscopic imaging method based on multi-angle projection provided by the present application can shorten the scanning time of sample imaging, does not need to rotate the sample, and reduces the difficulty of sample processing and placement.

[0063] 2. The present application does not need to perform 180° / 360° projection imaging on the sample, and only two angles of projection are used in the embodiment to reconstruct the three-dimensional information of the object, thereby reducing the influence of light bleaching / light toxicity on the sample.

[0064] 3. The present application can be compatible with the existing spatial scanning type microscopic imaging method, and a spatial light modulator 1 is added to the existing system to generate projections at different angles.

[0065] 4. The present application can adjust the size of the projection light according to the size of the sample and the imaging resolution requirement, and optimize the imaging time required.

[0066] 5. The application can realize the isotropy of the reconstructed image resolution by fusing the projection images of different angles.

[0067] The above merely describes the preferred embodiments of the present application, but should not be used to limit the present application, and any modification, equivalent replacement, and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of three-dimensional microscopic imaging based on multi-angle projection, characterized in that, The method comprises the following steps: S1: modulating incident light by using a spatial light modulator, so that the light emitted from the spatial light modulator generates a V-shaped light beam at the focal point of an objective lens, and the sample is projected in at least two angular directions, thereby obtaining a two-dimensional projection image; S2: performing three-dimensional reconstruction on the two-dimensional projection image by using a three-dimensional reconstruction algorithm to form a three-dimensional reconstruction image; S3: fusing the three-dimensional reconstruction images at different angles to generate a final fused three-dimensional image; In step S1, the phase modulation of the spatial light modulator causes the phase change of the light field incident to the back pupil of the objective lens, thereby generating a point spread function with a long axial length, which covers an axial area of a certain depth in a single transverse scan; The V-shaped light beam passes through the spatial light modulator, and different positions of the incident light reaching the objective lens are controlled to achieve the projection; The axial length of the point spread function is greater than the transverse length; The aspect ratio of the point spread function is ≥ 10:

1.

2. A method for three-dimensional microscopic imaging based on multi-angle projection as claimed in claim 1, characterized in that, The point spread function has an asymmetric side lobe in the axial direction, and the side lobe is used for axial encoding in three-dimensional reconstruction.

3. A multi-angle projection based three-dimensional microscopic imaging method according to any one of claims 1-2, characterized in that, The V-shaped light beam is projected at angles that are orthogonal to each other.

4. A multi-angle projection based three-dimensional microscopic imaging method according to any one of claims 1-2, characterized in that, In step S2, a neural network-based algorithm is used to perform three-dimensional reconstruction on the two-dimensional projection image, the input two-dimensional projection image is from a projection image at a certain angle, and the size is X*Y; the output three-dimensional reconstruction image has a size of X*Y*Z, and the three-dimensional space information of the output image is correspondingly outputted.

5. A method for three-dimensional microscopic imaging based on multi-angle projection as claimed in claim 4, characterized in that, In step S3, at least two three-dimensional reconstruction images are mapped in the same coordinate system, and the at least two three-dimensional reconstruction images are fused by supervised learning of a neural network to generate a final fused three-dimensional image.

6. A method for three-dimensional microscopic imaging based on multi-angle projection as claimed in claim 5, characterized in that, The mapping manner includes rotation, translation, shearing, and scaling.

Images