Three-dimensional microscopic imaging apparatus and imaging method therefor

Abstract

A three-dimensional microscopic imaging apparatus (10) and an imaging method therefor. A lighting source module (100) is used for emitting excitation light; filtering and focusing is performed on the excitation light; the light that has been subjected to filtering and focusing is amplified at a preset amplification factor, and relaying processing is performed on the amplified excitation light, such that excitation light that has been subjected to first relaying is obtained; a scanning module (200) deflects on the basis of a first voltage signal the excitation light that has been subjected to first relaying, relays the deflected excitation light to an objective lens (12) to obtain excitation light that has been subjected to second relaying, focuses to a target sub-field of view the excitation light that has been subjected to the second relaying, so as to excite a sample in the target sub-field of view, collects a fluorescence signal emitted from the sample, and introduces the fluorescence signal into an imaging module (300); and the imaging module (300) is used for performing optical spatial spectrum multiplexing imaging on the basis of spatial spectrum information of the fluorescence signal, and recording an optical spatial spectrum multiplexing imaging result by means of a camera (19).

Description

Three-dimensional microscopic imaging device and imaging method thereof

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410332563.4 and application date of March 22, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of optical microscopy, and in particular to a three-dimensional microscopic imaging device and an imaging method thereof. Background Art

[0004] Rapid three-dimensional microscopic imaging of organisms distributed in a large three-dimensional space has important scientific significance in biomedical research.

[0005] Microscopes capable of 3D imaging in related technologies, such as two-photon microscopes, confocal microscopes, and light-sheet microscopes, suffer from complex structures and high costs. Furthermore, these microscopes image point by point or surface by surface, requiring the sequential traversal of multiple axial planes for 3D imaging. This limits their imaging speed due to the mechanical inertia of the scanning device, making it difficult to meet the demands of fast 3D imaging applications.

[0006] Fourier light field microscopy can achieve high-speed three-dimensional imaging, and it only requires a single exposure to image a three-dimensional volume. Its working principle is to place a microlens array composed of multiple (number of microlenses: v ≥ 2) microlenses on the spatial spectrum surface of the microscope system to perform optical spatial spectrum multiplexing of the spatial spectrum information, and simultaneously record the image of the object space at multiple perspectives (number: v ≥ 2) on the camera target surface. Because the projection relationship between the object and the image is different when observing a three-dimensional object from different perspectives, the images of the same area at different perspectives in the imaging results have significant differences. This significant difference is precisely due to the depth information of the three-dimensional object. Therefore, the imaging results of the object space at different perspectives can be fused through post-calculation to obtain a three-dimensional image of the object space. Based on the above single exposure and post-processing, Fourier light field microscopy can achieve fast three-dimensional microscopic imaging.

[0007] However, due to the need to simultaneously record images from multiple perspectives, the 3D imaging performance of Fourier light-field microscopy places extremely high demands on the camera's imaging target surface (size, pixel count) and imaging sampling rate. Even with a camera with an extremely large target surface, Fourier light-field microscopy can only confine the field of view to a relatively small area for 3D imaging, while meeting the Nyquist sampling rate (sampling rate greater than 2, i.e., image resolution / camera pixel size > 2). This makes it difficult for current Fourier light-field microscopy to perform rapid 3D microscopic imaging of biological samples distributed over a large field of view, a challenge that urgently needs to be addressed. Summary of the Invention

[0008] The present application provides a three-dimensional microscopic imaging device and an imaging method thereof to solve the problem that related technologies are difficult to achieve three-dimensional microscopic imaging with a large field of view and large volume.

[0009] The first embodiment of the present application provides a three-dimensional microscopic imaging device, comprising: an illumination light source module, a scanning module and an imaging module, wherein:

[0010] The illumination light source module includes a light source assembly, a filter, a first lens, a second lens, a third lens, and a fourth lens arranged in sequence. The light source assembly is used to emit excitation light. The light source assembly, the filter, and the first lens are used to filter the excitation light to form filtered excitation light. The second lens is used to focus the filtered excitation light to form focused excitation light. The third lens and the fourth lens amplify the focused excitation light at a preset magnification and perform relay processing to obtain a first relayed excitation light.

[0011] The scanning module includes a dichroic mirror, a scanning galvanometer, a fifth lens, a sixth lens, a first reflector, and an objective lens arranged in sequence, the dichroic mirror is used to reflect the first relayed excitation light to the scanning galvanometer, the scanning galvanometer deflects the first relayed excitation light based on a first voltage signal, the fifth lens, the sixth lens, and the first reflector are used to relay the deflected excitation light to the objective lens to obtain a second relayed excitation light, the objective lens is used to focus the second relayed excitation light in a target sub-field of view, excite a sample in the target sub-field of view, and collect a fluorescence signal emitted by the sample, the fluorescence signal enters the imaging system through the first reflector, the sixth lens, the fifth lens, the scanning galvanometer, and the dichroic mirror in sequence;

[0012] The imaging module includes a seventh lens, a second reflector, an aperture, an eighth lens, a filter, a microlens array and a camera arranged in sequence. The seventh lens, the second reflector, the aperture and the eighth lens are used to relay the spatial spectrum information of the fluorescence signal to the filter. The filter is used to filter the fluorescence signal. The microlens array is used to perform optical spatial spectrum multiplexing imaging based on the spatial spectrum information, and the camera records the optical spatial spectrum multiplexing imaging results.

[0013] Optionally, in some embodiments, the above-mentioned three-dimensional microscopic imaging device further includes:

[0014] A total internal reflection prism and a digital micromirror device, wherein the total internal reflection prism and the digital micromirror device are located between the filter and the first lens, wherein:

[0015] The digital micromirror device is used to load a preset structured light image to form structured light illumination, and modulate the excitation light based on the structured light illumination to form modulated excitation light;

[0016] The total internal reflection prism is used to reflect the excitation light to the digital micromirror device, receive the modulated excitation light, and reflect the modulated excitation light to the first lens.

[0017] Optionally, in some embodiments, the above-mentioned three-dimensional microscopic imaging device further includes:

[0018] The tracking module comprises an infrared illumination source, a ninth lens, an infrared band dichroic mirror, a light trap, a tenth lens, and a tracking camera arranged in sequence.

[0019] The infrared illumination light source is used to generate an infrared band to illuminate all imaging fields of the objective lens. The ninth lens and the tenth lens are used to image all imaging fields supported by the objective lens onto the tracking camera. The infrared band dichroic mirror is used to reflect the excitation light passing through the objective lens to the light trap. The light trap is used to eliminate the excitation light passing through the objective lens. The tracking camera is used to record the imaging images of all imaging fields.

[0020] Optionally, in some embodiments, the tracking module further includes: a processing unit,

[0021] The processing unit is used to receive the imaging image emitted by the tracking camera, process the imaging image to obtain the position of the target sub-field of view where the tracking target is located, calculate the second voltage signal corresponding to the target sub-field of view, and send the second voltage signal to the scanning module.

[0022] Optionally, in some embodiments, the microlens array is composed of at least one group of sub-lenses having the same optical parameters and the same depth of focus;

[0023] Alternatively, the microlens array with extended depth of field is composed of at least two groups of sub-lenses with the same optical parameters and the same focus depth that focus at different depths and one sub-lens whose imaging depth of field includes the sum of the focus depths of multiple groups.

[0024] A second embodiment of the present application provides a three-dimensional microscopic imaging method, comprising:

[0025] Based on a preset division strategy, the target large field of view is divided into multiple (n≥2) sub-fields of view, and the position and scanning imaging order of each sub-field of view are determined. The analog voltage signal corresponding to the position of each sub-field of view is calculated, and the three-dimensional point spread function of each sub-field of view is pre-collected in a direction perpendicular to the sub-field of view;

[0026] Generate a voltage signal corresponding to any sub-field of view, and apply the voltage signal to the scanning galvanometer, so that the scanning galvanometer deflects the excitation light to the corresponding sub-field of view and excites the sample in the corresponding sub-field of view;

[0027] The fluorescence emitted by the sample is collected by an objective lens, and spatial spectrum information of the fluorescence is relayed to a microlens array at a preset magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the spatial spectrum information to obtain multi-view imaging results of the corresponding sub-field of view, and the multi-view imaging results are recorded by a camera;

[0028] Traversing all sub-view fields until multi-view imaging results of all sub-view fields are obtained, and dividing the multi-view imaging results of all sub-view fields according to different imaging perspectives to obtain imaging results of all sub-view fields at different imaging perspectives;

[0029] Using the pre-collected three-dimensional point spread function and deconvolution reconstruction algorithm, the imaging results of all the sub-fields of view at different imaging angles are reconstructed to obtain three-dimensional reconstructed images of all the sub-fields of view, and the three-dimensional reconstructed images of all the sub-fields of view are spliced ​​to obtain a three-dimensional microscopic image of the target large field of view.

[0030] Optionally, in some embodiments, the generating of the excitation light and the voltage signal of any sub-field of view, the scanning galvanometer deflecting the excitation light to the corresponding sub-field of view based on the voltage signal, and exciting the sample in the corresponding sub-field of view, comprises:

[0031] The excitation light is reflected to a digital micromirror assembly through a total internal reflection prism, and a preset structured light image is loaded on the digital micromirror assembly to form structured light illumination, so as to modulate the excitation light to obtain modulated excitation light.

[0032] Optionally, in some embodiments, after segmenting the multi-view imaging results of all sub-fields of view according to different imaging perspectives to obtain imaging results of all sub-fields of view at different imaging perspectives, the method further includes:

[0033] The imaging results of each sub-field of view at different imaging angles are processed based on the structured light tomography algorithm to obtain the imaging results of all sub-fields of view at different imaging angles after removing the background signals.

[0034] Optionally, in some embodiments, after traversing all the sub-fields of view until multi-view imaging results of all the sub-fields of view are obtained, the method further includes:

[0035] The multi-view imaging results of all the sub-fields are grouped according to the focus depth to obtain the grouped multi-view imaging results, and the grouped multi-view imaging results are segmented according to different imaging perspectives to obtain the multi-focus imaging results of all the sub-fields at different imaging perspectives.

[0036] Optionally, in some embodiments, before generating a voltage signal corresponding to any sub-field of view, the method further includes:

[0037] A tracking module image including the entire objective lens field of view is acquired, a sub-field of view position where a tracking target is located is analyzed based on the tracking module image, and an analog voltage signal of the sub-field of view position is determined.

[0038] Therefore, this application has at least the following beneficial effects:

[0039] (1) The embodiments of the present application reduce the hardware requirements that are difficult to overcome in Fourier light field microscopes for large-volume three-dimensional microscopic imaging by introducing a scanning system module, thereby achieving large-volume, high-resolution, and high-sampling-rate fast three-dimensional microscopic imaging.

[0040] (2) The embodiment of the present application introduces a scanning module so that the imaging area can be flexibly adjusted according to actual needs, and can realize full-field scanning imaging, as well as scanning imaging between any fields of view. It can also cooperate with the tracking module to perform three-dimensional tracking imaging of moving targets.

[0041] (3) The embodiments of the present application introduce a digital micromirror device into the illumination light source module to generate structured light illumination, and further combine it with a computational optical tomography algorithm to achieve background-removed three-dimensional imaging, retaining only the imaging information within the target volume, thereby greatly enhancing the imaging contrast of three-dimensional microscopic imaging and achieving large-volume, high-resolution, and high-sampling-rate fast three-dimensional microscopic imaging with background removal.

[0042] (4) The embodiment of the present application makes full use of the redundancy of spatial spectrum information in the Fourier light field microscope and proposes a microlens array with an extended depth of field of the (x+1) type (x is the number of microlens groups focusing at different depths, x≥2). It can increase the imaging depth of field of the Fourier light field microscope while retaining the original imaging field of view, imaging resolution and imaging speed, and realize large-volume, high-resolution, high-sampling-rate fast three-dimensional microscopic imaging with an extended depth of field.

[0043] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0045] FIG1 is a block diagram of a three-dimensional microscopic imaging device according to an embodiment of the present application;

[0046] FIG2 is a schematic diagram of a common microlens array provided according to one embodiment of the present application;

[0047] FIG3 is a schematic diagram of a microlens array with extended depth of field according to one embodiment of the present application;

[0048] FIG4 is a block diagram of a three-dimensional microscopic imaging device according to another embodiment of the present application;

[0049] FIG5 is a block diagram of a three-dimensional microscopic imaging device according to another embodiment of the present application;

[0050] FIG6 is a schematic diagram showing the principle of full-field scanning according to an embodiment of the present application;

[0051] FIG7 is a schematic diagram of the principle of tracking scanning according to one embodiment of the present application;

[0052] FIG8 is a schematic diagram of a process of a three-dimensional microscopic imaging method according to an embodiment of the present application;

[0053] FIG9 is a schematic flow chart of a three-dimensional microscopic imaging method according to one embodiment of the present application;

[0054] FIG10 is a schematic flow chart of a three-dimensional microscopic imaging method according to another embodiment of the present application;

[0055] FIG11 is a schematic flow chart of a three-dimensional microscopic imaging method according to another embodiment of the present application. DETAILED DESCRIPTION

[0056] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0057] Before introducing the three-dimensional microscopic imaging device and imaging method thereof according to the embodiment of the present application, the working principle of the Fourier light field microscope is first introduced.

[0058] Fourier light field microscopes need to record images from multiple perspectives (number: v ≥ 2) simultaneously for 3D reconstruction. Therefore, for the object space with the same field of view, Fourier light field microscopes need to record v times more information than ordinary wide field microscopes in a single exposure. At the same time, the lateral resolution R x After determination, the axial resolution R of the reconstructed three-dimensional image z It is negatively correlated with the tangent value of the maximum viewing angle θ during imaging, that is, R z =R x / tg(θ), that is, the larger the maximum viewing angle θ, the smaller the value of the axial resolution and the stronger the axial resolution ability.

[0059] Optical principles show that the propagation angle of light in space corresponds to its spatial frequency, and the higher the numerical aperture of the objective lens, the higher the spatial frequency of light that can be collected. Therefore, to achieve stronger axial resolution, among objectives supporting the same imaging field of view, high numerical aperture objectives should be preferred, and more images should be collected in the high-frequency region of their spatial spectrum, that is, the value of v should be higher. Therefore, to achieve high spatial resolution three-dimensional imaging, Fourier light field microscopy requires a very large camera target surface to match it.

[0060] At the same time, in order to image the fine structures in the body, the optical system of the Fourier light field microscope must not only have high spatial resolution, but also provide sufficient system magnification M to ensure that the sampling rate SR of the image in the camera pixel is higher than the Nyquist sampling rate, that is, SR = M·R x / d pixel >2, where d pixel is the pixel size of the camera. If the imaging field of the Fourier light field microscope is FOV, then the number of pixels required on the camera is at least As can be seen, as the imaging field of view (FOV) increases, the number of pixels required by the camera increases dramatically. This places high demands on the imaging equipment of Fourier light field microscopes, including a large target area, a high pixel count, and a small pixel size.

[0061] For example, if you need to focus on a 3mm field of view, with a magnification of M=5 and R xTo achieve imaging at v = 9 viewing angles with a lateral resolution of 2μm, even assuming 100% target surface utilization and a minimum sampling rate (SR) of 2, the camera requires at least 9 × 3000 × 3000 pixels and a target area of ​​45mm × 45mm, far exceeding the parameters achievable by ordinary commercial cameras. Furthermore, in typical designs, priority is given to acquiring images with higher spatial frequencies to improve axial resolution. To this end, the camera's target surface has significant redundancy, and the actual required parameters are higher than the assumed values.

[0062] It can be seen that the extremely high requirements for hardware mean that the current Fourier light field microscope can only perform small-volume imaging within a small field of view, and it is difficult to expand to large-volume, high-resolution, and high-sampling-rate fast three-dimensional imaging.

[0063] To solve the above problems, the embodiment of the present application provides a three-dimensional microscopic imaging device and imaging method thereof, which determines multiple sub-fields of view (number: n≥2) within a large field of view, collects the three-dimensional point spread function of each sub-field of view, receives excitation light and the voltage signal of any sub-field of view, and based on the voltage signal, excites the sample in the corresponding sub-field of view through the excitation light, collects the fluorescence emitted by the sample through the objective lens, and relays the spatial spectrum information of the fluorescence to the microlens array at a preset magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the spatial spectrum information to obtain multi-view imaging results of the corresponding sub-field of view, and records the multi-view imaging results through a camera. At the same time, the imaging area is flexibly adjustable according to the actual needs of the sample to be observed, and can realize full-field scanning imaging, as well as rapid scanning imaging between any fields of view. It can also cooperate with the tracking module to perform three-dimensional tracking imaging of moving targets. All target sub-fields of view are traversed until multi-view imaging results are obtained for all sub-fields of view. These multi-view imaging results are then segmented according to different imaging perspectives to obtain imaging results for all sub-fields at different imaging perspectives. Using a three-dimensional point spread function and a preset reconstruction algorithm, the imaging results for all sub-fields at different imaging perspectives are reconstructed to obtain three-dimensional reconstructed images of all sub-fields of view. The three-dimensional reconstructed images of all sub-fields of view are then spliced ​​or aligned to obtain a three-dimensional microscopic image of the target. This solves the problem that related technologies can only perform small-volume imaging within a small field of view, making it difficult to achieve large-volume, high-resolution, and fast three-dimensional imaging. Large-volume, high-resolution, and high-sampling-rate three-dimensional imaging results can now be obtained.

[0064] Specifically, FIG1 is a block diagram of a three-dimensional microscopic imaging device provided in an embodiment of the present application.

[0065] As shown in FIG. 1 , the three-dimensional microscopic imaging device 10 includes an illumination light source module 100 , a scanning module 200 and an imaging module 300 .

[0066] The lighting light source module 100 includes a light source assembly 1, a filter 2, a first lens 3, a second lens 4, a third lens 5 and a fourth lens 6 arranged in sequence. The light source assembly 1 is used to emit excitation light. The light source assembly 1, the filter 2 and the first lens 3 are used to filter the excitation light to form filtered excitation light. The second lens 4 is used to focus the filtered excitation light to form focused excitation light. The third lens 5 and the fourth lens 6 amplify the focused excitation light at a preset magnification and relay it to obtain the first relayed excitation light.

[0067] The light source assembly 1 may be a collimated LED lighting source, and the second lens may be a focus-adjustable lens.

[0068] Specifically, the light source assembly 1 emits excitation light and passes through the excitation end filter 2. Subsequently, the excitation light passes through the first lens 3 and the second lens 4. The collimating lens in the light source assembly 1 and the first lens 3 form a 4f system. The second lens 4 is located on the rear focal plane of the first lens 3. The second lens 4 can be used to adjust the focusing depth of the excitation light behind the objective lens.

[0069] Then, the third lens 5 and the fourth lens 6 form another 4f relay system to relay the excitation light to the scanning module 200 at a specific magnification. Assuming the focal length of the third lens 5 is |f5| and the focal length of the fourth lens 6 is |f6|, the magnification M 5-6 It can be expressed as M 5-6 =|f6| / |f5|.

[0070] The scanning module 200 includes a dichroic mirror 7, a scanning galvanometer 8, a fifth lens 9, a sixth lens 10, a first reflector 11 and an objective lens 12 arranged in sequence along the optical path of the scanning module 200. The dichroic mirror 7 is used to reflect the first relayed excitation light to the scanning galvanometer 8. The scanning galvanometer 8 deflects the first relayed excitation light based on the first voltage signal. The fifth lens 9, the sixth lens 10 and the first reflector 11 are used to relay the deflected excitation light to the objective lens 12 to obtain the second relayed excitation light. The objective lens 12 is used to focus the second relayed excitation light on the target sub-field of view, excite the sample in the target sub-field of view, and collect the fluorescence signal emitted by the sample. The fluorescence signal enters the imaging module 300 through the first reflector 11, the sixth lens 10, the fifth lens 9, the scanning galvanometer 8 and the dichroic mirror 7 in sequence.

[0071] The scanning galvanometer 8 may be an XY dual-axis scanning galvanometer, and the first voltage signal may be a voltage signal corresponding to the target sub-market.

[0072] Specifically, the dichroic mirror 7 can be a long-wavelength dichroic mirror, and reflects the excitation light relayed by the illumination module 100 to the XY dual-axis scanning galvanometer 8. After receiving the voltage signal given by the digital board, the XY dual-axis scanning galvanometer 8 generates a mechanical deflection angle. The excitation light is directed at an angle Deflect the propagation direction.

[0073] The fifth lens 9 and the sixth lens 10 form a 4f relay system, which relays the excitation light from the center plane of the scanning galvanometer 8 to the back focal plane of the objective lens 12. The first reflector 11 is used to change the propagation direction of the light beam so that the propagation direction of the excitation light is vertically downward to facilitate the installation of biological samples. The objective lens 12 focuses the excitation light after the deflection of the propagation direction into the target sub-field of view located at the back focal plane of the objective lens, forming uniform light illumination to excite the sample in the sub-field of view. Assume that the focal length of the fifth lens 9 is |f9| and the focal length of the sixth lens 10 is |f 10 |, the focal length of the objective lens 12 is |f 12 |, then the distance between the center of the sub-field of view and the optical axis is

[0074] Subsequently, the objective lens 12 collects the fluorescence signal emitted by the sample in the sub-field of view, and reversely passes through the first reflector 11, the sixth lens 10, the fifth lens 9 to the XY dual-axis scanning galvanometer 8. The propagation direction of the fluorescence signal at this time is After passing through the XY dual-axis scanning galvanometer mirror 8 in the reverse direction, the deflection angle is offset and the fluorescence signal propagates along the optical axis. The fluorescence signal propagating along the optical axis passes through the dichroic mirror 7 and enters the imaging module 300.

[0075] The imaging module 300 includes a seventh lens 13, a second reflector 14, an aperture 15, an eighth lens 16, a filter 17, a microlens array 18 and a camera 19, which are sequentially arranged along the optical path of the imaging module.

[0076] Among them, the seventh lens 13, the second reflector 14, the aperture 15 and the eighth lens 16 are used to relay the spatial spectrum information of the fluorescence signal to the filter 17, the filter 17 is used to filter the fluorescence signal, the microlens array 18 is used to perform optical spatial spectrum multiplexing imaging based on the spatial spectrum information, and the camera 19 records the optical spatial spectrum multiplexing imaging results.

[0077] Specifically, the fluorescence signal collected in scanning module 200 is focused by seventh lens 13 and reflected by second reflector 14, forming an image at aperture 15 located at the rear focal plane of seventh lens 13. Aperture 15 is used to limit the size of the sub-field of view to prevent overlap between images from different perspectives due to excessively large sub-fields of view during imaging. Subsequently, eighth lens 16 and seventh lens 13 form a 4f relay system, relaying the spatial spectrum information of the fluorescence signal to collection-end filter 17. After passing through collection-end filter 17, the spatial spectrum information of the fluorescence signal is filtered out of stray light other than the fluorescence signal and reaches microlens array 18 located at the rear focal plane of eighth lens 16. Microlens array 18 can be a conventional microlens array or a microlens array with extended depth of field. Microlens array 18 performs optical spatial spectrum multiplexing imaging on the spatial spectrum information of the fluorescence signal, and the imaging results are recorded by camera 19 located at the rear focal plane of microlens array 18.

[0078] Optionally, in some embodiments, the microlens array is composed of at least one group of sub-lenses with the same optical parameters and the same depth of focus; or, a microlens array with an extended depth of field is composed of at least two groups of sub-lenses with the same optical parameters and the same depth of focus that focus at different depths, and one sub-lens whose imaging depth of field includes the sum of the depths of focus of multiple groups. It should be noted that in the embodiments of the present application, the microlens array 18 can be selected as a normal microlens array or a microlens array with an extended depth of field as required. The microlens array 18 can be composed of multiple sub-lenses (number: v ≥ 2) to achieve imaging from multiple perspectives.

[0079] As shown in Figure 2, in common commercial or self-processed microlens arrays, all sub-lenses have the same optical parameters and depth of focus, that is, the three-dimensional point spread functions of all sub-lenses intersect at the same depth in the axial direction. When using a common microlens array for imaging, the imaging depth of field of a large-volume three-dimensional microscopic imaging device is the same as the imaging depth of field of a single sub-lens, both of which are DOF.

[0080] As shown in FIG3 , the extended depth of field microlens array divides all sub-lenses in the high-frequency part into (x+1) groups. Among the x groups of sub-lenses, the sub-lenses in each group have the same optical parameters and focus depth, while the focus depths of the sub-lenses in different groups are different. As shown in FIG3 , the imaging depth of field of the first group of microlenses is DOF1, and the imaging depth of field of the second group of microlenses is DOF2. The difference in the focus depths of the two groups of microlenses is ΔDOF, and there is The total imaging depth of field is Therefore, an extended depth of field is achieved in the axial direction. At the same time, the microlens array with extended depth of field places a lens with different optical parameters from the above x groups of lenses at the center of the spectrum, and its imaging depth of field DOF0 includes the depth of field of the above x groups of lenses, that is, And the focal depth is at the center of the focal depth of the above x groups of lenses.

[0081] It should be noted by relevant technicians in this field that the principle of the 4f relay system indicated in the embodiments of the present application is as follows. For example, the third lens 5 and the fourth lens 6 are placed in sequence along the direction of optical propagation, their optical centers coincide with the optical axis, and the third lens 5 and the fourth lens 6 form a group of 4f relay systems. The focal length of the third lens 5 is |f1|, and the focal length of the third lens 5 is |f2|. Then, the relay system can relay the object located at the front focal plane of the third lens 5 (at -|f1| from the principal surface of lens 1 along the direction of optical propagation) to the back focal plane of the fourth lens 6 (at |f2| from the principal surface of lens 2 along the direction of optical propagation), and the distance between the third lens 5 and the fourth lens 6 is |f1|+|f2|.

[0082] Therefore, the embodiments of the present application fully utilize the redundancy of spatial spectrum information in Fourier light field microscopes and propose a (x+1) type extended depth of field microlens array, which can increase the imaging depth of field of the Fourier light field microscope while retaining the original imaging field of view, imaging resolution and imaging speed, thereby realizing large-volume, high-resolution, high-sampling-rate fast three-dimensional microscopic imaging with extended depth of field.

[0083] Optionally, as shown in Figure 4, in some embodiments, the above-mentioned three-dimensional microscopic imaging device 10 further includes: a total internal reflection prism 20 and a digital micromirror device 21, and the total internal reflection prism 20 and the digital micromirror device 21 are located between the filter 2 and the first lens 3.

[0084] The digital micromirror device 21 is used to load a preset structured light image to form structured light illumination, and modulate the excitation light based on the structured light illumination to form modulated excitation light.

[0085] The total internal reflection prism 20 is used to reflect the excitation light to the digital micromirror device 21 , receive the modulated excitation light, and reflect the modulated excitation light to the first lens 3 .

[0086] Specifically, the light source assembly 1 emits an excitation light and passes through the excitation end filter 2. After being reflected by the total internal reflection prism 20, the excitation light reaches the target surface of the digital micromirror device 21. The target surface of the digital micromirror device 21 is pre-loaded with a structured light pattern, and the loaded pattern modulates the excitation light to generate structured light illumination. Subsequently, the modulated excitation light passes through the total internal reflection prism 20 in the opposite direction. Subsequently, the modulated excitation light passes through the 4f relay system composed of the first lens 3 and the third lens 5, and is relayed to the front focal plane of the fourth lens 6, where it forms an image of the pattern loaded on the target surface of the digital micromirror device 21, wherein the second lens 4 is located at the back focal plane of the first lens 3 and is used to adjust the depth of focus. Subsequently, a relay system is formed by the third lens 5 and the fourth lens 6 to accurately relay the excitation light to the scanning module 200 at a specific magnification. Assuming that the focal length of the third lens 5 is |f7| and the focal length of the fourth lens 6 is |f8|, then the magnification M 7-8 It can be expressed as M 7-8 =|f8| / |f7|.

[0087] The scanning module 200 includes a dichroic mirror 7, an XY biaxial scanning galvanometer 8, a fifth lens 9, a sixth lens 10, a first reflector 11, and an objective lens 12, which are sequentially placed along the optical path of the scanning module 200. The dichroic mirror 7 uses a long-wavelength dichroic mirror and reflects the excitation light relayed by the illumination light source module 100 to the XY biaxial scanning galvanometer 8. After receiving the voltage signal given by the digital board, the XY biaxial scanning galvanometer 8 generates a mechanical deflection angle. The excitation light is directed at an angle Deflect the propagation direction. The fifth lens 9 and the sixth lens 10 form a 4f relay system to relay the excitation light from the center plane of the scanning galvanometer to the rear focal plane of the objective lens 12. The first reflector 11 is used to change the propagation direction of the light beam so that it is vertically downward, which is convenient for installing biological samples. The objective lens 12 focuses the excitation light after the propagation direction is deflected into the target sub-field of view located at the rear focal plane of the objective lens, and forms a structured light illumination modulated by the pattern loaded by the target surface of the digital micromirror device 21 in the target sub-field of view to excite the sample in the sub-field of view. Assume that the focal length of the fifth lens 9 is |f 11 |, the focal length of the sixth lens 10 is |f 12 |, the distance between the objective lens and the lens is |f 14 |, then the distance between the center of the target sub-field of view and the optical axis is

[0088] Subsequently, the objective lens 12 collects the fluorescence signal emitted by the sample in the sub-field of view, and reversely passes through the first reflector 11, the sixth lens 10, the fifth lens 9 to the XY dual-axis scanning galvanometer 8. The propagation direction of the fluorescence signal at this time is After passing through the XY dual-axis scanning galvanometer mirror 8 in the reverse direction, the deflection angle is offset and the fluorescence signal propagates along the optical axis. The fluorescence signal propagating along the optical axis passes through the dichroic mirror 7 and enters the imaging module 300.

[0089] The imaging module 300 includes a seventh lens 13, a second reflector 14, an aperture 15, an eighth lens 16, a filter 17, a microlens array 18, and a camera 19, arranged in sequence along the imaging module's optical path. The fluorescence signal collected in the scanning module 200 is focused by the seventh lens 13 and reflected by the second reflector 14, forming an image at the aperture 15 located at the rear focal plane of the seventh lens 13. The aperture 15 is used to limit the size of the sub-field of view to prevent overlap between images from different perspectives caused by an excessively large sub-field of view during imaging. Subsequently, the eighth lens 16 and the seventh lens 13 form a 4f relay system, relaying the spatial spectrum information of the fluorescence signal to the collection filter 17. After passing through the collection filter 17, the spatial spectrum information of the fluorescence signal is filtered out of stray light other than the fluorescence signal and reaches the microlens array 18 located at the rear focal plane of the eighth lens 16. The microlens array 18 can be a common microlens array or a microlens array with an extended depth of field. The microlens array 18 performs optical spatial spectrum multiplexing imaging on the spatial spectrum information of the fluorescence signal, and the imaging results are recorded by a camera 19 located at the rear focal plane of the microlens array 18. The microlens array 18 can be a common microlens array or a microlens array with an extended depth of field. The microlens array 18 performs optical spatial spectrum multiplexing imaging on the spatial spectrum information of the fluorescence signal, and the imaging results are recorded by a camera 19 located at the rear focal plane of the microlens array 18.

[0090] Therefore, the embodiment of the present application introduces a digital micromirror device into the illumination light source module to generate structured light illumination, and further combines it with a computational optical tomography algorithm to achieve background-removed three-dimensional imaging, retaining only the imaging information within the target volume, thereby greatly enhancing the imaging contrast of three-dimensional microscopic imaging and achieving large-volume, high-resolution, and high-sampling-rate fast three-dimensional microscopic imaging with background removal.

[0091] Optionally, as shown in FIG5 , in some embodiments, the three-dimensional microscopic imaging device 10 further includes a tracking module 400 .

[0092] The tracking module 400 includes an infrared illumination source 22 , a ninth lens 23 , an infrared band dichroic mirror 24 , a light trap 25 , a tenth lens 26 and a tracking camera 27 , which are arranged in sequence.

[0093] The infrared illumination light source 22 is used to generate all imaging fields of the infrared band illumination objective lens 12. The ninth lens 23 and the tenth lens 26 constitute a 4f imaging system, which images all imaging fields supported by the objective lens 12 on the tracking camera 27. The infrared band dichroic mirror 24 is used to reflect the excitation light passing through the objective lens 12 to the light trap 24. The light trap 24 is used to eliminate the excitation light passing through the objective lens 12 to eliminate the interference of the excitation light passing through the objective lens 12 on the infrared illumination imaging. The tracking camera 27 is used to record the imaging images of all imaging fields.

[0094] Optionally, in some embodiments, the tracking module 400 further includes: a processing unit 28 .

[0095] Among them, the processing unit 28 can be a computer, which is used to receive the imaging image emitted by the tracking camera 27, process the imaging image to obtain the position of the target sub-field of view where the tracking target is located, and calculate the second voltage signal corresponding to the target sub-field of view, and send the second voltage signal to the scanning module 200.

[0096] Specifically, after the tracking camera 27 forms an image, the image is transmitted to the processing unit 28, and real-time image processing is performed to analyze the sub-field of view position of the tracking target at this moment, and calculate the voltage signal corresponding to the sub-field of view. The processing unit 28 then sends the second voltage signal to the scanning module through the digital board.

[0097] It should be noted that the scanning module 200 provided in the embodiment of the present application can provide flexible scanning methods, such as full-field scanning, scanning between arbitrary fields of view, and tracking scanning, among which, full-field scanning is to traverse all fields of view supported by the scanning objective lens and form images with a specific scanning trajectory; scanning between arbitrary fields of view is to arbitrarily select one or more fields of view (number: v≥2) within the field of view supported by the objective lens for scanning and imaging; tracking scanning is to cooperate with the tracking module 400 to scan in a continuously changing field of view, that is, within the field of view supported by the objective lens, the field of view position (ROI) of the tracking target is obtained through the tracking module 400, and the field of view position can change with time, and the position of the above-mentioned target is tracked in real time through the scanning module 200.

[0098] As shown in Figure 6, d FOV The field size of the entire field supported by the objective lens, d FOV_sub is the field of view size of a single sub-field of view, where d FOV_sub The size should be the maximum imaging size that can be supported by the camera target surface in a single exposure under the conditions that the imaging sampling rate and the number of imaging angles v are sufficient. Δd is the spacing between the sub-fields of view.

[0099] In order to obtain effective stitching results when performing full-field scanning imaging, a certain proportion of overlapping areas is required between adjacent sub-fields of view, that is, dFOV_sub -Δd. The overlapping area ratio can be designed independently, such as 10%, that is,

[0100] During full-field scanning imaging, the degree of change in the mechanical angle of the XY dual-axis scanning galvanometer mirror 8 affects its response time. Therefore, to avoid large mechanical angle changes in the XY dual-axis scanning galvanometer mirror during continuous scanning, the scanning direction indicated by the arrow in Figure 6 can be adopted. The XY dual-axis scanning galvanometer mirror starts from the X2Y2 field of view and passes through X2Y2→X3Y3, X3Y3→X2Y3, X2Y3→X1Y3, X1Y3→X1Y2, X1Y2→X1Y1, X1Y1→X2Y1, X2Y1→X3Y1, X3Y2→X3Y2, and X3Y2→X2Y2, completing a scanning cycle.

[0101] When performing tracking scanning imaging, as shown in Figure 7, d FOV is the field size of the entire field of view supported by the objective lens 12, ROI is the position of the tracking target at different times, when the tracking target is at d FOV When the robot moves freely within the space, the tracking module 400 and the scanning module 200 can maintain real-time tracking and real-time three-dimensional imaging of the tracking target.

[0102] Therefore, the embodiment of the present application introduces a scanning module so that the imaging area can be flexibly adjusted according to actual needs, and can realize full-field scanning imaging, as well as scanning imaging between any fields of view. It can also cooperate with the tracking module to perform three-dimensional tracking imaging of moving targets.

[0103] According to the three-dimensional microscopic imaging device proposed in the embodiment of the present application, an excitation light is emitted by an illumination light source module, and the excitation light is filtered and focused, and the filtered and focused excitation light is amplified at a preset magnification and relayed to obtain a first relayed excitation light, and a scanning module is used to deflect the first relayed excitation light based on a first voltage signal, and the deflected excitation light is relayed to the objective lens to obtain a second relayed excitation light, and the second relayed excitation light is focused in the target sub-field of view, exciting the sample in the target sub-field of view, and collecting the fluorescence signal emitted by the sample, and introducing the fluorescence signal into the imaging system, and using the imaging module to perform optical spatial spectrum multiplexing imaging based on the spatial spectrum information of the fluorescence signal, and the camera records the optical spatial spectrum multiplexing imaging results. Thus, the problem that the related technology is difficult to achieve large-field-of-view, large-volume three-dimensional microscopic imaging is solved, and high-resolution, high-sampling-rate, large-field fast three-dimensional imaging can be performed.

[0104] Next, the three-dimensional microscopic imaging method proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0105] FIG8 is a flow chart of a three-dimensional microscopic imaging method according to an embodiment of the present application.

[0106] As shown in FIG8 , the three-dimensional microscopic imaging method includes the following steps:

[0107] In step S101, based on a preset division strategy, the target large field of view is divided into multiple sub-fields of view (number: n≥2), and the position and scanning imaging order of each sub-field of view are determined, and the analog voltage signal corresponding to the position of each sub-field of view is calculated, and the three-dimensional point spread function of each sub-field of view is pre-collected in a direction perpendicular to the sub-field of view.

[0108] Among them, the target large field of view is the maximum field of view supported by the objective lens or any sub-field of view within the maximum field of view supported by the objective lens. The preset division strategy is to determine the imaging resolution Rx and the sampling multiple, and determine the maximum pixel of each sub-field of view based on the imaging resolution Rx and the sampling multiple, determine the maximum size of the sub-field of view based on the maximum pixel of each sub-field of view and the camera parameters, and divide it according to the maximum size of the sub-field of view. Specifically, the imaging resolution is determined to be Rx. If double sampling is required, the maximum size of each pixel in the sub-field of view is Rx / 2. The maximum value of the sub-field of view is the size of a sub-lens on the microlens array in pixels occupied on the camera. For example, each sub-lens corresponds to 1200*1200 pixels, and the maximum size of the sub-field of view is 1200*Rx / 2. When Rx=3μm, the maximum size of the sub-field of view is approximately 1.8mm*1.8mm. The number of sub-fields of view in the embodiment of the present application can be determined by the maximum field of view size supported by the objective lens. For example, a 4x objective lens can support a 5mm*5mm field of view. The number of target sub-fields of view (9) is obtained by dividing the maximum field of view size supported by the objective lens by the maximum size of the sub-field of view and rounding it up.

[0109] Specifically, the embodiment of the present application can determine the position of each sub-field of view (n in number) scanned and imaged within a large field of view by modulating the input voltage of the scanning galvanometer control component, and then encode the scanning order of each sub-field of view, for example, sub-field of view 1, sub-field of view 2...sub-field of view n. Within the field of view supported by the objective lens, the excitation light is focused on the front focal plane of the objective lens at different determined sub-fields of view by deflecting the direction of the excitation light at different angles by the scanning galvanometer. Within each determined sub-field of view, the fluorescent ball is moved in a direction perpendicular to the field of view with a preset step size (e.g., 1 μm). After each movement, an image at this depth is collected until the entire depth of field range (e.g., 200 μm) is traversed, thereby completing the collection of the three-dimensional point spread function of each viewing angle in the sub-field of view.

[0110] It should be noted that before passing through the scanning galvanometer, the center of the excitation light propagates along the optical axis of the system. After being deflected by the scanning galvanometer, the center propagation angle of the excitation light is is the mechanical rotation angle of the scanning galvanometer twice, that is The mechanical rotation angle generated by the scanning galvanometer It is proportional to the value of the voltage applied to the scanning galvanometer.

[0111] The equivalent focal length of the optical system that the excitation light passes through before reaching the sub-field of view is f o , the central propagation angle is The vector distance between the center position of the sub-field of view where the excitation light is focused after the objective lens and the center position of the field of view of the objective lens is

[0112] The position and number of pre-coded sub-fields within the large field of view can be adjusted as needed, including full-field scanning or scanning between arbitrary fields of view. Full-field scanning involves traversing the entire field of view supported by the scanning objective lens along a specific scanning trajectory and imaging. Random field scanning involves scanning and imaging one or more fields (number: n ≥ 2) within the field of view supported by the objective lens.

[0113] When collecting the three-dimensional point spread function of each viewing angle in each sub-field of view, the size of the selected fluorescent ball should be smaller than the lateral resolution R of the system. x , the preset step size moving in the direction perpendicular to the field of view should be smaller than the axial resolution R of the system z .

[0114] In step S102 , a voltage signal corresponding to any sub-field of view is generated and applied to a scanning galvanometer, so that the scanning galvanometer deflects the excitation light to the corresponding sub-field of view and excites the sample in the corresponding sub-field of view.

[0115] Specifically, the excitation light is generated by the illumination light source, and at the same time, the control component of the scanning galvanometer receives a voltage signal to scan it to any sub-field of view and generates a deflection of the corresponding angle. After the scanning galvanometer deflects the excitation light, it is projected to the sub-field of view through the objective lens, exciting the sample in the sub-field of view and giving the camera a trigger signal at the same time.

[0116] It should be noted that the mechanical deflection of the scanning galvanometer and the exposure of the camera are synchronized through analog / digital signals. After the scanning galvanometer undergoes a mechanical deflection, the camera simultaneously receives a trigger signal to start exposure and starts exposure. The camera's exposure time is less than the time interval between two adjacent trigger signals. Before the scanning galvanometer undergoes the next mechanical deflection, the camera completes the exposure and data transmission.

[0117] In step S103, the fluorescence emitted by the sample is collected through the objective lens, and the spatial spectrum information of the fluorescence is relayed to the microlens array at a preset magnification. The microlens array performs optical spatial multiplexing imaging on the spatial spectrum information to obtain multi-view imaging results of the corresponding sub-field of view, and the multi-view imaging results are recorded by the camera.

[0118] Among them, the multi-view imaging results are determined by the number of microlenses, and the number of viewing angles is determined by the number of microlenses.

[0119] Specifically, the objective lens collects the fluorescence emitted by the sample in the sub-field of view in step S102. The fluorescence passes back through the scanning galvanometer to offset the deflection angle generated by the scanning and propagates along the optical axis of the system. Subsequently, the spatial spectrum information of the fluorescence is relayed to the microlens array at a specific magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the above-mentioned spatial spectrum information. After receiving the trigger signal in step S102, the camera placed at the rear focal plane of the microlens array starts exposure and records the multi-view imaging results (the number of view angles is v) of the sub-field of view. After reaching the set exposure time, the exposure is stopped and the image is transmitted to the computer.

[0120] It should be noted that the microlens array in the embodiment of the present application is composed of multiple sub-lenses (number: v ≥ 2) to achieve imaging under multi-viewing angles. Ordinary commercial or independently designed and processed microlens arrays can be selected according to needs. All sub-lenses have the same optical parameters and focal depth, that is, the three-dimensional point spread functions of all sub-lenses intersect at the same depth in the axial direction.

[0121] In step S104, all sub-fields of view are traversed until multi-view imaging results of all sub-fields of view are obtained, and the multi-view imaging results of all sub-fields of view are divided according to different imaging perspectives to obtain imaging results of all sub-fields of view at different imaging perspectives.

[0122] Specifically, the embodiment of the present application can sequentially traverse all sub-fields of view according to the scanning order encoded in step S101, and repeat steps S102 to S103. After the traversal is completed, a cycle of acquisition is completed, and images of each sub-field of view are obtained (the number is n, which is the same as the number of sub-fields of view). Subsequently, the imaging results of each sub-field of view are divided according to different imaging perspectives to obtain imaging results of each perspective of each sub-field of view (the number is n×v).

[0123] In step S105, the imaging results of all sub-fields of view at different imaging angles are reconstructed using the pre-collected three-dimensional point spread function and the deconvolution reconstruction algorithm to obtain three-dimensional reconstructed images of all sub-fields of view, and the three-dimensional reconstructed images of all sub-fields of view are spliced ​​to obtain a three-dimensional microscopic image of the target large field of view.

[0124] Specifically, the embodiment of the present application requires three-dimensional reconstruction of the image of each sub-field of view. Specifically, using the three-dimensional point spread function of the sub-field of view collected in step S101 and the Fourier light field reconstruction algorithm based on the multi-view iterative deconvolution method, the imaging results of each view angle (number v) are reconstructed into a three-dimensional image (number 1). After completing the three-dimensional reconstruction of all sub-fields of view (number n), the three-dimensional reconstruction results of all sub-fields of view are then spliced ​​according to their three-dimensional positions to obtain a large-field-of-view, high-resolution, high-sampling-rate three-dimensional microscopic image (number 1).

[0125] It should be noted that if the position and number of sub-fields of view are adjusted according to actual needs in step S101, such as scanning and imaging between any discontinuous sub-fields of view, the imaging sub-fields of view do not need to be stitched, and the stitching step can be skipped according to needs.

[0126] In summary, in combination with what is shown in FIG9 , the embodiment of the present application determines the position of each sub-field of view for scanning imaging within a large field of view by modulating the input voltage of the scanning galvanometer control component, and then encodes the scanning order of each sub-field of view, and within the field of view supported by the objective lens, deflects the direction of the excitation light at different angles by the scanning galvanometer so that the excitation light is focused on the different sub-fields of view determined at the front focal plane of the objective lens. In each determined sub-field of view, the fluorescent ball is moved in a direction perpendicular to the field of view with a preset step length, and an image at this depth is collected after each movement until the entire depth of field range is traversed, completing the collection of the three-dimensional point spread function of each viewing angle in the sub-field of view. The excitation light is generated by the illumination light source, and at the same time, the control component of the scanning galvanometer receives a voltage signal that causes it to scan to a certain sub-field of view and generate a deflection of a preset angle. After the scanning galvanometer deflects the excitation light, it is projected to the sub-field of view through the objective lens, exciting the sample in the sub-field of view and giving the camera a trigger signal at the same time.

[0127] The objective lens collects the fluorescence emitted by the sample in the sub-field of view. The fluorescence passes back through the scanning galvanometer to offset the deflection angle generated by the scanning and propagates along the optical axis of the system. Subsequently, the spatial spectrum information of the fluorescence is relayed to the microlens array at a specific magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the above spatial spectrum information. The camera placed at the rear focal plane of the microlens array starts exposure after the trigger signal, records the multi-view imaging results of the sub-field of view (the number of view angles is v), stops exposure after reaching the set exposure time, and transmits the image to the computer. According to the coded scanning order, each preset sub-field of view is traversed in turn. After the traversal is completed, a cycle of acquisition is completed, and the image of each sub-field of view is obtained.

[0128] Subsequently, the imaging results of each sub-field of view are segmented according to different imaging perspectives to obtain the imaging results of each sub-field of view at each perspective. The image of each sub-field of view is then reconstructed in three dimensions. The imaging results of each perspective are reconstructed into a three-dimensional image using the three-dimensional point spread function of the corresponding sub-field of view and a Fourier light field reconstruction algorithm based on the multi-perspective iterative deconvolution method. After completing the three-dimensional reconstruction of all sub-fields of view, the three-dimensional reconstruction results of all sub-fields of view are spliced ​​according to their three-dimensional positions to obtain a three-dimensional microscopic image with a large field of view, high resolution, and high sampling rate. This solves the problem that related technologies can only perform small-volume imaging within a small field of view, making it difficult to achieve large-volume, high-resolution, and fast three-dimensional imaging.

[0129] Optionally, in some embodiments, excitation light and a voltage signal of any sub-field of view are generated, and the scanning galvanometer deflects the excitation light to the corresponding sub-field of view based on the voltage signal, and excites the sample in the corresponding sub-field of view, including: reflecting the excitation light to the digital micromirror assembly through a total internal reflection prism, and using the digital micromirror assembly to load a preset structured light image to form structured light illumination, so as to modulate the excitation light to obtain modulated excitation light.

[0130] Furthermore, in some embodiments, after dividing the multi-view imaging results of all sub-fields of view according to different imaging viewpoints to obtain the imaging results of all sub-fields of view at different imaging viewpoints, it also includes: processing the imaging results of each sub-field of view at different imaging viewpoints based on a structured light tomography algorithm to obtain the imaging results of all sub-fields of view at different imaging viewpoints after removing the background signals.

[0131] Specifically, as shown in FIG10 , the embodiment of the present application can also achieve large-volume, high-resolution, rapid three-dimensional microscopic imaging with background removed.

[0132] Step S301: By modulating the input voltage of the scanning galvanometer control component, the position of each sub-field of view (the number is n) scanned and imaged within the large field of view is determined, and then the scanning order of each sub-field of view is encoded (such as sub-field of view 1 → sub-field of view 2 → ... → sub-field of view n). Within the field of view supported by the objective lens, the excitation light direction is deflected at different angles by the scanning galvanometer so that the excitation light is focused on the different sub-fields determined at the front focal plane of the objective lens. In each determined sub-field of view, the fluorescent ball is moved in a direction perpendicular to the field of view with a preset step size (such as 1 μm). After each movement, an image at this depth is collected until the entire depth of field range (such as 200 μm) is traversed, completing the collection of the three-dimensional point spread function of each viewing angle in the sub-field of view.

[0133] Step S302: An illumination light source generates excitation light, which is then reflected onto a digital micromirror device (DMD) via a total internal reflection prism. The structured light pattern loaded onto the DMD modulates the excitation light to generate structured light illumination. Simultaneously, the control component of the scanning galvanometer receives a voltage signal that causes it to scan a sub-field of view and deflect at a preset angle. The scanning galvanometer deflects the modulated excitation light and projects it onto the sub-field of view through the objective lens, generating structured light illumination within the sub-field of view, thereby stimulating the sample within the sub-field of view and providing a trigger signal to the camera.

[0134] Furthermore, the excitation light is reflected by a total internal reflection prism onto a digital micromirror device (DMD). The DMD is composed of multiple micromirrors (e.g., 1920×1080), which can be loaded with any 8-bit image or binary image to generate any structured light. In this embodiment of the present application, the excitation light can be modulated by the structured light pattern loaded on the DMD to achieve a structured light illumination effect.

[0135] Step S303: The objective lens collects the fluorescence emitted by the sample within the sub-field of view in step S302. This fluorescence is then reflected back through the scanning galvanometer to offset the deflection angle caused by the scanning and propagates along the system's optical axis. The spatial spectrum information of this fluorescence is then relayed to the microlens array at a specific magnification. The microlens array performs optical spatial spectrum multiplexing imaging on this spatial spectrum information. Upon receiving a trigger signal, a camera placed at the rear focal plane of the microlens array begins exposure, recording the multi-view imaging results (number of views v) for the sub-field of view. Exposure stops after the set exposure time is reached and the image is transmitted to a computer.

[0136] Step S304: Traverse each preset sub-field of view in sequence according to the scanning order encoded in step S301, and repeat steps S302 to S303. After the traversal is completed, one cycle of acquisition is completed to obtain images of each sub-field of view (the number is n, the same as the number of sub-fields of view). Subsequently, the imaging results of each sub-field of view are divided according to different imaging perspectives to obtain imaging results of each perspective of each sub-field of view (the number is n×v). Subsequently, the background signal is removed from each imaging result image of each sub-field of view using the structured light tomography algorithm to obtain the imaging results of each perspective of each sub-field of view after removing the background signal (the number is n×v).

[0137] It should be noted that, in combination with structured light illumination in step S302, a structured light tomography algorithm, for example, a SIM (Structured illumination microscopy) algorithm, can be used to perform background signal removal processing on each imaging result image of each sub-field of view at each viewing angle to obtain the imaging results of each viewing angle of each sub-field of view after removing the background signal (the number is n×v).

[0138] Step S305: Perform three-dimensional reconstruction on the image of each sub-field of view, and use the three-dimensional point spread function of the sub-field of view collected in step S301 and the Fourier light field reconstruction algorithm based on the multi-view iterative deconvolution method to reconstruct the imaging results of each view after removing the background signal (the number is v) into a three-dimensional image after removing the background (the number is 1).

[0139] Step S306: Repeat step S305 to complete the 3D reconstruction of all sub-fields of view (number n). The background-removed 3D reconstruction results of all sub-fields of view obtained above are then spliced ​​according to their 3D positions to obtain a background-removed large-field-of-view, high-resolution, high-sampling-rate 3D microscopic image (number 1).

[0140] It should be noted that if the position and number of sub-fields of view are adjusted according to actual needs in step S301, such as scanning and imaging between any discontinuous sub-fields of view, the imaging sub-fields of view do not need to be spliced, and step S306 can be skipped as needed.

[0141] Optionally, in some embodiments, after traversing all sub-fields of view until multi-view imaging results of all sub-fields of view are obtained, it also includes: grouping the multi-view imaging results of all sub-fields of view according to the focus depth to obtain the grouped multi-view imaging results, and dividing the grouped multi-view imaging results according to different imaging perspectives to obtain multi-focus imaging results of all sub-fields of view at different imaging perspectives.

[0142] Specifically, as shown in FIG11 , the embodiment of the present application can also realize large-field three-dimensional imaging with an extended depth of field. It should be noted that if the embodiment of the present application is to obtain a large-field three-dimensional imaging result with an extended depth of field, the prerequisite is to use multiple groups of focused microlens arrays.

[0143] Step S401: By modulating the input voltage of the scanning galvanometer control component, the position of each sub-field of view (the number is n) scanned and imaged within the large field of view is determined, and then the scanning order of each sub-field of view is encoded (for example, sub-field of view 1 → sub-field of view 2 → ... → sub-field of view n). Within the field of view supported by the objective lens, the excitation light is focused on the front focal plane of the objective lens at different determined sub-fields of view by deflecting the direction of the excitation light at different angles by the scanning galvanometer. Within each determined sub-field of view, the fluorescent ball is moved in a direction perpendicular to the field of view with a preset step size (such as 1 μm), and an image at this depth is collected after each movement until the entire depth of field range (such as 200 μm) is traversed, completing the collection of the three-dimensional point spread function of each viewing angle in the sub-field of view.

[0144] Step S402: The illumination source generates excitation light. Simultaneously, the control component of the scanning galvanometer receives a voltage signal from a sub-field of view and deflects the light at a preset angle. The scanning galvanometer deflects the excitation light and projects it through the objective lens onto the sub-field of view, stimulating the sample within it and simultaneously triggering the camera to begin exposure.

[0145] Step S403: The objective lens collects the fluorescence emitted by the sample in the sub-field of view in step S402. The fluorescence is reversed through the scanning galvanometer to offset the deflection angle generated by the scanning and propagates along the optical axis of the system. Subsequently, the spatial spectrum information of the fluorescence is relayed to the microlens array with an extended depth of field at a specific magnification. The microlens array with an extended depth of field is characterized in that it includes x groups of high-NA lens groups focusing at different depths and one low-NA lens located at the center of the spatial spectrum. The imaging result of the low-NA lens contains signals across the entire depth of field range and can be used to enhance the three-dimensional reconstruction of the lens group at any set of focus depths. The microlens array performs optical spatial spectrum multiplexing imaging on the above-mentioned spatial spectrum information. The camera placed at the rear focal plane of the microlens array starts exposure after receiving a trigger signal, records the multi-view imaging results of the sub-field of view (the number of view angles is v), stops exposure after reaching the set exposure time, and transmits the image to the computer.

[0146] It should be noted that the microlens array is composed of multiple sub-lenses (number: v≥2) to achieve imaging under multiple viewing angles, which can be selected according to needs. In ordinary commercial or independently designed and processed microlens arrays, all sub-lenses have the same optical parameters and focal depth, that is, the three-dimensional point spread functions of all sub-lenses intersect at the same depth in the axial direction. The microlens array with extended depth of field mentioned in the embodiment of the present application divides all sub-lenses in the high-frequency part into x+1 groups, wherein the sub-lenses in each group of x groups of sub-lenses have the same optical parameters and focal depth, while the focal depths of sub-lenses between different groups are different, thereby achieving extended depth of field in the axial direction. At the same time, a lens with different optical parameters from the above-mentioned x groups of lenses is placed at the center of the spatial spectrum, and its imaging depth of field includes the depth of field of the above x groups of lenses, and the focal depth is at the center of the focal depth of the above x groups of lenses.

[0147] Step S404: traverse the preset sub-fields in sequence according to the scanning order encoded in step S401, and repeat steps S402 to S403. After the traversal is completed, a cycle of acquisition is completed to obtain images of each sub-field (the number is n, the same as the number of sub-fields). Subsequently, the imaging results of each sub-field are first divided into (x+1) groups according to their focus depth, and then the images of each group with different imaging angles in the x groups are segmented to obtain multi-focus imaging results of each sub-field and each angle (the number is ).

[0148] Step S405: Perform 3D reconstruction on each sub-field of view, and use the 3D point spread function of the sub-field of view acquired in step S101 and the Fourier light field reconstruction algorithm based on the multi-view iterative deconvolution method to reconstruct the imaging results of each view at different focus depths (the number is ) are reconstructed into x three-dimensional images (the number is x).

[0149] Step S406: Repeat step S405 to complete the 3D reconstruction of all sub-fields of view (number is n×x). Then, the 3D reconstruction results of all sub-fields of view obtained above are stitched together according to their 3D positions and focus depths to obtain a large field of view, high-resolution 3D result with extended depth of field (number is 1).

[0150] It should be noted that in step S401, the position and number of sub-fields of view are adjusted according to actual needs. For example, when scanning and imaging are performed between any discontinuous sub-fields of view, the imaging sub-fields of view do not need to be spliced, and step S406 can be skipped according to needs.

[0151] Optionally, in some embodiments, before generating a voltage signal corresponding to any sub-field of view, it includes: acquiring a tracking module image including the entire objective lens field of view, analyzing the sub-field of view position where the tracking target is located based on the tracking module image, and determining an analog voltage signal at the sub-field of view position.

[0152] Specifically, in combination with what is shown in FIG5 , the embodiment of the present application can also be combined with a tracking module to realize real-time tracking and real-time three-dimensional imaging of the tracking target. The tracking module obtains the position of the target sub-field of view where the tracking target is located, and calculates the voltage signal corresponding to the target sub-field of view where the tracking target is located. The scanning module controls the deflection of the scanning galvanometer based on the voltage signal to realize real-time tracking and real-time three-dimensional imaging of the tracking target.

[0153] It should be noted that the above explanations of the embodiment of the three-dimensional microscopic imaging device are also applicable to the three-dimensional microscopic imaging method of this embodiment, and will not be repeated here.

[0154] According to the three-dimensional microscopic imaging method proposed in the embodiment of the present application, by determining multiple sub-fields of view within a large field of view and collecting the three-dimensional point spread function of each sub-field of view, excitation light and a voltage signal of any sub-field of view are received, and based on the voltage signal, the sample in the corresponding sub-field of view is excited by the excitation light, and the fluorescence emitted by the sample is collected through the objective lens, and the spatial spectrum information of the fluorescence is relayed to the microlens array at a preset magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the spatial spectrum information to obtain multi-view imaging results of the corresponding sub-field of view, and the multi-view imaging results are recorded by a camera, and all sub-fields of view are traversed until the multi-view imaging results of all sub-fields of view are obtained, and the multi-view imaging results of all sub-fields of view are divided according to different imaging perspectives to obtain imaging results of all sub-fields of view at different imaging perspectives; using the three-dimensional point spread function and a preset reconstruction algorithm, the imaging results of all sub-fields of view at different imaging perspectives are reconstructed to obtain three-dimensional reconstructed images of all sub-fields of view, and the three-dimensional reconstructed images of all sub-fields of view are spliced ​​to obtain a target three-dimensional microscopic image. This solves the problem that related technologies can only perform small-volume imaging in a small field of view and are difficult to achieve large-volume, high-resolution, fast three-dimensional imaging. Large-volume, high-resolution, and high-sampling-rate three-dimensional imaging results can be obtained.

[0155] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0156] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0157] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0158] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0159] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0160] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A three-dimensional microscopic imaging device, characterized in that: include: Illumination light source module, scanning module and imaging module, wherein, The illumination light source module includes a light source assembly, a filter, a first lens, a second lens, a third lens, and a fourth lens arranged in sequence. The light source assembly is used to emit excitation light. The light source assembly, the filter, and the first lens are used to filter the excitation light to form filtered excitation light. The second lens is used to focus the filtered excitation light to form focused excitation light. The third lens and the fourth lens amplify the focused excitation light at a preset magnification and perform relay processing to obtain a first relayed excitation light. The scanning module includes a dichroic mirror, a scanning galvanometer, a fifth lens, a sixth lens, a first reflector, and an objective lens arranged in sequence, the dichroic mirror is used to reflect the first relayed excitation light to the scanning galvanometer, the scanning galvanometer deflects the first relayed excitation light based on a first voltage signal, the fifth lens, the sixth lens, and the first reflector are used to relay the deflected excitation light to the objective lens to obtain a second relayed excitation light, the objective lens is used to focus the second relayed excitation light in a target sub-field of view, excite a sample in the target sub-field of view, and collect a fluorescence signal emitted by the sample, the fluorescence signal enters the imaging system through the first reflector, the sixth lens, the fifth lens, the scanning galvanometer, and the dichroic mirror in sequence; The imaging module includes a seventh lens, a second reflector, an aperture, an eighth lens, a filter, a microlens array and a camera arranged in sequence. The seventh lens, the second reflector, the aperture and the eighth lens are used to relay the spatial spectrum information of the fluorescence signal to the filter. The filter is used to filter the fluorescence signal. The microlens array is used to perform optical spatial spectrum multiplexing imaging based on the spatial spectrum information, and the camera records the optical spatial spectrum multiplexing imaging results.

2. The device according to claim 1, characterized in that Also includes: A total internal reflection prism and a digital micromirror device, wherein the total internal reflection prism and the digital micromirror device are located between the filter and the first lens, wherein: The digital micromirror device is used to load a preset structured light image to form structured light illumination, and modulate the excitation light based on the structured light illumination to form modulated excitation light; The total internal reflection prism is used to reflect the excitation light to the digital micromirror device, receive the modulated excitation light, and reflect the modulated excitation light to the first lens.

3. The device according to claim 1, characterized in that Also includes: Tracking module, where The tracking module includes an infrared illumination source, a ninth lens, an infrared band dichroic mirror, a light trap, a tenth lens and a tracking camera arranged in sequence. The infrared illumination light source is used to generate an infrared band to illuminate all imaging fields of the objective lens. The ninth lens and the tenth lens are used to image all imaging fields supported by the objective lens onto the tracking camera. The infrared band dichroic mirror is used to reflect the excitation light passing through the objective lens to the light trap. The light trap is used to eliminate the excitation light passing through the objective lens. The tracking camera is used to record the imaging images of all imaging fields.

4. The device according to claim 3, characterized in that The tracking module further includes: a processing unit, The processing unit is used to receive the imaging image emitted by the tracking camera, process the imaging image to obtain the position of the target sub-field of view where the tracking target is located, calculate the second voltage signal corresponding to the target sub-field of view, and send the second voltage signal to the scanning module.

5. The device according to claim 1, characterized in that The microlens array is composed of at least one group of sub-lenses with the same optical parameters and the same focus depth; Alternatively, the microlens array with extended depth of field is composed of at least two groups of sub-lenses with the same optical parameters and the same focus depth that focus at different depths and one sub-lens whose imaging depth of field includes the sum of the focus depths of multiple groups.

6. A three-dimensional microscopic imaging method, applied to the three-dimensional microscopic imaging device according to claims 1-4, characterized in that: The following steps are involved: Based on a preset division strategy, the target large field of view is divided into multiple sub-fields of view, and the position and scanning imaging order of each sub-field of view are determined. The analog voltage signal corresponding to the position of each sub-field of view is calculated, and the three-dimensional point spread function of each sub-field of view is pre-collected in a direction perpendicular to the sub-field of view; Generate a voltage signal corresponding to any sub-field of view, and apply the voltage signal to the scanning galvanometer, so that the scanning galvanometer deflects the excitation light to the corresponding sub-field of view and excites the sample in the corresponding sub-field of view; The fluorescence emitted by the sample is collected by an objective lens, and spatial spectrum information of the fluorescence is relayed to a microlens array at a preset magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the spatial spectrum information to obtain multi-view imaging results of the corresponding sub-field of view, and the multi-view imaging results are recorded by a camera; Traversing all sub-view fields until multi-view imaging results of all sub-view fields are obtained, and dividing the multi-view imaging results of all sub-view fields according to different imaging perspectives to obtain imaging results of all sub-view fields at different imaging perspectives; Using the pre-collected three-dimensional point spread function and deconvolution reconstruction algorithm, the imaging results of all the sub-fields of view at different imaging angles are reconstructed to obtain three-dimensional reconstructed images of all the sub-fields of view, and the three-dimensional reconstructed images of all the sub-fields of view are spliced ​​to obtain a three-dimensional microscopic image of the target large field of view.

7. The method according to claim 6, characterized in that The step of generating an excitation light and a voltage signal for any sub-field of view, wherein the scanning galvanometer deflects the excitation light to the corresponding sub-field of view based on the voltage signal and excites a sample in the corresponding sub-field of view, comprises: The excitation light is reflected to a digital micromirror assembly through a total internal reflection prism, and a preset structured light image is loaded on the digital micromirror assembly to form structured light illumination, so as to modulate the excitation light to obtain modulated excitation light.

8. The method according to claim 6, characterized in that After dividing the multi-view imaging results of all sub-view fields according to different imaging view angles to obtain the imaging results of all sub-view fields at different imaging view angles, the method further includes: The imaging results of each sub-field of view at different imaging angles are processed based on the structured light tomography algorithm to obtain the imaging results of all sub-fields of view at different imaging angles after removing the background signals.

9. The method according to claim 6, characterized in that After traversing all the sub-fields of view until multi-view imaging results of all the sub-fields of view are obtained, the method further includes: The multi-view imaging results of all the sub-fields are grouped according to the focus depth to obtain the grouped multi-view imaging results, and the grouped multi-view imaging results are segmented according to different imaging perspectives to obtain the multi-focus imaging results of all the sub-fields at different imaging perspectives.

10. The method according to claim 6, characterized in that Before generating a voltage signal corresponding to any sub-field of view, the method includes: A tracking module image including the entire objective lens field of view is acquired, a sub-field of view position where a tracking target is located is analyzed based on the tracking module image, and an analog voltage signal of the sub-field of view position is determined.

Images