Three-dimensional microscopic imaging device and its imaging method

Abstract

This application relates to a three-dimensional microscopic imaging device and its imaging method. The device includes an illumination source module that emits excitation light, filters and focuses the excitation light, amplifies the filtered and focused excitation light at a preset magnification, and performs relay processing to obtain a first relayed excitation light. A scanning module deflects the first relayed excitation light based on a first voltage signal, relays the deflected excitation light to the objective lens to obtain a second relayed excitation light, and focuses the second relayed excitation light onto a target sub-field of view to excite the sample in the target sub-field of view. The scanning module collects the fluorescence signal emitted by the sample and introduces the fluorescence signal into the imaging system. An imaging module performs optical spatial spectrum multiplexing imaging based on the spatial spectrum information of the fluorescence signal, and a camera records the optical spatial spectrum multiplexing imaging results. This solves the problem of related technologies being unable to perform rapid three-dimensional microscopic imaging of samples distributed over a large field of view and large volume.

Description

Technical Field

[0001] This application relates to the field of optical microscopy, and in particular to a three-dimensional microscopic imaging device and its imaging method. Background Technology

[0002] Rapid three-dimensional microscopic imaging of organisms distributed in a large three-dimensional space has important scientific significance in biomedical research.

[0003] Microscopes with 3D imaging capabilities, such as two-photon microscopes, confocal microscopes, and light-sheet microscopes, suffer from drawbacks such as complex structures and high costs. Furthermore, these microscopes employ point-by-point or surface-by-surface imaging methods, requiring the sequential traversal of multiple planes along the axis for 3D imaging. This limits their imaging speed to the mechanical inertia of the scanning device, making it difficult to meet the application requirements for rapid 3D imaging.

[0004] Fourier transform light field microscopy enables high-speed three-dimensional imaging, requiring only a single exposure to image a three-dimensional volume. Its working principle involves placing a microlens array (number of microlenses: v≥2) at the spatial spectral plane of the microscopic system to optically multiplex the spatial spectral information, simultaneously recording images of the object space from multiple viewpoints (number of microlenses: v≥2) on the camera target surface. Because the projection relationship between the object and image differs when observing a three-dimensional object from different viewpoints, the images of the same area from different viewpoints exhibit significant differences in the imaging results. This significant difference is due to the depth information of the three-dimensional object. Therefore, post-processing can be used to fuse the imaging results of the object space from different viewpoints, thereby obtaining a three-dimensional image of the object space. Based on the above single exposure and post-processing, Fourier transform light field microscopy can achieve rapid three-dimensional microscopic imaging.

[0005] However, because Fourier transform light field microscopy requires the simultaneous recording of images from multiple viewpoints, its three-dimensional imaging performance places extremely high demands on the camera's imaging target surface (size, number of pixels) and imaging sampling rate. Even with a camera featuring an ultra-large target surface, Fourier transform light field microscopy can only confine the field of view to a relatively small area for three-dimensional imaging, thus hindering its ability to rapidly perform three-dimensional microscopic imaging of biological samples distributed across a large field of view and volume. This problem urgently needs to be addressed. Summary of the Invention

[0006] This application provides a three-dimensional microscopic imaging device and its imaging method to solve the problem that related technologies are difficult to achieve three-dimensional microscopic imaging with a large field of view and large volume.

[0007] A first aspect of this application provides a three-dimensional microscopic imaging device, comprising: an illumination source module, a scanning module, and an imaging module, wherein...

[0008] The lighting source module includes a light source component, a filter, a first lens, a second lens, a third lens, and a fourth lens arranged in sequence. The light source component is used to emit excitation light. The light source component, the filter, and the first lens are used to filter the excitation light to form filtered excitation light. The second lens is used to focus the filtered excitation light to form focused excitation light. The third lens and the fourth lens amplify the focused excitation light at a preset magnification and perform relay processing to obtain a first relayed excitation light.

[0009] The scanning module includes a dichroic mirror, a scanning galvanometer, a fifth lens, a sixth lens, a first reflecting mirror, and an objective lens arranged in sequence. The dichroic mirror reflects the first relayed excitation light to the scanning galvanometer. The scanning galvanometer deflects the first relayed excitation light based on a first voltage signal. The fifth lens, the sixth lens, and the first reflecting mirror relay the deflected excitation light to the objective lens to obtain a second relayed excitation light. The objective lens focuses the second relayed excitation light onto the target sub-field of view, excites the sample in the target sub-field of view, and collects the fluorescence signal emitted by the sample. The fluorescence signal enters the imaging system sequentially through the first reflecting mirror, the sixth lens, the fifth lens, the scanning galvanometer, and the dichroic mirror.

[0010] The imaging module includes a seventh lens, a second mirror, an aperture, an eighth lens, a filter, a microlens array, and a camera arranged in sequence. The seventh lens, the second mirror, the aperture, and the eighth lens are used to relay the spatial spectrum information of the fluorescence signal to the filter. The filter is used to filter the fluorescence signal. The microlens array is used to perform optical spatial spectrum multiplexing imaging based on the spatial spectrum information, and the camera records the optical spatial spectrum multiplexing imaging results.

[0011] Optionally, in some embodiments, the above-described three-dimensional microscopic imaging device further includes:

[0012] A total internal reflection prism and a digital micromirror device are provided, wherein the total internal reflection prism and the digital micromirror device are located between the filter and the first lens, wherein...

[0013] The digital micromirror device is used to load a preset structured light image to form structured light illumination, and to modulate the excitation light based on the structured light illumination to form modulated excitation light;

[0014] The total internal reflection prism is used to reflect the excitation light to the digital micromirror device, receive the modulated excitation light, and reflect the modulated excitation light to the first lens.

[0015] Optionally, in some embodiments, the above-described three-dimensional microscopic imaging device further includes:

[0016] The tracking module includes, in sequence, an infrared illumination source, a ninth lens, an infrared dichroic mirror, an optical trap, a tenth lens, and a tracking camera.

[0017] The infrared illumination source is used to generate infrared illumination for all imaging fields of the objective lens. The ninth lens and the tenth lens are used to image all imaging fields supported by the objective lens onto the tracking camera. The infrared dichroic mirror is used to reflect the excitation light transmitted through the objective lens to the optical trap. The optical trap is used to eliminate the excitation light transmitted through the objective lens. The tracking camera is used to record the imaging images of all imaging fields.

[0018] Optionally, in some embodiments, the tracking module further includes: a processing unit.

[0019] The processing unit is used to receive the imaging image emitted by the tracking camera, process the imaging image to obtain the position of the target sub-field of view where the tracking target is located, calculate the second voltage signal corresponding to the target sub-field of view, and send the second voltage signal to the scanning module.

[0020] Optionally, in some embodiments, the microlens array consists of at least one group of sublenses having the same optical parameters and the same depth of focus;

[0021] Alternatively, a microlens array for extended depth of field consists of at least two sets of sublenses with the same optical parameters and the same depth of focus, each focusing at a different depth, and a sublens whose imaging depth of field includes the sum of multiple depths of focus.

[0022] A second aspect of this application provides a method for three-dimensional microscopic imaging, comprising:

[0023] Based on a preset partitioning strategy, the target large field of view is divided into multiple (n≥2) sub-fields of view, and the position and scanning imaging order of each sub-field of view are determined. The analog voltage signal corresponding to the position of each sub-field of view is calculated, and the three-dimensional point spread function of each sub-field of view is pre-acquired along the direction perpendicular to the sub-field of view.

[0024] A voltage signal corresponding to any sub-field of view is generated and applied to a scanning galvanometer, causing the scanning galvanometer to deflect the excitation light to the corresponding sub-field of view and excite the sample within the corresponding sub-field of view;

[0025] The fluorescence emitted by the sample is collected by the objective lens, and the spatial spectrum information of the fluorescence is relayed to the microlens array at a preset magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the spatial spectrum information to obtain the multi-view imaging results of the corresponding sub-field of view, and the multi-view imaging results are recorded by the camera.

[0026] Traverse all sub-fields of view until the multi-view imaging results of all sub-fields of view are obtained, and then divide the multi-view imaging results of all sub-fields of view according to different imaging angles to obtain the imaging results of all sub-fields of view under different imaging angles.

[0027] Using the pre-acquired 3D point spread function and deconvolution reconstruction algorithm, the imaging results of all sub-fields of view under different imaging angles are reconstructed to obtain 3D reconstructed images of all sub-fields of view, and the 3D reconstructed images of all sub-fields of view are stitched together to obtain a 3D microscopic image of the target large field of view.

[0028] Optionally, in some embodiments, generating the excitation light and the voltage signal of any sub-field of view, and the scanning galvanometer deflecting the excitation light to the corresponding sub-field of view based on the voltage signal, and exciting the sample within the corresponding sub-field of view, includes:

[0029] The excitation light is reflected to the digital micromirror assembly by a total internal reflection prism, and a preset structured light image is loaded onto the digital micromirror assembly to form structured light illumination, so as to modulate the excitation light to obtain modulated excitation light.

[0030] Optionally, in some embodiments, after segmenting the multi-view imaging results of all sub-fields of view according to different imaging angles to obtain the imaging results of all sub-fields of view under different imaging angles, the method further includes:

[0031] The imaging results of each sub-field of view under different imaging angles are processed based on the structured light tomography algorithm to obtain the imaging results of all sub-fields of view under different imaging angles after background signal removal.

[0032] Optionally, in some embodiments, after traversing all sub-fields of view until multi-view imaging results of all sub-fields of view are obtained, the method further includes:

[0033] The multi-view imaging results of all sub-fields of view are grouped according to the focus depth to obtain the grouped multi-view imaging results. The grouped multi-view imaging results are then divided according to different imaging angles to obtain the multi-focus imaging results of all sub-fields of view under different imaging angles.

[0034] Optionally, in some embodiments, before generating the voltage signal corresponding to any sub-field of view, the process includes:

[0035] Acquire an image of the tracking module that includes the entire objective lens field of view, analyze the sub-field of view location of the tracking target based on the tracking module image, and determine the analog voltage signal of the sub-field of view location.

[0036] Therefore, this application has at least the following beneficial effects:

[0037] (1) By introducing a scanning system module, the embodiments of this application reduce the hardware requirements that are difficult to overcome in large-volume three-dimensional microscopic imaging by Fourier light field microscope, and realize rapid three-dimensional microscopic imaging with large volume, high resolution and high sampling rate.

[0038] (2) The embodiments of this application introduce a scanning module to make the imaging area flexibly adjustable according to actual needs, which can realize full field of view scanning imaging, can also realize scanning imaging between any field of view, and can also be used in conjunction with a tracking module to perform three-dimensional tracking imaging of moving targets.

[0039] (3) The embodiments of this application introduce a digital micromirror device in the illumination source module to generate structured light illumination, and further combine it with computational optical tomography algorithm to achieve background-free three-dimensional imaging, retaining only the imaging information within the target volume, which can greatly enhance the imaging contrast of three-dimensional microscopic imaging, and realize background-free large-volume, high-resolution, high-sampling-rate fast three-dimensional microscopic imaging.

[0040] (4) The embodiments of this application make full use of the redundancy of spatial spectrum information in Fourier light field microscopes and propose a (x+1) type (x is the number of microlens groups focusing at different depths, x≥2) extended depth of field microlens array. It can increase the imaging depth of Fourier light field microscope while retaining the original imaging field of view, imaging resolution and imaging speed, and realize large volume, high resolution and high sampling rate fast three-dimensional microscopic imaging with extended depth of field.

[0041] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0042] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 This is a block diagram of a three-dimensional microscopic imaging device provided according to an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of a conventional microlens array according to an embodiment of this application;

[0045] Figure 3This is a schematic diagram of a microlens array for extended depth of field provided according to an embodiment of this application;

[0046] Figure 4 This is a block diagram of a three-dimensional microscopic imaging apparatus according to another embodiment of this application;

[0047] Figure 5 This is a block diagram of a three-dimensional microscopic imaging apparatus according to yet another embodiment of this application;

[0048] Figure 6 This is a schematic diagram illustrating the principle of full-field scanning according to an embodiment of this application;

[0049] Figure 7 This is a schematic diagram illustrating the principle of tracking scanning according to an embodiment of this application;

[0050] Figure 8 This is a schematic flowchart of a three-dimensional microscopic imaging method provided according to an embodiment of this application;

[0051] Figure 9 This is a schematic flowchart of a three-dimensional microscopic imaging method according to an embodiment of this application;

[0052] Figure 10 This is a schematic flowchart of a three-dimensional microscopic imaging method according to another embodiment of this application;

[0053] Figure 11 This is a flowchart illustrating a three-dimensional microscopic imaging method according to yet another embodiment of this application. Detailed Implementation

[0054] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0055] Before introducing the three-dimensional microscopic imaging device and imaging method of the embodiments of this application, let's first introduce the working principle of the Fourier light field microscope.

[0056] Fourier transform light field microscopy requires the simultaneous recording of images from multiple viewpoints (number: v≥2) for 3D reconstruction. Therefore, for the same object space, Fourier transform light field microscopy needs to record v times the information of a conventional wide-field microscope in a single exposure. Simultaneously, at a lateral resolution R... x Once determined, the axial resolution R of the reconstructed 3D image is... z It is negatively correlated with the tangent of the maximum viewing angle θ during imaging, i.e., R z =Rx / tg(θ), that is, the larger the maximum viewing angle θ, the smaller the value of the axial resolution and the stronger the axial resolution capability.

[0057] As optical principles show, the angle of light propagation in space corresponds to its spatial frequency, and a higher numerical aperture of an objective lens allows it to collect light with higher spatial frequencies. Therefore, to achieve stronger axial resolution, among objectives supporting the same imaging field of view, objectives with higher numerical apertures should be prioritized, acquiring more images in the high-frequency region of their spatial spectrum, i.e., a higher value for v. Thus, to achieve high spatial resolution three-dimensional imaging, Fourier light field microscopy requires a very large camera target surface to match it.

[0058] Meanwhile, in order to image the fine structures within organisms, the optical system of a Fourier light field microscope must not only have high spatial resolution, but also provide sufficient system magnification M to ensure that the image formed has a sampling rate SR higher than the Nyquist sampling rate, i.e., SR = M·R. x / d pixel >2, where d pixel This refers to the camera's pixel size. If the imaging field of view of a Fourier light field microscope is FOV, then the minimum number of pixels required on the camera is [number missing]. It is evident that as the field of view (FOV) increases, the number of pixels required by the camera increases dramatically. This places high demands on the imaging equipment of Fourier transform light field microscopes, including large target area, high pixel count, and small pixel size.

[0059] For example, if you need to achieve a magnification of M=5 within a 3mm field of view and R... x Achieving imaging at a lateral resolution of 2μm across 9 viewing angles (v=9) requires at least 9×3000×3000 pixels and a target area of ​​45mm×45mm, even with 100% target surface utilization and a minimum sampling rate of SR=2. This requirement far exceeds the specifications achievable by typical commercial cameras. Furthermore, in typical designs, acquiring images at higher spatial frequencies is prioritized to improve axial resolution. Therefore, the camera's target surface would have significant redundancy, and the actual required parameters would be higher than the assumed values ​​above.

[0060] It is evident that the extremely high requirements for hardware limit the current Fourier light field microscope to small-volume imaging within a small field of view, making it difficult to extend to large-volume, high-resolution, high-sampling-rate rapid three-dimensional imaging.

[0061] To address the aforementioned issues, this application provides a three-dimensional microscopic imaging device and method. This involves determining multiple sub-fields of view (n≥2) within a large field of view, acquiring the three-dimensional point spread function of each sub-field of view, receiving excitation light and voltage signals from any sub-field of view, and, based on the voltage signals, exciting a sample within the corresponding sub-field of view with excitation light. The fluorescence emitted by the sample is collected through an objective lens, and the spatial spectrum information of the fluorescence is relayed to a microlens array at a preset magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the spatial spectrum information to obtain multi-view imaging results for the corresponding sub-field of view, which are then recorded by a camera. Furthermore, the imaging area is flexibly adjustable according to the actual needs of the sample being observed, enabling full-field scanning imaging, rapid scanning imaging between any fields of view, and three-dimensional tracking imaging of moving targets using a tracking module. The process iterates through all target sub-fields of view until multi-view imaging results for all sub-fields of view are obtained. These results are then segmented according to different imaging angles to obtain imaging results for each sub-field of view under different imaging angles. Using a 3D point spread function and a pre-defined reconstruction algorithm, the imaging results of all sub-fields of view under different imaging angles are reconstructed to obtain 3D reconstructed images of all sub-fields of view. These 3D reconstructed images are then stitched together or aligned to obtain a 3D microscopic image of the target. This solves the problem that related technologies can only perform small-volume imaging within a small field of view, making it difficult to achieve large-volume, high-resolution, and rapid 3D imaging. It enables the acquisition of large-volume, high-resolution, and high-sampling-rate 3D imaging results.

[0062] Specifically, Figure 1 This is a block diagram of a three-dimensional microscopic imaging device provided in an embodiment of this application.

[0063] like Figure 1 As shown, the three-dimensional microscopic imaging device 10 includes: an illumination source module 100, a scanning module 200, and an imaging module 300.

[0064] The lighting source module 100 includes a light source assembly 1, a filter 2, a first lens 3, a second lens 4, a third lens 5, and a fourth lens 6 arranged in sequence. The light source assembly 1 is used to emit excitation light. The light source assembly 1, the filter 2, and the first lens 3 are used to filter the excitation light to form filtered excitation light. The second lens 4 is used to focus the filtered excitation light to form focused excitation light. The third lens 5 and the fourth lens 6 amplify the focused excitation light at a preset magnification and perform relay processing to obtain the first relayed excitation light.

[0065] Among them, the light source component 1 can be a collimated LED lighting source, and the second lens can be an adjustable focus lens.

[0066] Specifically, the light source assembly 1 emits excitation light and passes through the excitation end filter 2. Subsequently, the excitation light passes through the first lens 3 and the second lens 4. The collimating lens in the light source assembly 1 and the first lens 3 form a 4f system. The second lens 4 is located on the back focal plane of the first lens 3. The focusing depth of the excitation light behind the objective lens can be adjusted by the second lens 4.

[0067] Subsequently, another 4f relay system is formed by the third lens 5 and the fourth lens 6 to relay the excitation light to the scanning module 200 at a specific magnification. Let the focal length of the third lens 5 be |f5| and the focal length of the fourth lens 6 be |f6|, then the magnification M... 5-6 It can be represented as M 5-6 =|f6| / |f5|.

[0068] The scanning module 200 includes a dichroic mirror 7, a scanning galvanometer 8, a fifth lens 9, a sixth lens 10, a first reflecting mirror 11, and an objective lens 12 arranged sequentially along the optical path of the scanning module 200. The dichroic mirror 7 is used to reflect the excitation light after the first relay to the scanning galvanometer 8. The scanning galvanometer 8 deflects the excitation light after the first relay based on the first voltage signal. The fifth lens 9, the sixth lens 10, and the first reflecting mirror 11 are used to relay the deflected excitation light to the objective lens 12 to obtain the excitation light after the second relay. The objective lens 12 is used to focus the excitation light after the second relay onto the target sub-field of view, excite the sample in the target sub-field of view, and collect the fluorescence signal emitted by the sample. The fluorescence signal enters the imaging module 300 sequentially through the first reflecting mirror 11, the sixth lens 10, the fifth lens 9, the scanning galvanometer 8, and the dichroic mirror 7.

[0069] Among them, the scanning galvanometer 8 can be an XY dual-axis scanning galvanometer, and the first voltage signal can be the voltage signal corresponding to the target sub-market.

[0070] Specifically, the dichroic mirror 7 can be a long-pass dichroic mirror, which reflects the excitation light relayed by the illumination module 100 to the XY dual-axis scanning galvanometer 8. After receiving the voltage signal given by the digital board, the XY dual-axis scanning galvanometer 8 generates a mechanical deflection angle. The above excitation light is at an angle Deflection propagation direction.

[0071] The fifth lens 9 and the sixth lens 10 form a 4f relay system, relaying the excitation light from the center plane of the scanning galvanometer 8 to the rear focal plane of the objective lens 12. The first reflecting mirror 11 is used to change the propagation direction of the beam, making the propagation direction of the excitation light vertically downward, so as to facilitate the mounting of biological samples. The objective lens 12 focuses the excitation light, after deflecting its propagation direction, into the target sub-field of view located at the rear focal plane of the objective lens, forming uniform illumination to excite the sample in the sub-field of view. Let the focal length of the fifth lens 9 be |f9| and the focal length of the sixth lens 10 be |f 10|, The focal length of objective lens 12 is |f 12 |, then the distance between the center of the sub-field of view and the optical axis is

[0072] Subsequently, objective lens 12 collects the fluorescence signal emitted by the sample in this sub-field of view, and transmits it in reverse through first reflecting mirror 11, sixth lens 10, and fifth lens 9 to XY dual-axis scanning galvanometer 8. The direction of propagation of this fluorescence signal at this time is... After passing through the XY dual-axis scanning galvanometer 8 in the reverse direction, its deflection angle is canceled, and it becomes a fluorescence signal that propagates along the optical axis. The fluorescence signal propagating along the optical axis passes through the dichroic mirror 7 and enters the imaging module 300.

[0073] The imaging module 300 includes a seventh lens 13, a second mirror 14, an aperture 15, an eighth lens 16, a filter 17, a microlens array 18, and a camera 19 arranged sequentially along the optical path of the imaging module.

[0074] Among them, the seventh lens 13, the second mirror 14, the aperture 15 and the eighth lens 16 are used to relay the spatial spectrum information of the fluorescence signal to the filter 17. The filter 17 is used to filter the fluorescence signal. The microlens array 18 is used to perform optical spatial spectrum multiplexing imaging based on the spatial spectrum information, and the camera 19 records the optical spatial spectrum multiplexing imaging results.

[0075] Specifically, the fluorescence signal collected in the scanning module 200 is focused by the seventh lens 13 and reflected by the second mirror 14, and imaged at the aperture 15 located at the rear focal plane of the seventh lens 13. The aperture 15 is used to limit the sub-field of view size to avoid image overlap between different viewing angles caused by an excessively large sub-field of view size during imaging. Subsequently, the eighth lens 16 and the seventh lens 13 form a 4f relay system to relay the spatial spectrum information of the fluorescence signal to the collection end filter 17. After passing through the collection end filter 17, the spatial spectrum information of the fluorescence signal is filtered to remove stray light other than the fluorescence signal, and reaches the microlens array 18 located at the rear focal plane of the eighth lens 16. The microlens array 18 can be a general microlens array or a microlens array that extends the depth of field. The microlens array 18 performs optical spatial spectrum multiplexing imaging on the spatial spectrum information of the fluorescence signal, and the imaging result is recorded by the camera 19 located at the rear focal plane of the microlens array 18.

[0076] Optionally, in some embodiments, the microlens array consists of at least one set of sub-lenses with the same optical parameters and the same depth of focus; or, the extended depth-of-field microlens array consists of at least two sets of sub-lenses with the same optical parameters and the same depth of focus that focus at different depths, and a sub-lens whose imaging depth of field includes the sum of multiple sets of depth of focus. It should be noted that in the embodiments of this application, the microlens array 18 can be selected as a conventional microlens array or an extended depth-of-field microlens array as needed. The microlens array 18 can be composed of multiple sub-lenses (number: v≥2) to achieve imaging from multiple viewpoints.

[0077] like Figure 2 As shown, in a typical commercially available or self-manufactured microlens array, all sublenses have the same optical parameters and focusing depth. That is, the three-dimensional point spread function of all sublenses intersects at the same depth in the axial direction. When using a typical microlens array for imaging, the imaging depth of a large-volume three-dimensional microscopic imaging device is the same as the imaging depth of a single sublens, both being DOF.

[0078] like Figure 3 As shown, the extended depth-of-field microlens array divides all sub-lenses in the high-frequency range into (x+1) groups. Within each x groups, the sub-lenses have the same optical parameters and depth of focus, while the depth of focus differs between sub-lenses in different groups. Figure 3 As shown, the imaging depth of the first group of microlenses is DOF1, and the imaging depth of the second group of microlenses is DOF2. The difference in focusing depth between the two groups of microlenses is ΔDOF, and there is... Total image depth of field is Therefore, an extended depth of field is achieved along the axial direction. Simultaneously, the extended depth-of-field microlens array places a lens with different optical parameters from the aforementioned x groups of lenses at the center of the spectrum; its imaging depth of field DOF0 includes the depth of field of the aforementioned x groups of lenses, i.e. And the focusing depth is at the center of the focusing depth of the above x groups of lenses.

[0079] Those skilled in the art should note that the principle of the 4f relay system indicated in the embodiments of this application is as follows: For example, the third lens 5 and the fourth lens 6 are placed sequentially along the optical path propagation direction, with their optical centers coinciding with the optical axis. The third lens 5 and the fourth lens 6 form a 4f relay system. The focal length of the third lens 5 is |f1| and the focal length of the fourth lens 6 is |f2|. Then, the relay system can relay an object located at the front focal plane of the third lens 5 (a distance of |f1| from the principal surface of lens 1 along the optical path propagation direction) to the rear focal plane of the fourth lens 6 (a distance of |f2| from the principal surface of lens 2 along the optical path propagation direction). The distance between the third lens 5 and the fourth lens 6 is |f1| + |f2|.

[0080] Therefore, the embodiments of this application make full use of the redundancy of spatial spectral information in Fourier light field microscopy and propose a (x+1) type extended depth-of-field microlens array, which can increase the imaging depth of Fourier light field microscopy while retaining the original imaging field of view, imaging resolution and imaging speed, and realize large-volume, high-resolution, high-sampling-rate rapid three-dimensional microscopic imaging with extended depth of field.

[0081] Optionally, such as Figure 4 As shown, in some embodiments, the three-dimensional microscopic imaging device 10 described above further includes: a total internal reflection prism 20 and a digital micromirror device 21, wherein the total internal reflection prism 20 and the digital micromirror device 21 are located between the filter 2 and the first lens 3.

[0082] The digital micromirror device 21 is used to load a preset structured light image to form structured light illumination, and to modulate the excitation light based on the structured light illumination to form modulated excitation light.

[0083] The total internal reflection prism 20 is used to reflect the excitation light to the digital micromirror device 21, receive the modulated excitation light, and reflect the modulated excitation light to the first lens 3.

[0084] Specifically, the light source assembly 1 emits excitation light, which passes through the excitation end filter 2. After being reflected by the total internal reflection prism 20, the excitation light reaches the target surface of the digital micromirror device 21. A structured light pattern is pre-loaded on the target surface of the digital micromirror device 21, and the excitation light is modulated by the loaded pattern to generate structured light illumination. Subsequently, the modulated excitation light is reflected back through the total internal reflection prism 20. Then, the modulated excitation light passes through a 4f relay system composed of the first lens 3 and the third lens 5, relaying to the front focal plane of the fourth lens 6, where it forms an image of the pattern loaded on the target surface of the digital micromirror device 21. The second lens 4 is located at the rear focal plane of the first lens 3 and is used to adjust the depth of focus. Subsequently, a relay system composed of the third lens 5 and the fourth lens 6 accurately relays the excitation light to the scanning module 200 at a specific magnification. Let the focal length of the third lens 5 be |f7| and the focal length of the fourth lens 6 be |f8|, then the magnification M... 7-8 It can be represented as

[0085] M 7-8 =|f8| / |f7|.

[0086] The scanning module 200 includes a dichroic mirror 7, an XY dual-axis scanning galvanometer 8, a fifth lens 9, a sixth lens 10, a first reflecting mirror 11, and an objective lens 12, all arranged sequentially along the optical path of the scanning module 200. The dichroic mirror 7 is a long-pass dichroic mirror and reflects the excitation light relayed by the illumination source module 100 to the XY dual-axis scanning galvanometer 8. Upon receiving a voltage signal from the digital board, the XY dual-axis scanning galvanometer 8 generates a mechanical deflection angle. The above excitation light is at an angle The propagation direction is deflected. The fifth lens 9 and the sixth lens 10 form a 4f relay system, relaying the excitation light from the central plane of the scanning galvanometer to the rear focal plane of the objective lens 12. The first reflecting mirror 11 is used to change the propagation direction of the beam, making it vertically downward, facilitating the mounting of biological samples. The objective lens 12 focuses the excitation light, after deflection, onto the target sub-field of view located at the rear focal plane of the objective lens, forming structured light illumination modulated by the pattern loaded on the target surface of the digital micromirror device 21 in the target sub-field of view, thereby exciting the sample in that sub-field of view. Let the focal length of the fifth lens 9 be |f|. 11 The sixth lens has a focal length of |f. 12 |, the objective lens distance is |f 14 |, then the distance between the center of the target subfield of view and the optical axis is

[0087] Subsequently, objective lens 12 collects the fluorescence signal emitted by the sample in this sub-field of view, and transmits it in reverse through first reflecting mirror 11, sixth lens 10, and fifth lens 9 to XY dual-axis scanning galvanometer 8. The direction of propagation of this fluorescence signal at this time is... After passing through the XY dual-axis scanning galvanometer 8 in the reverse direction, its deflection angle is canceled, and it becomes a fluorescence signal that propagates along the optical axis. The fluorescence signal propagating along the optical axis passes through the dichroic mirror 7 and enters the imaging module 300.

[0088] The imaging module 300 includes a seventh lens 13, a second mirror 14, an aperture 15, an eighth lens 16, a filter 17, a microlens array 18, and a camera 19 arranged sequentially along the optical path of the imaging module. The fluorescence signal collected in the scanning module 200 is focused by the seventh lens 13 and reflected by the second mirror 14, and imaged at the aperture 15 located at the rear focal plane of the seventh lens 13. The aperture 15 is used to limit the sub-field size to avoid image overlap between different viewing angles caused by an excessively large sub-field size during imaging. Subsequently, the eighth lens 16 and the seventh lens 13 form a 4f relay system to relay the spatial spectrum information of the fluorescence signal to the collection-end filter 17. After passing through the collection-end filter 17, the spatial spectrum information of the fluorescence signal is filtered to remove stray light other than the fluorescence signal, and reaches the microlens array 18 located at the rear focal plane of the eighth lens 16. The microlens array 18 can be a conventional microlens array or a microlens array with extended depth of field. The microlens array 18 performs optical spatial spectrum multiplexing imaging on the spatial spectrum information of the fluorescence signal, and the imaging result is recorded by a camera 19 located at the rear focal plane of the microlens array 18.

[0089] Therefore, the embodiments of this application introduce a digital micromirror device in the illumination source module to generate structured light illumination, and further combine it with computational optical tomography algorithm to achieve background-free three-dimensional imaging, retaining only the imaging information within the target volume, which can greatly enhance the imaging contrast of three-dimensional microscopic imaging, and realize background-free large-volume, high-resolution, high-sampling-rate fast three-dimensional microscopic imaging.

[0090] Optionally, such as Figure 5 As shown, in some embodiments, the three-dimensional microscopic imaging device 10 described above further includes a tracking module 400.

[0091] The tracking module 400 includes an infrared illumination source 22, a ninth lens 23, an infrared dichroic mirror 24, an optical trap 25, a tenth lens 26, and a tracking camera 27 arranged in sequence.

[0092] Infrared illumination source 22 is used to generate all imaging fields of view of infrared illumination objective lens 12. Ninth lens 23 and tenth lens 26 constitute a 4f imaging system to image all imaging fields of view supported by objective lens 12 onto tracking camera 27. Infrared dichroic mirror 24 is used to reflect the excitation light transmitted through objective lens 12 to optical trap 24. Optical trap 24 is used to eliminate the excitation light transmitted through objective lens 12 to eliminate the interference of the excitation light transmitted through objective lens 12 on infrared illumination imaging. Tracking camera 27 is used to record imaging images of all imaging fields of view.

[0093] Optionally, in some embodiments, the tracking module 400 further includes a processing unit 28.

[0094] The processing unit 28 can be a computer. The processing unit 28 is used to receive the imaging image emitted by the tracking camera 27, process the imaging image to obtain the position of the target sub-field of view where the tracking target is located, calculate the second voltage signal corresponding to the target sub-field of view, and send the second voltage signal to the scanning module 200.

[0095] Specifically, after the tracking camera 27 captures an image, it transmits the image to the processing unit 28 for real-time image processing. The processing unit analyzes the sub-field of view position of the tracking target at this moment and calculates the voltage signal corresponding to the sub-field of view. Then, the processing unit 28 sends the second voltage signal to the scanning module through the digital board.

[0096] It should be noted that the scanning module 200 provided in this application embodiment can provide flexible scanning methods, such as full field of view scanning, arbitrary field-to-field scanning, and tracking scanning. Among them, full field of view scanning means traversing the entire field of view supported by the scanning objective lens with a specific scanning trajectory and imaging; arbitrary field-to-field scanning means arbitrarily selecting one or more fields of view (number: v≥2) within the field of view supported by the objective lens for scanning and imaging; tracking scanning is to cooperate with the tracking module 400 to scan in a continuously changing field of view, that is, within the field of view supported by the objective lens, the tracking module 400 obtains the field of view position (ROI) of the tracking target, which can change with time, and the scanning module 200 tracks the position of the target in real time.

[0097] like Figure 6 As shown, d FOV d is the field size of the entire field of view supported by the objective lens. FOV_sub Let be the field of view size of a single sub-field of view, where d FOV_sub The size should be the maximum imaging size that the camera target surface can support in a single exposure, provided that the imaging sampling rate and the number of imaging angles v are sufficient. Δd is the interval between sub-fields of view.

[0098] In full-field scanning imaging, to obtain effective stitching results, there needs to be a certain proportion of overlap between adjacent sub-fields of view, i.e., d. FOV_sub -Δd. The overlap ratio can be designed independently, for example, 10%, i.e.

[0099] During full-field scanning imaging, the degree of change in the mechanical angle of the XY dual-axis scanning galvanometer 8 affects its response time. Therefore, to avoid large mechanical angle changes in the XY dual-axis scanning galvanometer during continuous scanning, measures such as... Figure 6 The scanning direction is indicated by the middle arrow. The XY dual-axis scanning galvanometer starts from the X2Y2 field of view and goes through X2Y2→X3Y3, X3Y3→X2Y3, X2Y3→X1Y3, X1Y3→X1Y2, X1Y2→X1Y1, X1Y1→X2Y1, X2Y1→X3Y1, X3Y2→X3Y2, X3Y2→X2Y2, completing one cycle of scanning.

[0100] When performing tracking scan imaging, such as Figure 7 As shown, d FOV The field of view is the total field of view supported by objective lens 12, and the ROI is the position of the tracked target at different times. FOV When moving freely within the device, the tracking module 400 and the scanning module 200 can maintain real-time tracking and real-time 3D imaging of the target.

[0101] Therefore, the embodiments of this application introduce a scanning module to make the imaging area flexibly adjustable according to actual needs, which can realize full-field scanning imaging, scanning imaging between any field of view, and can also be used in conjunction with a tracking module to perform three-dimensional tracking imaging of moving targets.

[0102] The three-dimensional microscopic imaging device proposed in this application emits excitation light through an illumination source module, filters and focuses the excitation light, amplifies the filtered and focused excitation light at a preset magnification, and relays it to obtain a first relayed excitation light. A scanning module deflects the first relayed excitation light based on a first voltage signal, relays the deflected excitation light to the objective lens to obtain a second relayed excitation light, and focuses the second relayed excitation light onto the target sub-field of view, exciting the sample in the target sub-field of view. The device collects the fluorescence signal emitted by the sample and introduces the fluorescence signal into the imaging system. An imaging module performs optical spatial spectrum multiplexing imaging based on the spatial spectrum information of the fluorescence signal, and a camera records the optical spatial spectrum multiplexing imaging result. This solves the problem of related technologies struggling to achieve large field-of-view and large-volume three-dimensional microscopic imaging, enabling high-resolution, high-sampling-rate, large-field-of-view rapid three-dimensional imaging.

[0103] Next, the three-dimensional microscopic imaging method proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0104] Figure 8 This is a flowchart illustrating a three-dimensional microscopic imaging method according to an embodiment of this application.

[0105] like Figure 8 As shown, the three-dimensional microscopic imaging method includes the following steps:

[0106] In step S101, based on a preset partitioning strategy, the target large field of view is divided into multiple sub-fields of view (number: n≥2), and the position and scanning imaging order of each sub-field of view are determined. The analog voltage signal corresponding to the position of each sub-field of view is calculated, and the three-dimensional point spread function of each sub-field of view is pre-acquired along the direction perpendicular to the sub-field of view.

[0107] The target large field of view refers to either the maximum field of view supported by the objective lens or any sub-field of view within the maximum field of view supported by the objective lens. The preset division strategy is to determine the imaging resolution Rx and the sampling factor, and then determine the maximum number of pixels in each sub-field of view based on the imaging resolution Rx and the sampling factor. Finally, the maximum size of each sub-field of view is determined based on the maximum number of pixels in each sub-field of view and camera parameters, and the sub-field of view is divided according to the maximum size of the sub-field of view. Specifically, if the imaging resolution is determined to be Rx, and a double sampling factor is required, then the maximum size of each pixel in the sub-field of view is Rx / 2. The maximum size of the sub-field of view is the number of pixels occupied by a sub-lens on the microlens array on the camera. For example, if each sub-lens corresponds to 1200*1200 pixels, then the maximum size of the sub-field of view is 1200*Rx / 2. When Rx = 3μm, the maximum size of the sub-field of view is approximately 1.8mm*1.8mm. The number of sub-fields of view in this embodiment can be determined by the maximum field of view size supported by the objective lens. For example, a 4x objective lens can support a 5mm*5mm field of view. The number of target sub-fields of view (9) is obtained by dividing the maximum field of view size supported by the objective lens by the maximum size of the sub-field of view and rounding up.

[0108] Specifically, in this embodiment, the input voltage of the scanning galvanometer control component can be modulated to determine the positions of each sub-field of view (number n) for scanning imaging within a large field of view. The scanning sequence of each sub-field of view is then encoded, for example, sub-field of view 1, sub-field of view 2, ..., sub-field of view n. Within the field of view supported by the objective lens, the excitation light direction is deflected at different angles by the scanning galvanometer, causing the excitation light to be focused at the front focal plane of the objective lens onto the determined sub-fields of view. Within each determined sub-field of view, the fluorescent sphere is moved in a direction perpendicular to the field of view with a preset step size (e.g., 1 μm). After each movement, an image at that depth is acquired until the entire depth of field (e.g., 200 μm) is traversed, completing the acquisition of the three-dimensional point spread function of each viewpoint within that sub-field of view.

[0109] It should be noted that before passing through the scanning galvanometer, the center of the excitation light propagates along the optical axis of the system. After being deflected by the scanning galvanometer, the propagation angle of the center of the excitation light changes. The mechanical rotation angle of the scanning galvanometer twice as much as, that is The mechanical rotation angle generated by the scanning galvanometer It is proportional to the value of the voltage applied to the scanning galvanometer.

[0110] The equivalent focal length of the optical system that the excitation light passes through before reaching the sub-field of view is f. o The central propagation angle is The vector distance between the center of the sub-field of view focused behind the objective lens and the center of the objective lens's field of view is...

[0111] The position and number of pre-coded sub-fields of view within a large field of view can be adjusted as needed, and may include full field of view scanning or scanning between arbitrary fields of view. Full field of view scanning means traversing and imaging the entire field of view supported by the scanning objective with a specific scanning trajectory, while scanning between arbitrary fields of view means arbitrarily selecting one or more fields of view (number: n≥2) within the field of view supported by the objective for scanning and imaging.

[0112] When acquiring the 3D point spread function of each viewpoint in each sub-field of view, the size of the selected fluorescent microsphere should be smaller than the lateral resolution R of the system. x The preset step size for moving along the direction perpendicular to the field of view should be less than the axial resolution R of the system. z .

[0113] In step S102, a voltage signal corresponding to any sub-field of view is generated and applied to the scanning galvanometer, so that the scanning galvanometer deflects the excitation light to the corresponding sub-field of view and excites the sample in the corresponding sub-field of view.

[0114] Specifically, an excitation light is generated by an illumination source, and at the same time, the control component of the scanning galvanometer receives a voltage signal that causes it to scan to any sub-field of view and generates a deflection at a corresponding angle. After the scanning galvanometer deflects the excitation light, it is projected onto the sub-field of view through the objective lens to excite the sample in the sub-field of view, and at the same time, a trigger signal is given to the camera.

[0115] It should be noted that the mechanical deflection of the scanning galvanometer and the exposure of the camera are synchronized through analog / digital signals. After the scanning galvanometer undergoes a mechanical deflection, the camera simultaneously receives a trigger signal to start exposure and begins exposure. The camera's exposure time is less than the time interval between two adjacent trigger signals. The camera completes the exposure and data transmission before the scanning galvanometer undergoes the next mechanical deflection.

[0116] In step S103, the fluorescence emitted by the sample is collected through the objective lens, and the spatial spectrum information of the fluorescence is relayed to the microlens array at a preset magnification. The microlens array performs optical spatial multiplexing imaging on the spatial spectrum information to obtain multi-view imaging results of the corresponding sub-field of view, and the multi-view imaging results are recorded by the camera.

[0117] The multi-view imaging result is determined by the number of microlenses; the number of microlenses corresponds to the number of viewing angles.

[0118] Specifically, the objective lens collects the fluorescence emitted by the sample within the sub-field of view in step S102. This fluorescence is reversed by the scanning galvanometer to cancel the deflection angle generated by the scanning and propagates along the optical axis of the system. Subsequently, the spatial spectrum information of the fluorescence is relayed to the microlens array at a specific magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the above spatial spectrum information. The camera placed at the back focal plane of the microlens array starts exposure after receiving the trigger signal in step S102, records the multi-view (number of viewpoints is v) imaging results of the sub-field of view, stops exposure after the set exposure time is reached, and transmits the image to the computer.

[0119] It should be noted that the microlens array in this embodiment consists of multiple sub-lenses (number: v≥2) to achieve imaging from multiple perspectives. The microlens array can be selected according to the requirements, either from ordinary commercial or self-designed and manufactured microlenses. All sub-lenses have the same optical parameters and focusing depth, that is, the three-dimensional point spread function of all sub-lenses intersects at the same depth in the axial direction.

[0120] In step S104, all sub-fields of view are traversed until the multi-view imaging results of all sub-fields of view are obtained, and the multi-view imaging results of all sub-fields of view are divided according to different imaging angles to obtain the imaging results of all sub-fields of view under different imaging angles.

[0121] Specifically, in this embodiment, all sub-fields of view can be traversed sequentially according to the scanning order encoded in step S101, and steps S102 to S103 can be repeated. After the traversal is completed, one cycle of acquisition is completed, and images of each sub-field of view (the number is n, the same as the number of sub-fields of view) are obtained. Subsequently, the imaging results of each sub-field of view are segmented according to different imaging angles to obtain the imaging results of each sub-field of view from each angle (the number is n×v).

[0122] In step S105, the imaging results of all sub-fields under different imaging angles are reconstructed using the pre-acquired three-dimensional point spread function and deconvolution reconstruction algorithm to obtain three-dimensional reconstructed images of all sub-fields, and the three-dimensional reconstructed images of all sub-fields are stitched together to obtain a three-dimensional microscopic image of the target large field of view.

[0123] Specifically, this embodiment requires 3D reconstruction of the image of each sub-field of view. Specifically, using the 3D point spread function of the sub-field of view acquired in step S101 and a Fourier light field reconstruction algorithm based on a multi-view iterative deconvolution method, the imaging results of each viewpoint (number v) are reconstructed into a 3D image (number n). After completing the 3D reconstruction of all sub-fields of view (number n), the obtained 3D reconstruction results of all sub-fields of view are then stitched together according to their 3D positions to obtain a large field of view, high resolution, and high sampling rate 3D microscopic image (number n).

[0124] It should be noted that if the position and number of sub-fields of view are adjusted according to actual needs in step S101, such as scanning and imaging between any discontinuous sub-fields of view, then the imaging sub-fields of view do not need to be stitched together, and the stitching step can be skipped as needed.

[0125] In summary, combining Figure 9 As shown, this embodiment determines the position of each sub-field of view for scanning imaging within a large field of view by modulating the input voltage of the scanning galvanometer control component. The scanning sequence of each sub-field of view is then encoded. Within the field of view supported by the objective lens, the excitation light is deflected at different angles by the scanning galvanometer, causing the excitation light to be focused at the front focal plane of the objective lens onto the determined sub-fields of view. Within each determined sub-field of view, a fluorescent sphere is moved at a preset step size along a direction perpendicular to the field of view. After each movement, an image at that depth is acquired until the entire depth of field is traversed, completing the acquisition of the three-dimensional point spread function for each viewpoint within that sub-field of view. Excitation light is generated by the illumination source, and simultaneously, the scanning galvanometer control component receives a voltage signal that causes it to scan to a certain sub-field of view, generating a preset angle of deflection. After the scanning galvanometer deflects the excitation light, it is projected onto the sub-field of view through the objective lens, exciting the sample within that sub-field of view and simultaneously sending a trigger signal to the camera.

[0126] The objective lens collects the fluorescence emitted by the sample within the sub-field of view. This fluorescence is then reversed by the scanning galvanometer to cancel out the deflection angle caused by the scanning and propagates along the system's optical axis. Subsequently, the spatial spectrum information of this fluorescence is relayed to the microlens array at a specific magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the aforementioned spatial spectrum information. A camera placed at the rear focal plane of the microlens array begins exposure after a trigger signal, recording the multi-view (number of viewpoints is v) imaging results of the sub-field of view. After the set exposure time is reached, exposure stops and the image is transmitted to the computer. The preset sub-fields of view are traversed sequentially according to the coded scanning order. After the traversal is completed, one cycle of acquisition is finished, and images of each sub-field of view are obtained.

[0127] Subsequently, the imaging results of each sub-field of view are segmented according to different imaging angles to obtain the imaging results of each sub-field of view from each perspective. Three-dimensional reconstruction is then performed on the image of each sub-field of view. Using the three-dimensional point spread function of the corresponding sub-field of view and a Fourier light field reconstruction algorithm based on a multi-view iterative deconvolution method, the imaging results from each perspective are reconstructed into a three-dimensional image. After completing the three-dimensional reconstruction of all sub-fields of view, the obtained three-dimensional reconstruction results of all sub-fields of view are stitched together according to their three-dimensional positions to obtain a large-field-of-view, high-resolution, high-sampling-rate three-dimensional microscopic image. This solves the problem that related technologies can only perform small-volume imaging within a small field of view, making it difficult to achieve large-volume, high-resolution, and rapid three-dimensional imaging.

[0128] Optionally, in some embodiments, generating an excitation light and a voltage signal for any sub-field of view, and a scanning galvanometer deflecting the excitation light to the corresponding sub-field of view based on the voltage signal, and exciting the sample in the corresponding sub-field of view, including: reflecting the excitation light to the digital micromirror assembly through a total internal reflection prism, and using the digital micromirror assembly to load a preset structured light image to form structured light illumination, so as to modulate the excitation light to obtain modulated excitation light.

[0129] Furthermore, in some embodiments, after segmenting the multi-view imaging results of all sub-fields of view according to different imaging views to obtain the imaging results of all sub-fields of view under different imaging views, the method further includes: processing the imaging results of each sub-field of view under different imaging views based on the structured light tomography algorithm to obtain the imaging results of all sub-fields of view under different imaging views after background signal removal.

[0130] Specifically, such as Figure 10 As shown, the embodiments of this application can also achieve large-volume, high-resolution rapid three-dimensional microscopic imaging with background removal.

[0131] Step S301: By modulating the input voltage of the scanning galvanometer control component, the positions of each sub-field of view (number n) for scanning imaging within the large field of view are determined. The scanning sequence of each sub-field of view is then encoded (e.g., sub-field of view 1 → sub-field of view 2 → … → sub-field of view n). Within the field of view supported by the objective lens, the excitation light direction is deflected at different angles by the scanning galvanometer, causing the excitation light to be focused at the front focal plane of the objective lens onto the determined sub-field of view. Within each determined sub-field of view, the fluorescent sphere is moved in a direction perpendicular to the field of view with a preset step size (e.g., 1 μm). After each movement, an image at that depth is acquired until the entire depth of field (e.g., 200 μm) has been traversed, completing the acquisition of the three-dimensional point spread function for each viewpoint within that sub-field of view.

[0132] Step S302: Excitation light is generated by an illumination source and then reflected onto a digital micromirror device (DMM) via a total internal reflection prism. A structured light pattern loaded onto the DMM modulates the excitation light to generate structured light illumination. Simultaneously, the control component of the scanning galvanometer receives a voltage signal that causes it to scan to a specific sub-field of view and generates a deflection at a preset angle. The scanning galvanometer deflects the modulated excitation light and projects it onto the sub-field of view through the objective lens, forming structured light illumination within the sub-field of view, exciting the sample within the sub-field of view, and simultaneously sending a trigger signal to the camera.

[0133] Furthermore, the excitation light is reflected onto a digital micromirror device (DMD) via a total internal reflection prism. The DMD consists of multiple miniature mirrors (e.g., 1920 × 1080), which can be loaded with any 8-bit image or binarized image to generate arbitrary structured light. In this embodiment, the excitation light can be modulated by a structured light pattern loaded onto the DMD to achieve a structured light illumination effect.

[0134] Step S303: The objective lens collects the fluorescence emitted by the sample within the sub-field of view from step S302. This fluorescence is reversed by the scanning galvanometer to cancel the deflection angle generated by the scanning and propagates along the system's optical axis. Subsequently, the spatial spectrum information of this fluorescence is relayed to the microlens array at a specific magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the above spatial spectrum information. The camera placed at the rear focal plane of the microlens array begins exposure after receiving a trigger signal, records the multi-view (number of viewpoints is v) imaging results of the sub-field of view, stops exposure after the set exposure time, and transmits the image to the computer.

[0135] Step S304: Following the scanning order encoded in step S301, sequentially traverse each preset sub-field of view, and repeat steps S302-S303. After traversal, one acquisition cycle is completed, obtaining images of each sub-field of view (the number is n, the same as the number of sub-fields of view). Subsequently, the imaging results of each sub-field of view are segmented according to different imaging angles, obtaining imaging results of each sub-field of view from each angle (the number is n×v). Then, a structured light tomography algorithm is used to process the background signal of each image of each sub-field of view from each angle, obtaining the background-removed imaging results of each sub-field of view from each angle (the number is n×v).

[0136] It should be noted that, in step S302, when structured light illumination is combined, a structured light tomography algorithm, such as the SIM (Structured illumination microscopy) algorithm, can be used to process the background signal of each image of each sub-field of view to obtain the image of each sub-field of view after background signal removal (the number of images is n×v).

[0137] Step S305: Perform 3D reconstruction on the image of each sub-field of view. Using the 3D point spread function of the sub-field of view acquired in step S301 and the Fourier light field reconstruction algorithm based on the multi-view iterative deconvolution method, the imaging results of each view after removing the background signal (number of v) are reconstructed into a background-removed 3D image (number of 1).

[0138] Step S306: Repeat step S305 to complete the 3D reconstruction of all sub-fields of view (number of which is n). Then, stitch together the background-removed 3D reconstruction results of all sub-fields of view according to their 3D positions to obtain a background-removed 3D microscopic image with a large field of view, high resolution, and high sampling rate (number of which is 1).

[0139] It should be noted that if the position and number of sub-fields of view are adjusted according to actual needs in step S301, such as scanning and imaging between any discontinuous sub-fields of view, then the imaging sub-fields of view do not need to be stitched together, and step S306 can be skipped as needed.

[0140] Optionally, in some embodiments, after traversing all sub-fields of view until the multi-view imaging results of all sub-fields of view are obtained, the method further includes: grouping the multi-view imaging results of all sub-fields of view according to the focus depth to obtain the grouped multi-view imaging results, and dividing the grouped multi-view imaging results according to different imaging angles to obtain the multi-focus imaging results of all sub-fields of view under different imaging angles.

[0141] Specifically, such as Figure 11 As shown, the embodiments of this application can also realize large field-of-view 3D imaging with extended depth of field. It should be noted that in order to obtain the large field-of-view 3D imaging result with extended depth of field in the embodiments of this application, the premise is that multiple sets of focusing microlens arrays are used.

[0142] Step S401: By modulating the input voltage of the scanning galvanometer control component, the positions of each sub-field of view (number n) for scanning imaging within the large field of view are determined, and then the scanning sequence of each sub-field of view is encoded (e.g., sub-field of view 1 → sub-field of view 2 → … → sub-field of view n). Within the field of view supported by the objective lens, the excitation light direction is deflected at different angles by the scanning galvanometer, so that the excitation light is focused on the determined different sub-fields of view at the front focal plane of the objective lens. Within each determined sub-field of view, the fluorescent sphere is moved in a direction perpendicular to the field of view with a preset step size (e.g., 1 μm), and an image at this depth is acquired after each movement, until the entire depth of field range (e.g., 200 μm) is traversed, completing the acquisition of the three-dimensional point spread function of each viewpoint in the sub-field of view.

[0143] Step S402: An excitation light is generated by the illumination source, and simultaneously the control component of the scanning galvanometer receives a voltage signal from a sub-field of view and generates a deflection at a preset angle. After the scanning galvanometer deflects the excitation light, it is projected onto the sub-field of view through the objective lens, exciting the sample within the sub-field of view, and simultaneously sending a trigger signal to the camera to start exposure.

[0144] Step S403: The objective lens collects the fluorescence emitted by the sample within the sub-field of view from step S402. This fluorescence is reversed by the scanning galvanometer to cancel the deflection angle generated by the scanning and propagates along the system optical axis. Subsequently, the spatial spectrum information of this fluorescence is relayed to a microlens array for extended depth of field at a specific magnification. This extended depth of field microlens array is characterized by including x groups of high-NA lenses focusing at different depths and one low-NA lens located at the center of the spatial spectrum. The imaging result of the low-NA lens contains signals across the entire depth of field range and can be used to enhance the three-dimensional reconstruction of any group of lenses at any focus depth. The microlens array performs optical spatial spectrum multiplexing imaging on the aforementioned spatial spectrum information. A camera placed at the rear focal plane of the microlens array begins exposure upon receiving a trigger signal, records the multi-view (number of viewpoints is v) imaging results of the sub-field of view, stops exposure after reaching the set exposure time, and transmits the image to the computer.

[0145] It should be noted that the microlens array consists of multiple sub-lenses (number: v≥2) to achieve imaging from multiple viewpoints, which can be selected according to requirements. In ordinary commercially available or custom-designed and manufactured microlens arrays, all sub-lenses have the same optical parameters and focusing depth; that is, the three-dimensional point spread function of all sub-lenses intersects at the same depth in the axial direction. However, the extended depth-of-field microlens array mentioned in this embodiment divides all sub-lenses in the high-frequency range into x+1 groups. Within each x groups, the sub-lenses have the same optical parameters and focusing depth, while the focusing depths of sub-lenses in different groups differ, thus achieving extended depth of field in the axial direction. Simultaneously, a lens with different optical parameters from the x groups is placed at the center of the spatial spectrum. Its imaging depth of field includes the depth of field of the x groups, and its focusing depth is centered at the focusing depth of the x groups.

[0146] Step S404: Following the scanning order encoded in step S401, sequentially traverse each preset sub-field of view, and repeat steps S402-S403. After traversal, one acquisition cycle is completed, obtaining images of each sub-field of view (the number is n, the same as the number of sub-fields of view). Then, firstly, the imaging results of each sub-field of view are divided into (x+1) groups according to their focus depth. Then, the images of each group with different imaging angles are segmented to obtain multi-focus imaging results for each sub-field of view and each angle (the number is...).

[0147] Step S405: Perform 3D reconstruction on each sub-field of view, using the 3D point spread function of the sub-field of view acquired in step S101 and the Fourier light field reconstruction algorithm based on the multi-view iterative deconvolution method to reconstruct the image results of each view at different focal depths (number of images is not specified). Reconstruct x 3D images (the number of images is x).

[0148] Step S406: Repeat step S405 to complete the 3D reconstruction of all sub-fields of view (number of which is n×x). Then, stitch together the 3D reconstruction results of all sub-fields of view obtained above according to their 3D positions and focus depths to obtain a large field of view, high-resolution 3D result with extended depth of field (number of which is 1).

[0149] It should be noted that in step S401, the position and number of sub-fields of view are adjusted according to actual needs. For example, if scanning and imaging are performed between any discontinuous sub-fields of view, then the imaging sub-fields of view do not need to be stitched together, and step S406 can be skipped as needed.

[0150] Optionally, in some embodiments, before generating a voltage signal corresponding to any sub-field of view, the method includes: acquiring a tracking module image encompassing the entire objective lens field of view, analyzing the sub-field of view location of the tracking target based on the tracking module image, and determining an analog voltage signal for the sub-field of view location.

[0151] Specifically, in combination Figure 5 As shown, the embodiments of this application can also be combined with a tracking module to realize real-time tracking and real-time three-dimensional imaging of the tracking target. The tracking module obtains the position of the target sub-field of view where the tracking target is located and calculates the voltage signal corresponding to the target sub-field of view where the tracking target is located. The scanning module controls the scanning galvanometer to deflect based on the voltage signal, thereby realizing real-time tracking and real-time three-dimensional imaging of the tracking target.

[0152] It should be noted that the foregoing explanation of the three-dimensional microscopic imaging device embodiment also applies to the three-dimensional microscopic imaging method of this embodiment, and will not be repeated here.

[0153] According to the three-dimensional microscopic imaging method proposed in this application, multiple sub-fields of view within a large field of view are determined, and the three-dimensional point spread function of each sub-field of view is acquired. The excitation light and the voltage signal of any sub-field of view are received. Based on the voltage signal, the sample in the corresponding sub-field of view is excited by the excitation light, and the fluorescence emitted by the sample is collected by the objective lens. The spatial spectrum information of the fluorescence is relayed to the microlens array at a preset magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the spatial spectrum information to obtain the multi-view imaging results of the corresponding sub-field of view. The multi-view imaging results are recorded by the camera. All sub-fields of view are traversed until the multi-view imaging results of all sub-fields of view are obtained. The multi-view imaging results of all sub-fields of view are segmented according to different imaging angles to obtain the imaging results of all sub-fields of view under different imaging angles. Using the three-dimensional point spread function and the preset reconstruction algorithm, the imaging results of all sub-fields of view under different imaging angles are reconstructed to obtain the three-dimensional reconstructed images of all sub-fields of view. The three-dimensional reconstructed images of all sub-fields of view are stitched together to obtain the target three-dimensional microscopic image. This solves the problem that related technologies can only perform small-volume imaging within a small field of view, making it difficult to achieve rapid 3D imaging with large volume and high resolution. It can obtain 3D imaging results with large volume, high resolution, and high sampling rate.

[0154] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0155] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0156] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0157] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0158] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0159] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A three-dimensional microscopic imaging device, characterized in that, include: The module consists of an illumination source module, a scanning module, and an imaging module, among which... The lighting source module includes a light source component, a filter, a first lens, a second lens, a third lens, and a fourth lens arranged in sequence. The light source component is used to emit excitation light. The light source component, the filter, and the first lens are used to filter the excitation light to form filtered excitation light. The second lens is used to focus the filtered excitation light to form focused excitation light. The third lens and the fourth lens amplify the focused excitation light at a preset magnification and perform relay processing to obtain a first relayed excitation light. The scanning module includes a dichroic mirror, a scanning galvanometer, a fifth lens, a sixth lens, a first reflecting mirror, and an objective lens arranged in sequence. The dichroic mirror reflects the first relayed excitation light to the scanning galvanometer. The scanning galvanometer deflects the first relayed excitation light based on a first voltage signal. The fifth lens, the sixth lens, and the first reflecting mirror relay the deflected excitation light to the objective lens to obtain a second relayed excitation light. The objective lens focuses the second relayed excitation light onto the target sub-field of view, excites the sample in the target sub-field of view, and collects the fluorescence signal emitted by the sample. The fluorescence signal enters the imaging system sequentially through the first reflecting mirror, the sixth lens, the fifth lens, the scanning galvanometer, and the dichroic mirror. The imaging module includes a seventh lens, a second mirror, an aperture, an eighth lens, a filter, a microlens array, and a camera arranged in sequence. The seventh lens, the second mirror, the aperture, and the eighth lens are used to relay the spatial spectrum information of the fluorescence signal to the filter. The filter is used to filter the fluorescence signal. The microlens array is used to perform optical spatial spectrum multiplexing imaging based on the spatial spectrum information, and the camera records the optical spatial spectrum multiplexing imaging results.

2. The apparatus according to claim 1, characterized in that, Also includes: A total internal reflection prism and a digital micromirror device are provided, wherein the total internal reflection prism and the digital micromirror device are located between the filter and the first lens, wherein... The digital micromirror device is used to load a preset structured light image to form structured light illumination, and to modulate the excitation light based on the structured light illumination to form modulated excitation light; The total internal reflection prism is used to reflect the excitation light to the digital micromirror device, receive the modulated excitation light, and reflect the modulated excitation light to the first lens.

3. The apparatus according to claim 1, characterized in that, Also includes: The tracking module, in which, The tracking module includes, in sequence, an infrared illumination source, a ninth lens, an infrared dichroic mirror, an optical trap, a tenth lens, and a tracking camera. The infrared illumination source is used to generate infrared illumination for all imaging fields of the objective lens. The ninth lens and the tenth lens are used to image all imaging fields supported by the objective lens onto the tracking camera. The infrared dichroic mirror is used to reflect the excitation light transmitted through the objective lens to the optical trap. The optical trap is used to eliminate the excitation light transmitted through the objective lens. The tracking camera is used to record the imaging images of all imaging fields.

4. The apparatus according to claim 3, characterized in that, The tracking module further includes: a processing unit. The processing unit is used to receive the imaging image emitted by the tracking camera, process the imaging image to obtain the position of the target sub-field of view where the tracking target is located, calculate the second voltage signal corresponding to the target sub-field of view, and send the second voltage signal to the scanning module.

5. The apparatus according to claim 1, characterized in that, The microlens array consists of at least one set of sublenses with the same optical parameters and the same depth of focus; Alternatively, a microlens array for extended depth of field consists of at least two sets of sublenses with the same optical parameters and the same depth of focus, each focusing at a different depth, and a sublens whose imaging depth of field includes the sum of multiple depths of focus.

6. A method for three-dimensional microscopic imaging, applied to the three-dimensional microscopic imaging apparatus according to claims 1-4, characterized in that, Includes the following steps: Based on a preset partitioning strategy, the target large field of view is divided into multiple sub-fields of view, and the position and scanning imaging order of each sub-field of view are determined. The analog voltage signal corresponding to the position of each sub-field of view is calculated, and the three-dimensional point spread function of each sub-field of view is pre-acquired along the direction perpendicular to the sub-field of view. A voltage signal corresponding to any sub-field of view is generated and applied to a scanning galvanometer, causing the scanning galvanometer to deflect the excitation light to the corresponding sub-field of view and excite the sample within the corresponding sub-field of view; The fluorescence emitted by the sample is collected by the objective lens, and the spatial spectrum information of the fluorescence is relayed to the microlens array at a preset magnification. The microlens array performs optical spatial spectrum multiplexing imaging on the spatial spectrum information to obtain the multi-view imaging results of the corresponding sub-field of view, and the multi-view imaging results are recorded by the camera. Traverse all sub-fields of view until the multi-view imaging results of all sub-fields of view are obtained, and then divide the multi-view imaging results of all sub-fields of view according to different imaging angles to obtain the imaging results of all sub-fields of view under different imaging angles. Using the pre-acquired 3D point spread function and deconvolution reconstruction algorithm, the imaging results of all sub-fields of view under different imaging angles are reconstructed to obtain 3D reconstructed images of all sub-fields of view, and the 3D reconstructed images of all sub-fields of view are stitched together to obtain a 3D microscopic image of the target large field of view.

7. The method according to claim 6, characterized in that, The process of generating excitation light and a voltage signal for any sub-field of view, and the scanning galvanometer deflecting the excitation light to the corresponding sub-field of view based on the voltage signal, and exciting the sample within the corresponding sub-field of view, includes: The excitation light is reflected to the digital micromirror assembly by a total internal reflection prism, and a preset structured light image is loaded onto the digital micromirror assembly to form structured light illumination, so as to modulate the excitation light to obtain modulated excitation light.

8. The method according to claim 6, characterized in that, After segmenting the multi-view imaging results of all sub-fields of view according to different imaging viewpoints to obtain the imaging results of all sub-fields of view under different imaging viewpoints, the method further includes: The imaging results of each sub-field of view under different imaging angles are processed based on the structured light tomography algorithm to obtain the imaging results of all sub-fields of view under different imaging angles after background signal removal.

9. The method according to claim 6, characterized in that, After traversing all the sub-fields of view until the multi-view imaging results of all the sub-fields of view are obtained, the process also includes: The multi-view imaging results of all sub-fields of view are grouped according to the focus depth to obtain the grouped multi-view imaging results. The grouped multi-view imaging results are then divided according to different imaging angles to obtain the multi-focus imaging results of all sub-fields of view under different imaging angles.

10. The method according to claim 6, characterized in that, Before generating the voltage signal corresponding to any sub-field of view, the process includes: Acquire an image of the tracking module that includes the entire objective lens field of view, analyze the sub-field of view location of the tracking target based on the tracking module image, and determine the analog voltage signal of the sub-field of view location.

Images