Fluorescence and phase dual-mode microscopic imaging device based on deep learning

Abstract

The application discloses a kind of fluorescence and phase dual-mode microscopic imaging device based on deep learning, including super-resolution fluorescence microscopic module, quantitative phase microscopic module and image processing module, super-resolution fluorescence microscopic module obtains multiple single-layer optical section images under different phase stripe structure light illumination, quantitative phase microscopic module obtains multiple transmission intensity images of sample under different phase modulation, image processing module is used to reconstruct the super-resolution fluorescence image of sample according to multiple single-layer optical section images, reconstruct the quantitative phase image of sample according to multiple transmission intensity images, obtain the super-resolution fluorescence image of sample current position using the quantitative phase image of sample specific thickness position.The application integrates super-resolution fluorescence microscopy and quantitative phase microscopy mode into the same system, can obtain the fluorescence and phase image of sample simultaneously, provides multi-dimensional information for sample;Corresponding fluorescence image can be obtained from single phase image, and multiple subcellular structures of sample are identified and segmented.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of microscopic imaging technology, and particularly relates to a fluorescence and phase dual-mode microscopic imaging device based on deep learning. BACKGROUND

[0002] Optical microscopes have the advantages of relatively simple structure, low requirements for sample preparation, and the ability to image multi-scale living samples for a long time, and have become an important means for people to understand and study the microscopic world. Among them, fluorescence microscopes can achieve high-contrast imaging of specific structures with the help of fluorescent markers. In recent years, in order to break the optical diffraction limit, super-resolution optical microscopy has developed rapidly. Structured illumination microscopy (SIM) technology has the advantages of fast imaging speed, low phototoxicity, no excessive harsh requirements for fluorescent molecules and labeling methods, strong tomographic ability, and is particularly favored by biomedical researchers.

[0003] SIM technology is a wide-field super-resolution optical microscopic imaging technology, which illuminates the sample with structured light of different directions and different phase shifts and records the generated "Moire fringes" in turn, and finally reconstructs the high-frequency information of the sample using the corresponding algorithm to obtain a super-resolution image of the sample. In addition, SIM has the ability of super-resolution imaging and three-dimensional tomographic imaging. By recording intensity images with phase shifts of 0°, 120° and 240°, a wide-field tomographic image at any axial position z can be calculated.

[0004] Due to the phototoxicity and photobleaching of fluorescent markers, fluorescence microscopy is not suitable for long-term observation of samples. Quantitative phase microscopy collects the phase information of the object light wave after passing through the sample, which can not only achieve high-contrast imaging, but also quantitatively obtain the three-dimensional morphology and refractive index distribution of micro-objects. Most subcellular organelles in cells have different refractive indices from the cytoplasm, so quantitative phase microscopy can simultaneously image more than ten subcellular organelles. Taewoo et al. proposed a quantitative phase microscopy technology based on Zernike phase contrast (Spatial Light Interference Microscope, SLIM), which used an objective lens with a numerical aperture of 1.4 and a halogen lamp with a central wavelength of 590 nm to obtain a lateral resolution of 350 nm and a temporal resolution of 16 Hz. The optical structure of the common-path interference makes SLIM very immune to environmental disturbances, and the wide-spectrum illumination improves the image quality and avoids speckle noise caused by laser illumination. However, the spatial resolution and temporal resolution of this system are still insufficient to detect the fine structure of organelles in living cells, and only the overall morphological changes of cells can be detected.

[0005] Fluorescence images can specifically observe the organelles of interest, and phase images can simultaneously observe all subcellular organelles (whose refractive index is different from that of the cytoplasm). However, current structured light illumination super-resolution fluorescence microscopy and label-free phase microscopy are often in a separate state, and cannot provide fluorescence / phase dual-mode images for the same sample. SUMMARY

[0006] In order to solve the above problems existing in the prior art, the present application provides a fluorescence and phase dual-mode microscopic imaging device based on deep learning. The technical problem to be solved by the present application is realized by the following technical scheme:

[0007] The present application provides a fluorescence and phase dual-mode microscopic imaging device based on deep learning, comprising a super-resolution fluorescence microscopy module, a quantitative phase microscopy module and an image processing module,

[0008] The super-resolution fluorescence microscopy module comprises a monochromatic extended illumination sub-module, an illumination modulation sub-module and a fluorescence image acquisition sub-module, wherein the monochromatic extended illumination sub-module is used to generate monochromatic extended illumination light with a certain angular distribution; the illumination modulation sub-module is used to modulate the monochromatic extended illumination light to generate stripe structured light with different phases; and the fluorescence image acquisition sub-module is used to record a plurality of single-layer light slice images of the sample under illumination of the stripe structured light with different phases.

[0009] The quantitative phase microscopy module comprises a ring-shaped inclined illumination sub-module, a microscopic imaging sub-module, a light beam modulation sub-module and a phase image acquisition sub-module, wherein the ring-shaped inclined illumination sub-module is used to generate ring-shaped illumination light and converge the illumination light on the sample in an inclined direction to generate non-scattered light unaffected by the sample and scattered light containing sample information; the microscopic imaging sub-module is used to collect and amplify the scattered light and non-scattered light of the sample; the spatial light modulator in the light beam modulation sub-module is used to modulate the phase of the non-scattered light of the sample; and the phase image acquisition sub-module is used to record the transmission intensity images generated by the scattered light and the non-scattered light modulated by different phases.

[0010] The image processing module comprises a fluorescence image reconstruction sub-module, a phase image reconstruction sub-module and a deep learning sub-module, wherein the fluorescence image reconstruction sub-module is used to reconstruct a super-resolution fluorescence image of the sample according to a plurality of single-layer light slice images of the sample under illumination of the stripe structured light with different phases; the phase image reconstruction sub-module is used to reconstruct a quantitative phase image of the sample according to a plurality of transmission intensity images generated by the non-scattered light modulated by different phases; and the deep learning sub-module is used to obtain a super-resolution fluorescence image of the current position of the sample by using the quantitative phase image of the specific thickness position of the sample, so as to identify and segment different subcellular structures in the sample.

[0011] In one embodiment of the present application, the monochromatic extended illumination sub-module comprises a laser and a first microscopic objective and a first lens arranged in sequence along the optical axis of the laser, and the illumination modulation sub-module comprises a digital micromirror array, a second lens, a spectral filter and a third lens arranged in sequence along the optical axis of the laser, wherein,

[0012] The laser is used to emit single-frequency laser; the first microscopic objective and the first lens are used to expand the single-frequency laser generated by the laser to cover the entire working surface of the digital micromirror array;

[0013] The digital micromirror array is used to modulate the monochromatic extended illumination light from the monochromatic extended illumination sub-module to generate a stripe structured light with different phase shift amounts; the second lens is used to focus the stripe structured light on the spectral filter; the spectral filter is used to perform spectral selection on the stripe structured light modulated by the digital micromirror array, and only allows the spectrum of ±1 diffraction light to pass; and the third lens is used to expand and collimate the light beam selected by the spectral filter into parallel illumination light.

[0014] In one embodiment of the present application, the fluorescence image acquisition sub-module comprises a second mirror, a first dichroic mirror, a fourth lens, a third mirror, a second microscopic objective, a filter and a first camera, wherein,

[0015] The second mirror, the first dichroic mirror, the fourth lens, the fourth mirror and the second microscopic objective are arranged in sequence along the optical axis of the laser, and the front focal plane of the second microscopic objective is used to arrange a sample;

[0016] The filter and the first camera are arranged in sequence on the side of the first dichroic mirror away from the fourth lens, used to receive the light beam reflected from the sample, and obtain a plurality of single-layer optical slice images of the current thickness position of the sample under different phase stripe structured light illumination.

[0017] In one embodiment of the present application, the digital micromirror array is used to load binary structured light stripes and sequentially perform 0, 2π / 3, 4π / 3 phase shift operations;

[0018] The first camera is used to record three corresponding single-layer optical slice images respectively when the digital micromirror array performs 0, 2π / 3, 4π / 3 phase shift operations.

[0019] In one embodiment of the present application, the annular oblique illumination sub-module comprises a light source mounting rack and a plurality of LEDs, the light source mounting rack is a plurality of equal-height and concentric annular mounting structures, the plurality of LEDs are uniformly mounted on the annular mounting structures in the circumferential direction, and the light source mounting rack is arranged on the other side of the sample away from the second microscope objective, capable of generating annular illumination light converging in the oblique direction and irradiating on the sample;

[0020] The microscopic imaging sub-module comprises a fourth lens, a second dichroic mirror, a fourth mirror, a linear polarizer and a fifth lens arranged in sequence along the light path direction of the annular illumination light, wherein the fourth lens is located at the back focal plane of the second microscope objective and is confocal with the fifth lens, the second dichroic mirror is obliquely arranged between the fourth lens and the first dichroic mirror, for reflecting the light beam from the fourth lens to the fourth mirror, and the linear polarizer is used to modulate the light beam from the fourth mirror into linearly polarized light.

[0021] In one embodiment of the present application, the light beam modulation sub-module comprises a beam splitter prism and a spatial light modulator arranged in sequence along the light path direction of the annular illumination light, and the phase image acquisition sub-module comprises a fifth mirror, a sixth lens and a second camera arranged in sequence, wherein,

[0022] The beam splitter prism is used to transmit the light beam from the fifth lens to the spatial light modulator, and reflect the light beam modulated by the spatial light modulator to the fifth mirror;

[0023] The spatial light modulator is used to load 0, π / 2, π and 3π / 2 phase modulation on the non-scattered light;

[0024] The sixth lens is arranged at the back focal plane of the spatial light modulator; and the second camera is used to receive the light beam transmitted back from the sample, and obtain four transmission intensity images of the sample when the spatial light modulator loads 0, π / 2, π and 3π / 2 phase modulation, respectively.

[0025] In one embodiment of the present application, the transmission intensity image detected by the phase image acquisition sub-module is represented as:

[0026]

[0027] Wherein, represents the spatial coordinates of the scatterer, represents the total intensity of the non-scattered light, represents the total intensity of the scattered light, represents the mutual coherence function between the scattered light and the non-scattered light, represents the average phase modulation function of the sample to the light field, π / 2, π, and 3π / 2 represent the modulation phase loaded by the spatial light modulator;

[0028] The phase image reconstruction submodule is specifically used for:

[0029] The phase distribution of the sample was calculated using a phase-shifting algorithm:

[0030]

[0031] in, and They respectively represent the modulation phase Transmission intensity images at π / 2, π, and 3π / 2;

[0032] The mutual coherence function between scattered and non-scattered light was calculated using a phase-shifting algorithm.

[0033] The refractive index distribution of the sample is obtained by deconvolution, and then the thickness distribution of the sample is obtained.

[0034] In one embodiment of the present invention, the expression for the single-layer optical slice image obtained by the fluorescence image acquisition submodule is:

[0035] D m (x,y)=D in (x,y)·{1+cos[2πkx+2(m-1)π / 3]}+D out (x,y)

[0036] Where m = 1, 2, 3 represent the phase shift number, and D in (x,y) represents the focal component of a single-layer optical slice image, D out (x,y) represents the defocused component of a single-layer light slice image, and k is the spatial frequency of the fringe;

[0037] The fluorescence image reconstruction submodule is specifically used for:

[0038] The focal component D is obtained by solving the expression of the single-layer optical slice diagram. in (x,y):

[0039]

[0040] For the coke component D in Perform deconvolution on (x, y) to obtain the in-focus component after deconvolution, which is the planar image D of the sample at the current thickness position. SIM (x,y):

[0041]

[0042] Among them, among them, represents a focal component D in (x,y) the Fourier spectrum of, represents an optical transfer function, represents a two-dimensional inverse Fourier transform, is a frequency domain apodization function, w is a Wiener coefficient.

[0043] In an embodiment of the present application, the deep learning sub-module comprises a pre-trained U-Hybrid Transformer network model, the U-Hybrid Transformer network model takes the phase image obtained by the phase image reconstruction sub-module as original data, takes the fluorescence image obtained by the fluorescence image reconstruction sub-module as reference data, and is trained by using a large number of fluorescence-phase paired images as a data set to obtain the pre-trained U-Hybrid Transformer network model, the fluorescence-phase paired image refers to a super-resolution fluorescence image and a quantitative phase image obtained at the same thickness position of a sample.

[0044] In an embodiment of the present application, the U-Hybrid Transformer network model comprises an encoder, a decoder and a skip connection module, wherein,

[0045] The encoder is composed of a plurality of hybrid attention mechanism modules and a convolution-free down-sampling module, the hybrid attention mechanism module is used to realize multi-scale semantic feature extraction of the input phase image, and the convolution-free down-sampling module is used to down-sample the extracted multi-scale semantic features;

[0046] The decoder is composed of a plurality of hybrid attention mechanism modules and an up-sampling transposed convolution layer;

[0047] The skip connection module is used to fuse the features of the corresponding layers of the encoder and the decoder, so as to learn the feature semantics at different stages.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] 1. The fluorescence and phase dual-mode microscopic imaging device based on deep learning of the present application integrates super-resolution fluorescence microscopy and quantitative phase microscopy modes into the same imaging system; the fluorescence image and the phase image of the sample can be obtained at the same time, multi-dimensional information of the same sample is provided, and the device has great application value in the fields of biomedicine and the like, and the corresponding fluorescence image can be quickly output by using the phase image without fluorescence labeling.

[0050] 2、The super-resolution fluorescence microscopy module of the present application has extremely high modulation speed (>=9856Hz) by generating and switching binary structured light fringes on a digital micromirror array (DMD); high-contrast structured light can be generated in the entire imaging field of view by filtering the spectrum of the illumination light (blocking the 0-order light spectrum and only allowing the spectrum of the +1 diffraction light to pass).

[0051] 3、The image processing module of the present application takes the acquired phase image as the original data and the fluorescence image as the reference data, trains the U-Hybrid Transformer network model based on a large number of fluorescence-phase paired images obtained by the fluorescence-phase dual-mode microscopic imaging device as a data set, and obtains the corresponding pre-trained U-Hybrid Transformer network model. The phase image without fluorescence labeling is input into the network, and the trained U-Hybrid Transformer network model can quickly output the corresponding fluorescence image. The ability to predict the corresponding fluorescence image of the sample without fluorescence labeling of the sample is realized, which can be widely applied to the prediction and identification of various biological tissues, provides a method for obtaining images of different imaging modalities, makes the analysis of the sample more comprehensive and convenient, and has great application value in the fields of biomedicine and the like.

[0052] The present application will be further described in detail below in combination with the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a structural schematic diagram of a fluorescence-phase dual-mode microscopic imaging device based on deep learning provided by an embodiment of the present application;

[0054] Figure 2 is a deep network architecture schematic diagram of a U-Hybrid Transformer network model provided by an embodiment of the present application;

[0055] Figure 3 is a detailed structure schematic diagram of a U-Hybrid Transformer network model provided by an embodiment of the present application;

[0056] Figure 4 is a result schematic diagram of four-step phase shift operation and reconstruction of a lily stigma cross-section sample by a quantitative phase microscopy module of the present application;

[0057] Figure 5is a schematic diagram of the results of three-step phase shift operation and reconstruction of a lily stigma transverse section sample by the super-resolution fluorescence microscopy module of the embodiment of the present application;

[0058] Figure 6 is a schematic diagram of experimental results of imaging 240nm fluorescent polystyrene beads by the deep learning-based super-resolution fluorescence microscopy and phase dual-mode microscopy imaging device of the embodiment of the present application.

[0059] BRIEF DESCRIPTION OF DRAWINGS

[0060] 1-laser; 2-first microscope objective; 3-first lens; 4-first mirror; 5-digital micromirror array; 6-second lens; 7-spectrum filter; 8-third lens; 9-second mirror; 10-first dichroic mirror; 11-fourth lens; 12-third mirror; 13-second microscope objective; 14-sample; 15-filter; 16-first camera; 17-light source mounting rack; 18-plurality of LEDs; 19-second dichroic mirror; 20-fourth mirror; 21-linear polarizer; 22-fifth lens; 23-diffractive prism; 24-spatial light modulator; 25-fifth mirror; 26-sixth lens; 27-second camera. DETAILED DESCRIPTION

[0061] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined object, the deep learning-based fluorescence and phase dual-mode microscopy imaging device according to the present application is described in detail below in combination with the drawings and specific embodiments.

[0062] The foregoing and other technical contents, features and effects of the present application can be clearly presented in the detailed description of the specific embodiments below in combination with the drawings. Through the description of the specific embodiments, the technical means and effects taken by the present application to achieve the predetermined object can be understood more deeply and specifically. However, the attached drawings are provided for reference and illustration only, and are not intended to limit the technical solutions of the present application.

[0063] It should be noted that, in this document, relational terms such as first and second, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a... " does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0064] The embodiment of the present application provides a fluorescence and phase dual-mode microscopic imaging device based on deep learning, which comprises a super-resolution fluorescence microscopic module, a quantitative phase microscopic module and an image processing module. The super-resolution fluorescence microscopic module comprises a monochromatic extended illumination sub-module, an illumination modulation sub-module and a fluorescence image acquisition sub-module. The monochromatic extended illumination sub-module is used for generating monochromatic extended illumination light with a certain angle distribution; the illumination modulation sub-module is used for modulating the monochromatic extended illumination light to generate stripe structured light with different phases; and the fluorescence image acquisition sub-module is used for recording a plurality of single-layer optical section images of a sample under illumination of the stripe structured light with different phases.

[0065] Please refer to Figure 1 , Figure 1 is a structure diagram of the fluorescence and phase dual-mode microscopic imaging device based on deep learning provided by the embodiment of the present application. The monochromatic extended illumination sub-module of the embodiment comprises a laser 1 and a first microscope objective 2 and a first lens 3 arranged in sequence along the optical axis direction of the laser 1, wherein the laser 1 is used for emitting single-frequency laser; and the first microscope objective 2 and the first lens 3 are used for expanding the single-frequency laser generated by the laser 1 to cover the entire working surface of a digital micromirror array 5 included in the illumination modulation sub-module.

[0066] The illumination modulation sub-module comprises the digital micromirror array 5, a second lens 6, a spectrum filter 7 and a third lens 8 arranged in sequence along the optical axis direction, wherein the digital micromirror array 5 is used for modulating the monochromatic extended illumination light from the first lens 3 to generate stripe structured light with different phase shift amounts; the second lens 6 is used for focusing the stripe structured light on the spectrum filter 7; the spectrum filter 7 is used for performing spectrum selection on the stripe structured light modulated by the digital micromirror array 5, and only allows the spectrum of ±1 diffraction light to pass; and the third lens 8 is used for expanding and collimating the light beam selected by the spectrum filter 7 into parallel illumination light. It should be noted that the digital micromirror array 5 can also be replaced by other appropriate digital projection devices such as a spatial light modulator.

[0067] Further, the illumination modulation sub-module of the embodiment further comprises a first reflector 4 obliquely arranged between the first lens 3 and the digital micromirror array 5, which is used for reflecting the monochromatic extended illumination light from the first lens 3 to the working surface of the digital micromirror array 5.

[0068] Please refer to Figure 1The fluorescence image acquisition submodule of the embodiment includes a second mirror 9, a first dichroic mirror 10, a fourth lens 11, a third mirror 12, a second microscope objective 13, a filter 15, and a first camera 16. The second mirror 9, the first dichroic mirror 10, the fourth lens 11, the fourth mirror 12, and the second microscope objective 13 are sequentially arranged along the optical axis direction of the laser 1, and the sample 14 is arranged at the front focal plane of the second microscope objective 13. The filter 15 and the first camera 16 are sequentially arranged on the side of the first dichroic mirror 10 away from the fourth lens 11, for receiving the light beam reflected from the sample 14, and obtaining multiple single-layer light slice images of the current thickness position of the sample 14 under the illumination of the stripe structure light of different phases. The first camera 16 of the embodiment is a scientific grade sCMOS camera.

[0069] The binary structure light stripe is loaded on the digital micromirror array 5 of the embodiment, and the 0, 2π / 3, and 4π / 3 phase shift operations are sequentially performed. Correspondingly, the first camera 16 is used to record three corresponding single-layer light slice images when the digital micromirror array 5 performs the 0, 2π / 3, and 4π / 3 phase shift operations, respectively.

[0070] Further, the quantitative phase microscopy module includes a ring-shaped oblique illumination submodule, a microscopic imaging submodule, a light beam modulation submodule, and a phase image acquisition submodule. The ring-shaped oblique illumination submodule is used to generate ring-shaped illumination light and converge and irradiate the ring-shaped illumination light on the sample in an oblique direction, to generate non-scattered light not affected by the sample and scattered light containing sample information. The microscopic imaging submodule is used to collect and amplify the scattered light and the non-scattered light of the sample. The light beam modulation submodule is used to perform phase modulation on the non-scattered light of the sample. The phase image acquisition submodule is used to record the transmission intensity images generated by the scattered light and the non-scattered light modulated by different phases.

[0071] Continuing to refer to Figure 1 The ring-shaped oblique illumination submodule of the embodiment includes a light source mounting frame 17 and a plurality of LEDs 18. The light source mounting frame is a plurality of ring-shaped mounting structures with the same height and concentricity. The plurality of LEDs are uniformly mounted on the ring-shaped mounting structures in the circumferential direction. The light source mounting frame 17 is arranged on the other side of the sample away from the second microscope objective 13, and can generate ring-shaped illumination light converging and irradiating on the sample in an oblique direction.

[0072] In the embodiment, the selection of the LEDs 18 needs to consider that the LEDs have a very small divergence angle, a large power, a short central wavelength, and a narrow spectral range, so that the quantitative phase microscopy module has high temporal and spatial resolution and ensures that the spatial light modulator can accurately modulate the phase of the light field. Preferably, the diameter of the LED 18 is 5 mm, the wavelength range is 470 nm±10 nm, and the power of a single LED is 0.36 watts.

[0073] Further, the microscopic imaging sub-module comprises a fourth lens 11, a second dichroic mirror 19, a fourth mirror 20, a linear polarizer 21 and a fifth lens 22 arranged in sequence along the light path direction of the annular illumination light, wherein the fourth lens 11 is located at the back focal plane of the second microscopic objective 13 and is confocal with the fifth lens 22, the second dichroic mirror 19 is obliquely arranged between the fourth lens 11 and the first dichroic mirror 10, and is used for reflecting the light beam from the fourth lens 11 to the fourth mirror 20, and the linear polarizer 21 is used for modulating the light beam into linearly polarized light.

[0074] The light beam modulation sub-module comprises a beam splitter prism 23 and a spatial light modulator 24 arranged in sequence along the light path direction of the annular illumination light, and the phase image acquisition sub-module comprises a fifth mirror 25, a sixth lens 26 and a second camera 27 arranged in sequence, wherein the beam splitter prism 23 is used for transmitting the light beam from the fifth lens 22 to the spatial light modulator 24, and reflecting the light beam modulated by the spatial light modulator 24 to the fifth mirror 25; the spatial light modulator 24 is used for loading 0, π / 2, π and 3π / 2 phase modulation on the non-scattered light; the sixth lens 26 is arranged at the back focal plane of the spatial light modulator 24; and the second camera 27 is used for receiving the light beam transmitted back from the sample 14, and obtaining four groups of transmission intensity images of the sample when the spatial light modulator 24 loads 0, π / 2, π and 3π / 2 phase modulation respectively. The second camera 27 of the embodiment is a scientific grade sCMOS camera.

[0075] In the process of using the super-resolution fluorescence microscope module, the single-frequency laser emitted by the laser 1 passes through the first microscopic objective 2 and the first lens 3 in sequence, and after beam expansion and collimation, it is used as monochromatic extended illumination light. The monochromatic extended illumination light is incident on the digital micromirror array 5 through the first mirror 4, the digital micromirror array 5 loads a binary structured light fringe and performs 0, 2π / 3, 4π / 3 phase shift operations in sequence, modulates the light beam, generates structured light with different phases, and then passes through the second lens 6, the spectral filter 7 and the third lens 8. The spectral filter 7 performs spectral selection filtering (blocks zero order) on the light beam modulated by the digital micromirror array 5, only allows the spectrum of ±1 diffraction light to pass through, and finally the illumination light beam is imaged onto the sample 14. The ±1 diffraction light interferes in the focal plane of the sample 14 and generates two-dimensional binary structured illumination on the sample 14. The fluorescence excited after illuminating the sample returns along the original light path (still passing through the second microscopic objective 13, the mirror 12 and the fourth lens 11 in sequence), passes through the first dichroic mirror 10 and the filter 15, and is received by the first camera 16. That is, at a thickness position of the sample 14, the fringe loaded on the digital micromirror array 5 needs to be subjected to 0, 2π / 3, 4π / 3 three-step phase shift operations, and three groups of fluorescence intensity images are recorded by the first camera 16, i.e. three groups of single-layer optical section images of the sample 14 at a thickness position.

[0076] During the use of the quantitative phase microscopy module, the position of the sample 14 is kept unchanged, the laser 1 is turned off, the LEDs are turned on, and the plurality of LEDs 18 are obliquely incident on the same thickness position of the sample 14 as an illumination light source to generate non-scattered light and scattered light related to the sample, which are collimated into parallel light after the second microscope objective 13 and the fourth lens 11. Among them, the second microscope objective 13 is used to collect the scattered light and the non-scattered light. After passing through the microscopic imaging submodule composed of the fourth lens 11 and the fifth lens 22, the linear polarizer 21 and the light splitting prism 23, the light beam converges on the spatial light modulator 24. The spatial light modulator 24 is used to load the non-scattered light with four-step phase shift of 0, π / 2, π, 3π / 2. The modulated light beam is received by the second camera 27 after passing through the light splitting prism 23, the fifth mirror 25 and the sixth lens 26. That is, at a thickness position of the sample, the spatial light modulator 24 needs to be loaded with four-step phase shift of 0, π / 2, π, 3π / 2 on the stripe, and four sets of transmission intensity images of the sample 14 at the current thickness position are recorded by the second camera 27.

[0077] Further, the image processing module includes a fluorescence image reconstruction submodule, a phase image reconstruction submodule and a deep learning submodule, wherein the fluorescence image reconstruction submodule is used to reconstruct the super-resolution fluorescence image of the sample according to a plurality of single-layer light slice images of the sample under the illumination of the stripe structured light at different phases; the phase image reconstruction submodule is used to reconstruct the quantitative phase image of the sample according to a plurality of transmission intensity images generated by the non-scattered light modulated at different phases, and the deep learning submodule is used to obtain the super-resolution fluorescence image of the current position of the sample by using the quantitative phase image of the specific thickness position of the sample, so as to accurately identify and segment different subcellular structures in the sample.

[0078] Specifically, the light emitted by each LED 18 in the quantitative phase microscopy module is natural light with a certain wavelength range, which is composed of a large number of random wave trains, and the vibration direction, propagation direction and phase difference of each wave train are random, so the wave trains are incoherent with each other, and the intensity of each wave train is superimposed. However, for a certain wave train, it is coherent itself, so under the illumination of the annular oblique illumination submodule, the light field distribution under the illumination of a certain wave train is first solved by coherent processing means, and then the intensity distribution under the simultaneous illumination of each wave train is synthesized by incoherent processing means.

[0079] Now consider monochromatic illumination light, that is, the wavelengths of all LEDs are the same, and the angular frequency is represented as w, and it is assumed that the sample is a scatterer with a refractive index distribution of , wherein represents the spatial coordinates of the scatterer. The scalar light field distribution of the monochromatic wave light field during the transmission in the scatterer is According to the Zernike phase contrast principle, the light field after the monochromatic illumination light passing through the sample 14 can be divided into non-scattering light which is not affected by the sample and scattering light which is affected by the refractive index and thickness of the sample. It is worth noting that in the actual imaging process, all LEDs are lit at the same time, and because the light fields of different frequencies are incoherent, and the illumination at different angles is realized by different light sources in actual situations, the light fields generated by different illumination angles are also incoherent. Therefore, without the modulation of the spatial light modulator 24, the total light intensity distribution detected by the phase image acquisition submodule under the illumination of the annular inclined illumination submodule, i.e., the transmission intensity image, can be represented as:

[0080]

[0081] where the angle bracket <> represents the average processing in the time scale, represents the illumination wave vector, represents the total intensity of the non-scattering light, represents the total intensity of the scattering light, represents the mutual coherence function between the scattering light and the non-scattering light, represents the average phase modulation function of the sample to the light field.

[0082] In this embodiment, the non-scattering light not affected by the sample is modulated by the spatial light modulator 24 (modulation phases π / 2, π, and 3π / 2), while the high-order scattering light with sample information is not modulated by any device. At this time, the total light intensity detected by the phase image acquisition submodule is represented as:

[0083]

[0084] The phase distribution of the sample can be calculated using the phase shift algorithm:

[0085]

[0086] wherein and represent the transmission intensity images collected by the phase image acquisition submodule when the modulation phases π / 2, π, and 3π / 2, respectively.

[0087] The mutual coherence function between the scattering field and the non-scattering field can be calculated by the phase shift algorithm (formula (3)) And the refractive index distribution of the sample can be obtained by deconvolution processing, and then the thickness distribution of the sample is obtained, thereby improving the spatial resolution and image contrast of the system.

[0088] Further, after the first camera 16 of the super-resolution fluorescence microscopy module of the embodiment collects a plurality of single-layer light slice images of the sample, the fluorescence image acquisition submodule is configured to obtain a planar image of the current thickness position, i.e., a super-resolution fluorescence image of the current position, according to the plurality of single-layer light slice images of the corresponding thickness position of the sample. Specifically, on the sample plane, the distribution expression of the sinusoidal / cosine stripe light field is as follows:

[0089]

[0090] wherein I is the average light intensity of the illumination light, △ is the stripe modulation degree, k is the spatial frequency of the stripe, and φ is the phase shift of the structured light; for simplicity, I = 1 and △ = 1 are set herein. In the imaging process, the structured light stripe only exists in the region with an axial distance of about 500 nm near the focal plane (the stripe contrast decreases with the increase of the defocus distance).

[0091] The fluorescence microscopy imaging under structured light illumination can distinguish the in-focus part and the out-of-focus part in the depth of field range of the objective lens. Specifically, the part with the structured stripe is the in-focus component, and the part without the structured stripe is the out-of-focus component. Therefore, the composition of the wide-field two-dimensional image recorded by the first camera 16 can be divided into two parts: one part is in-focus, and the other part is out-of-focus. The mathematical expression is as follows:

[0092] D(x,y) = D in (x,y) + D out (x,y) (4)

[0093] wherein D(x,y) represents the single-layer light slice image recorded by the camera under uniform wide-field illumination, D in (x,y) represents the in-focus component of the single-layer light slice image, and D out (x,y) represents the out-of-focus component of the single-layer light slice image. Under the structured light illumination, the distribution expression of the sinusoidal / cosine light field is as follows:

[0094]

[0095] For simplicity, I = 1 and △ = 1 are set herein. The structured stripe only modulates the in-focus component D in (x,y) and has no effect on the out-of-focus component D out (x,y). Therefore, the single-layer light slice image under the structured light illumination can rewrite (4) as:

[0096] D m (x,y) = D in (x,y) · {1 + cos[2πkx + 2(m-1)π / 3]} + D out (x,y) (5)

[0097] Where m = 1, 2, 3 represent the number of phase shifts. The above formula assumes that the increment of the phase shift in each phase shift operation is 2π / 3, and the corresponding phase shifts are 0, 2π / 3, and 4π / 3, respectively.

[0098] Therefore, the fluorescence image reconstruction submodule is specifically used for:

[0099] The focal component D is obtained by solving the expression of the single-layer optical slice diagram. in (x,y):

[0100]

[0101] Furthermore, to compensate for the influence of the system's point spread function on imaging resolution, the effect on the focal component D is investigated. in (x, y) is deconvolved to obtain the in-focus component after deconvolution, which is the planar image D of the sample at the current thickness position. SIM (x,y):

[0102]

[0103] in, Indicates the presence of coke component D in Fourier spectrum of (x,y) The optical transfer function (OTF) is mathematically the Fourier transform of the point spread function (PSF) of an imaging system. This represents the two-dimensional inverse Fourier transform. Here, w is the frequency domain apodization function (used to reduce artifacts in the image), and w is the Wiener coefficient, empirically chosen to be 0.2. To achieve better reconstruction results, a Gaussian apodization function is used.

[0104] By recording three intensity patterns of the sample under structured light illumination with phase shifts of 0, 2π / 3, and 4π / 3 respectively, the focused image of the sample after removing the defocused background can be solved using formulas (6) and (7). Furthermore, for ease of comparison, a wide-field pattern image under uniform illumination can be obtained by averaging D1(x,y), D2(x,y), and D3(x,y).

[0105] D wide-field (x,y)=[D1(x,y)+D2(x,y)+D3(x,y)] / 3 (8)

[0106] D1(x,y), D2(x,y), D3(x,y) respectively represent the imaging results under structured light illumination when the phase shift number m is 1, 2, 3, that is, the corresponding phase shift is 0, 2π / 3, 4π / 3 respectively.

[0107] Subsequently, the axial distance between the sample 14 and the second microscope objective 13 is adjusted by a piezoelectric translation stage, so that the second microscope objective 13 focuses on different layers in the thickness direction of the sample 14 and repeatedly obtains a plurality of wide-field two-dimensional imaging images of the sample at different axial positions, and the planar images at different axial positions can be obtained.

[0108] The present application can generate high-contrast fringe structured light in the entire imaging field of view by loading binary structured light fringe on the digital micromirror array 5 and filtering the spectrum of the illumination light (blocking the 0-order light spectrum and only allowing the spectrum of ±1 diffraction light to pass).

[0109] In summary, the super-resolution fluorescence microscopy module of the present application has extremely high modulation speed (≥9856Hz) by generating and switching binary structured light fringe on the digital micromirror array; and high-contrast fringe structured light can be generated in the entire imaging field of view by filtering the spectrum of the illumination light (blocking the 0-order light spectrum and only allowing the spectrum of ±1 diffraction light to pass). The quantitative phase microscopy module of the present application has a common-path structure of object parameters, and therefore has high measurement accuracy and stability; and the LED-based multi-angle annular super-oblique partially coherent illumination greatly improves the image quality and spatial resolution.

[0110] In addition, as described above, the image processing module of the present embodiment further comprises a deep learning sub-module, which can directly obtain a super-resolution fluorescence image of the current position of the sample using the quantitative phase image of the specific position of the sample without fluorescent labeling of the sample, thereby providing a fast and convenient method for obtaining sample information of cells. The deep learning sub-module of the present embodiment comprises a pre-trained U-Hybrid Transformer network model, and the quantitative phase image of the current position of the sample is input into the pre-trained U-Hybrid Transformer network model, so that a super-resolution fluorescence image at the current position of the sample can be obtained, thereby facilitating further segmentation of the subcellular structure of the sample. The U-Hybrid Transformer network model is trained using a training data set composed of a large number of fluorescence-phase paired images obtained by the fluorescence and phase dual-mode microscopic imaging device, wherein the fluorescence-phase paired images refer to a super-resolution fluorescence image (specific structure is labeled by fluorescence) and a quantitative phase image obtained at the same thickness position of the sample, that is, the super-resolution fluorescence image and the quantitative phase image obtained at the same thickness position of the sample by the fluorescence and phase dual-mode microscopic imaging device are used as an image pair, and a large number of such image pairs are used as a training data set to train the U-Hybrid Transformer network model, and by continuously optimizing the error between the reference fluorescence image and the predicted fluorescence image, the pre-trained U-Hybrid Transformer network model with optimal weights is finally obtained.

[0111] In particular, see Figure 2 , Figure 2is a U-Hybrid Transformer network architecture schematic diagram provided by an embodiment of the present application. The phase image obtained by the dual-mode microscopic imaging device is taken as the original data, and the fluorescence image is taken as the reference data. A large amount of data set is trained through fluorescence-phase pairing, and a pre-trained U-Hybrid Transformer network model is obtained. The phase image is input into the pre-trained U-Hybrid Transformer network model, and finally the corresponding fluorescence image is obtained. The lower left graph is the unmarked phase image of the input cos7 cell, and the right graph is the corresponding fluorescence image output by the pre-trained U-Hybrid Transformer model. That is, the pre-trained model is obtained by training the U-Hybrid Transformer network model through fluorescence-phase paired images. The phase image obtained by the quantitative phase microscopic imaging system is taken as the pre-trained model input, and the fluorescence image at the corresponding thickness position of the sample can be directly obtained without the need of fluorescence labeling on the sample. By using this method, various structures of cells can be processed, so that the analysis of the sample is more comprehensive and convenient, and it can be used for the prediction and identification of various biological tissues, and has great application value in the fields of biomedicine and the like.

[0112] See Figure 3 , Figure 3 is a detailed structure schematic diagram of the U-Hybrid Transformer network model provided by an embodiment of the present application. The present application proposes a Transformer model with a hybrid attention mechanism (U-Hybrid Transformer) in combination with the U-Net network. This is a design combining the traditional U-Net model and the attention mechanism in the Transformer. The traditional U-Net model realizes the fusion of multi-scale information through the skip connection, and the U-Hybrid Transformer of the present embodiment replaces the convolution layer on the basis of the traditional U-Net, and uses the attention mechanism of the Transformer to more flexibly capture the global relationship in the image. The Transformer module is embedded into the U-Net structure, replacing some layers in the original U-Net encoder. This design makes the U-Hybrid Transformer have stronger modeling capability.

[0113] Specifically, the U-Hybrid Transformer network model of the present embodiment is composed of an encoder (Encoder), a decoder (Decoder) and a skip connection module (Skip Connection), Figure 3Each block corresponds to a multi-channel feature map. The encoder is composed of multiple hybrid attention mechanism blocks (HTB) and non-convolutional downsampling (Patch Merging) to obtain different scale semantic features from the original phase image using the encoder of the U-Hybrid Transformer network, wherein the multiple hybrid attention mechanism modules (HTB modules) are key components in the U-Hybrid Transformer, which are used to realize multi-scale semantic feature extraction of the input phase image. Each HTB module contains three different types of attention mechanisms: window self-attention mechanism (W-MSA), sliding window self-attention mechanism (SW-MSA), and channel attention mechanism (CAB), as Figure 3As shown in the middle right figure. First, the input feature map is standardized by applying Layer Normalization (Layer Norm in the figure). Through the sliding window self-attention mechanism (SW-MSA), the model can capture the dependency between features in the local area. Then the channel attention mechanism (CAB) is applied to capture the relationship between global channels. The output of SW-MSA and CAB is added to obtain the output of the first part. Finally, the input feature map is directly added to the output of the first part to retain the original information. The second part, through LayerNorm, layer normalizes the output of the fusion stage, and then applies a multilayer perceptron (MLP) for nonlinear mapping to introduce higher-order feature transformation. Finally, the output of the MLP is added to the output of the first stage to obtain the output of the second part. This makes it possible to obtain nearly global attention capability while reducing the computational complexity from the square of the image size to a linear relationship, greatly reducing the computational complexity. The self-attention has strong representation ability, and the channel attention can use global information. The purpose of the HTB module is to combine the advantages of the two, and use a series of different governance mechanisms to retain high-frequency information and deep feature extraction. PatchMerging performs down-sampling after each feature extraction through feature fusion, increasing the receptive field of the next window attention operation on the original image, thereby performing multi-scale feature extraction on the input image. The decoder is composed of the HTB module and the up-ample convolution layer (Up-ample), which helps to learn the high-level semantic information of the image. And use the skip connection module (Skip Connection) to fuse the features of the corresponding layers of the encoder and the decoder, which aims to enable the network to learn the feature semantics at different stages to enhance the receptive field and semantic representation ability of the network, while avoiding the gradient vanishing phenomenon in network training. The U-Hybrid Transformer selects mean square error (MSE) as the loss function, which is simple and intuitive, easy to derive and optimize, and is used to measure the difference between the network output and the true label. The pre-trained U-Hybrid Transformer network model can directly obtain the corresponding fluorescence image from the phase image, and can be widely used in the prediction and identification of various biological tissues. In other embodiments, the deep learning sub-module can also use other appropriate deep learning networks.

[0114] The application is based on a fluorescence and phase dual-mode microscopic imaging device based on deep learning, which integrates super-resolution fluorescence microscopy and quantitative phase microscopy modes into the same imaging system; the fluorescence image and the phase image of the sample can be obtained simultaneously, multi-dimensional information of the same sample is provided, and great application value is obtained in the fields of biomedicine and the like. The image processing module of the application trains the U-Hybrid Transformer network by using the phase and fluorescence images as true value images, the corresponding fluorescence image can be obtained from a single phase image by using the trained network, and then dozens of subcellular structures of the sample are accurately identified and segmented.

[0115] The effectiveness of the super-resolution fluorescence microscopy and quantitative phase microscopy dual-mode imaging device and method based on deep learning is verified by experiments.

[0116] Experiment one:

[0117] Please refer to Figure 4 , Figure 4 is a schematic diagram of the results of the 0, π / 2, π, 3π / 2 phase shift operation and reconstruction of the lily anther stigma transverse sample by the quantitative phase microscopy module of the embodiment of the application. The phase shift amounts of 0, π / 2, π, 3π / 2 are sequentially loaded on the spatial light modulator 24, and four intensity images of the sample under different phase shifts are recorded. Then, the phase image is reconstructed by the phase shift algorithm of formula (3), and the imaging result is as shown in Figure 4 (e). Compared with the bright field image of Figure 4 (e) and Figure 4 (a), the phase image well shows the thickness information that cannot be displayed by the traditional bright field microscope, and the thickness information of the sample can be quantitatively obtained through the phase image and the refractive index value.

[0118] Experiment two:

[0119] Please refer to Figure 5 , Figure 5 is a schematic diagram of the results of the 0, 2π / 3, 4π / 3 phase shift operation and reconstruction of the lily anther stigma transverse sample by the super-resolution fluorescence microscopy module of the embodiment of the application. The lily anther stigma transverse sample optical section is used as the measured sample to verify the tomographic imaging capability of the optical section microscopic imaging method proposed in the application. For each axial plane position, three gray scale images with phase shift amounts of 0, 2π / 3, 4π / 3 are sequentially loaded on the digital micromirror array 5, and three intensity images D1(x, y), D2(x, y), D3(x, y) of the sample under different phase shift structured light illuminations are recorded. Then, the fluorescence microscopic images of different axial planes are obtained by the corresponding algorithms of formula (6) and (7), and the imaging result is as shown in Figure 5 (d). Compared with Figure 5Compared to the wide-field illumination images in (a)-(c), the fluorescence microscopy images of the samples effectively suppressed the defocused background, achieved tomography, and demonstrated excellent optical sectioning capabilities.

[0120] Experiment 3:

[0121] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating the experimental results of imaging 240nm fluorescent polystyrene microspheres using a deep learning-based super-resolution fluorescence microscopy and quantitative phase dual-mode microscopy imaging device. By adding phase shifts of 0, π / 2, π, and 3π / 2 to the non-scattered light, the phase image acquisition submodule acquired four phase-shifted images. The phase images reconstructed using the phase-shifting algorithm are shown below. Figure 6 As shown in (a). Considering that the refractive index n of polystyrene is approximately 1.55, and the refractive index n of the culture medium... m The value is 1.33, therefore the theoretical maximum phase change caused by the 240nm polystyrene microsphere is 2π·(nn). m )·d / λ=2π·(1.55-1.33)·240 / 470=0.71rad, while the actual measured maximum phase change is 0.7rad. Simultaneously, three grayscale images with phase shifts of 0, 2π / 3, and 4π / 3 were loaded onto the digital micromirror array 5, and three intensity images of the sample under different phase-shifted structured light illuminations were recorded. Subsequently, using the corresponding algorithms of formulas (6) and (7), the imaging results obtained are as follows: Figure 6 As shown in (b), both the first camera 16 and the second camera 27 are Hamamatsu cameras with a pixel count of 2048×2048 and a single pixel size of 6.5μm×6.5μm. The magnification is 100× in both fields of view, and the fields of view captured by the cameras are consistent. By simultaneously capturing phase shift maps of the sample in different modes, the phase map and fluorescence map under the same field of view can be obtained after numerical reconstruction. This characteristic can be used to specifically identify various cellular structures and features.

[0122] In summary, the deep learning-based super-resolution fluorescence microscopy and quantitative phase dual-mode imaging microscopy system proposed in this invention exhibits high stability and acquires high-quality images. It can simultaneously obtain both phase and fluorescence images of a sample, and through training a convergent deep neural network, it can predict the corresponding fluorescence image from the phase image. This system can be widely applied to multimodal imaging of biological tissues and the identification of cell-specific features.

[0123] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0124] In addition, each function module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a hardware plus software function module.

[0125] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application should not be limited to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, and all of them should be regarded as falling within the protection scope of the present application.

Claims

1. A fluorescence and phase dual-mode microscopic imaging device based on deep learning, characterized in that, It includes a super-resolution fluorescence microscopy module, a quantitative phase microscopy module, and an image processing module. The super-resolution fluorescence microscopy module includes a monochromatic extended illumination submodule, an illumination modulation submodule, and a fluorescence image acquisition submodule. The monochromatic extended illumination submodule is used to generate monochromatic extended illumination light with a certain angular distribution. The illumination modulation submodule is used to modulate the monochromatic extended illumination light to generate fringe structured light with different phases. The fluorescence image acquisition submodule is used to record multiple single-layer optical slices of the sample under fringe structured light illumination with different phases. The quantitative phase microscopy module includes an annular tilted illumination submodule, a microscopic imaging submodule, a beam modulation submodule, and a phase image acquisition submodule. The annular tilted illumination submodule generates annular illumination light that converges along the tilt direction onto the sample, producing unscattered light unaffected by the sample and scattered light containing sample information. The microscopic imaging submodule collects and amplifies the scattered and unscattered light from the sample. The spatial light modulator in the beam modulation submodule performs phase modulation on the unscattered light from the sample. The phase image acquisition submodule records the transmission intensity images generated by the scattered light and the unscattered light modulated by different phases. The image processing module includes a fluorescence image reconstruction submodule, a phase image reconstruction submodule, and a deep learning submodule. The fluorescence image reconstruction submodule is used to reconstruct a super-resolution fluorescence image of the sample based on multiple monolayer optical slices of the sample under striped structured light illumination at different phases. The phase image reconstruction submodule is used to reconstruct a quantitative phase image of the sample based on multiple transmission intensity images generated by nonscattered light modulated by different phases. The deep learning submodule is used to obtain a super-resolution fluorescence image of the sample at the current position using the quantitative phase image at a specific thickness position of the sample, so as to identify and segment different subcellular structures in the sample. The deep learning submodule includes a pre-trained U-Hybrid Transformer network model. The U-Hybrid Transformer network model uses the phase image obtained by the phase image reconstruction submodule as the raw data and the fluorescence image obtained by the fluorescence image reconstruction submodule as the reference data. It is trained using a large number of fluorescence-phase paired images as a dataset to obtain the pre-trained U-Hybrid Transformer network model. The fluorescence-phase paired images refer to the super-resolution fluorescence image and quantitative phase image obtained at the same thickness position of the sample. The U-Hybrid Transformer network model includes an encoder, a decoder, and a skip connection module. The encoder consists of multiple hybrid attention mechanism modules and a non-convolutional downsampling module. The hybrid attention mechanism modules are used to extract multi-scale semantic features from the input phase image, and the non-convolutional downsampling module is used to downsample the extracted multi-scale semantic features. The decoder consists of multiple hybrid attention mechanism modules and an upsampled transposed convolutional layer. The skip connection module is used to fuse features from corresponding layers of the encoder and decoder to learn semantic features at different stages.

2. The deep learning-based fluorescence and phase dual-mode microscopic imaging device according to claim 1, characterized in that, The monochromatic extended illumination submodule includes a laser (1) and a first microscope objective (2) and a first lens (3) arranged sequentially along the optical axis of the laser (1). The illumination modulation submodule includes a digital micromirror array (5), a second lens (6), a spectrum filter (7), and a third lens (8) arranged sequentially along the optical axis of the laser (1). The laser (1) is used to emit a single-frequency laser; the first microscope objective (2) and the first lens (3) are used to expand the single-frequency laser generated by the laser (1) to cover the entire working surface of the digital micromirror array (5); The digital micromirror array (5) is used to modulate the monochromatic extended illumination light from the monochromatic broad-spectrum illumination submodule to generate striped structured light with different phase shifts. The second lens (6) is used to focus the striped structured light onto the spectrum filter (7). The spectrum filter (7) is used to perform spectrum selection on the striped structured light modulated by the digital micromirror array (5), allowing only the spectrum of ±1 diffracted light to pass through. The third lens (8) is used to expand and collimate the beam that has been spectrum selected by the spectrum filter (7) into parallel illumination light.

3. The deep learning-based fluorescence and phase dual-mode microscopic imaging device according to claim 2, characterized in that, The fluorescence image acquisition submodule includes a second reflecting mirror (9), a first dichroic mirror (10), a fourth lens (11), a third reflecting mirror (12), a second microscope objective (13), a filter (15), and a first camera (16), wherein, The second reflecting mirror (9), the first dichroic mirror (10), the fourth lens (11), the third reflecting mirror (12) and the second microscope objective (13) are arranged sequentially along the optical axis of the laser (1), and the front focal plane of the second microscope objective (13) is used to set the sample (14); The filter (15) and the first camera (16) are sequentially arranged on the side of the first dichroic mirror (10) away from the fourth lens (11) to receive the light beam reflected from the sample (14) and obtain multiple single-layer light slice images of the sample (14) at the current thickness position under striped structured light illumination of different phases.

4. The deep learning-based fluorescence and phase dual-mode microscopic imaging device according to claim 3, characterized in that, The digital micromirror array (5) is used to load binary structured light stripes and sequentially perform 0, 2 / 3、4 / 3 Phase shift operation; The first camera (16) is used to perform 0, 2 / 3、4 During the / 3 phase shift operation, three corresponding single-layer optical slice images are recorded respectively.

5. The deep learning-based fluorescence and phase dual-mode microscopic imaging device according to claim 3, characterized in that, The annular tilted illumination submodule includes a light source mounting bracket (17) and multiple LEDs (18). The light source mounting bracket (17) is a ring-shaped mounting structure with multiple equal heights and concentricity. The multiple LEDs (18) are uniformly mounted on the ring-shaped mounting structure in the circumferential direction. The light source mounting bracket (17) is located on the other side of the sample away from the second microscope objective (13), which can generate annular illumination light that converges and illuminates the sample in the tilt direction. The microscopic imaging submodule includes a fourth lens (11), a second dichroic mirror (19), a fourth reflecting mirror (20), a linear polarizer (21), and a fifth lens (22) arranged sequentially along the optical path of the annular illumination light. The fourth lens (11) is located at the back focal plane of the second microscope objective (13) and is confocal with the fifth lens (22). The second dichroic mirror (19) is tilted between the fourth lens (11) and the first dichroic mirror (10) to reflect the light beam from the fourth lens (11) to the fourth reflecting mirror (20). The linear polarizer (21) is used to modulate the light beam from the fourth reflecting mirror (20) into linearly polarized light.

6. The deep learning-based fluorescence and phase dual-mode microscopic imaging device according to claim 5, characterized in that, The beam modulation submodule includes a beam splitter (23) and a spatial light modulator (24) arranged sequentially along the optical path of the annular illumination light. The phase image acquisition submodule includes a fifth reflector (25), a sixth lens (26), and a second camera (27) arranged sequentially. The beam splitter (23) is used to transmit the light beam from the fifth lens (22) to the spatial light modulator (24) and reflect the light beam modulated by the spatial light modulator (24) to the fifth mirror (25). The spatial light modulator (24) is used to cyclically apply phase modulation of 0, π / 2, π and 3π / 2 to the non-scattered light; The sixth lens (26) is disposed at the back focal plane of the spatial light modulator (24); the second camera (27) is used to receive the light beam transmitted back from the sample (14) and to obtain four transmission intensity images of the sample when the spatial light modulator (24) is loaded with phase modulation of 0, π / 2, π and 3π / 2 respectively.

7. The deep learning-based fluorescence and phase dual-mode microscopic imaging device according to claim 6, characterized in that, The transmission intensity image detected by the phase image acquisition submodule is represented as follows: ， in, Represents the spatial coordinates of the scatterer. Represents the total intensity of non-scattered light. Represents the total intensity of the scattered light. This represents the coherence function between scattered and unscattered light. This represents the average phase modulation function of the sample to the light field. =0, / 2, , and 3 / 2 represents the modulation phase loaded by the spatial light modulator; The phase image reconstruction submodule is specifically used for: The phase distribution of the sample was calculated using a phase-shifting algorithm: ， in, , , and They respectively represent the modulation phase =0, / 2, , and 3 Transmission intensity image at / 2; The mutual coherence function between scattered and non-scattered light was calculated using a phase-shifting algorithm. ; The refractive index distribution of the sample is obtained by deconvolution, and then the thickness distribution of the sample is obtained.

8. The deep learning-based fluorescence and phase dual-mode microscopic imaging device according to claim 6, characterized in that, The expression for the single-layer optical slice image obtained by the fluorescence image acquisition submodule is: m ( , ) = in ( , ) {1 + cos[2 + 2( m-1) / 3]} + out ( , ) in, m=1 , , Indicates the number of phase shifts. in ( , ) represents the focal component of a single-layer optical slice. out ( , ) represents the defocused component in a single-layer optical slice image. k The spatial frequency of the stripes; The fluorescence image reconstruction submodule is specifically used for: The focal component is solved based on the expression of the single-layer optical slice diagram. in ( , ): ； For the aforementioned coke components Perform deconvolution to obtain the in-focus component after deconvolution, which is the planar image of the sample at the current thickness position. : ， Among them, among them, Indicates the composition of the coke Fourier spectrum, Represents the optical transfer function. This represents the two-dimensional inverse Fourier transform. Let be the apodization function in the frequency domain. It is the Wiener coefficient.

Images