A dual-modality real-time microscopic imaging device and method

Abstract

This invention provides a dual-modal real-time microscopic imaging device, comprising: an illumination component configured to illuminate a sample; an imaging component configured to receive light signals from the sample and emit scattered light signals and fluorescence signals via different optical paths; a full-field camera configured to receive scattered light signals emitted from the imaging component; a fluorescence imaging two-dimensional array detector and a GISC camera configured to switchably receive fluorescence signals emitted from the imaging component; and an image processing processor electrically connected to the full-field camera, the fluorescence imaging two-dimensional array detector, and the GISC camera. The dual-modal real-time microscopic imaging device and method of this invention solve the problems of low spatial resolution, inability to simultaneously perform real-time dual-modal imaging, and difficulties in multi-color real-time imaging in existing dual-modal phase and fluorescence imaging methods.

Description

Technical Field

[0001] This invention relates to the field of microscopic imaging, and more specifically to a dual-modal (phase / fluorescence) real-time microscopic imaging device and method. Background Technology

[0002] Optical microscopy, with its advantages of being non-destructive, non-contact, highly specific, highly sensitive, biocompatible, and capable of providing dynamic imaging of functional information, has become the preferred method for imaging living organisms and cells. Since the 2008 Nobel Prize in Chemistry was awarded to Osamu Shimomura, the Japanese scientist who discovered fluorescent proteins, and Marty Chalfie and Roger Tsien, the American scientists who made outstanding contributions, fluorescent labeling technology has seen significant development and widespread application in biomedicine. In particular, the combination of fluorescence microscopy, fluorescence confocal microscopy, and recently developed super-resolution fluorescence microscopy techniques such as stochastic optical reconstruction super-resolution imaging (STORM) and structured illumination imaging (SIM) has extended its application to all aspects of biomedical research in the dynamic in vivo labeled microscopic observation of biomarkers and target molecules. However, the combination with wide-field super-resolution imaging techniques still has limitations in terms of the real-time performance of multicolor labeled imaging.

[0003] STORM technology based on single-molecule localization has the advantages of high spatial resolution and large field of view, but the requirement for high-precision single-molecule localization limits the density of labeled fluorescent dye molecules. To reconstruct a super-resolution image, it usually requires sampling thousands to tens of thousands of images. The temporal resolution is too low to achieve real-time imaging.

[0004] The temporal resolution of SIM technology is limited only by the detector's sampling frame rate, and it can achieve a large field of view and low illumination intensity, giving it a certain leading advantage in temporal resolution, field of view, and illumination intensity. However, its spatial resolution is limited to 100nm by the imaging mechanism. SIM technology can achieve real-time monochromatic imaging, but it requires structured light illumination and multi-frame reconstruction imaging.

[0005] To avoid multicolor crosstalk, current multicolor imaging mainly uses multi-wavelength laser time-division excitation in conjunction with rotating filters. However, the fastest switching speed of rotating filters is currently 35-50ms, which seriously affects the real-time performance of multicolor imaging of living biological cells.

[0006] Currently, the inventors' team has developed a wide-field fluorescence super-resolution microscopy imaging technique based on sparsely constrained ghost imaging (GISC). This technique combines quantum imaging technology with compressed sensing technology from modern information theory, offering higher image information acquisition efficiency compared to SIM and STORM technologies. This allows for continuous improvement in the spatial resolution, temporal resolution, and triangulation space of the field of view in super-resolution fluorescence microscopy. Furthermore, the technique has been demonstrated to achieve single-frame super-resolution imaging with a resolution of 80nm. Similarly, when combined with STORM technology, the frame rate required for STORM imaging can be reduced by more than a hundredfold. In particular, GISC super-resolution imaging technology features modulation-coded imaging, enabling the integration of multi-color coding technology to achieve single-frame multi-color super-resolution imaging, thus solving the problem of slow multi-color imaging speed. The GISC imaging technology has been granted a national patent (Chinese patent number: ZL201510394995.9); at the same time, a nucleic acid detection and gene sequencing scheme based on GISC imaging technology has been proposed and granted a national patent (Chinese patent number: ZL202010982860.5), and an international patent application has been filed for PCT / CN2021 / 114636.

[0007] Furthermore, real-time observation of live-cell imaging and biochips using optical microscopy requires not only functional observation after fluorescent labeling but also label-free dynamic evolution observation of the entire cell. Currently, label-free microscopic observation of transparent live cells mainly relies on various optical phase microscopy imaging techniques, including interferometric phase imaging techniques such as phase-contrast microscopy, differential interferometry microscopy, and holographic microscopy. However, obtaining true morphological information of cells requires quantifying the detected phase, typically necessitating illumination imaging with multiple preset phases and multiple reconstructions to obtain quantitative phase information. This makes real-time observation difficult with interferometric phase imaging. Another type of technology is phase retrieval algorithms based on coherent diffraction, mainly including coherent diffraction imaging (CDI) and Fourier layered imaging (FPM). To increase the convergence of phase retrieval algorithms, multiple angle illuminations and layered imaging are usually required, followed by aperture synthesis to recover the phase. However, this also suffers from limitations in real-time phase imaging.

[0008] The inventor team has independently developed a full-field camera imaging technology. Through strong constraints on the real image plane and the Fourier plane, the phase recovery of the object image is achieved, thereby obtaining the phase information of the light field. It also has the ability to achieve phase recovery imaging with single-frame sampling, which enables optical phase microscopy imaging to achieve real-time performance. The core patent has been applied for (application number: 202111548124.X).

[0009] In the field of dual-modal microscopy, in 2020, the teams of Chen Liangyi and Shi Kebin at the Institute of Molecular Medicine, Peking University, collaborated to combine label-free optical diffraction tomography (ODT) with SIM fluorescence imaging technology to achieve a novel dual-modal super-resolution microscope (SR-FACT), and applied for a related patent with publication number CN111610621B. The fluorescence super-resolution is better than 100 nanometers, while the label-free three-dimensional imaging resolution is approximately 200 nanometers in the XY direction and 500 nanometers in the Z direction. The temporal resolution of dual-modal live-cell super-resolution imaging can reach 0.8 Hz, but real-time imaging is not yet possible.

[0010] Meanwhile, Shi Kebin's team at Peking University also applied for a patent related to dual-modal microscopy imaging, with publication number CN113702288B, combining two-photon fluorescence imaging and FPM technology.

[0011] In summary, existing dual-modal imaging technologies are based on ODT and SIM or two-photon fluorescence imaging. However, dual-modal super-resolution microscopy (SR-FACT) requires multi-angle illumination and multi-frame acquisition to achieve phase imaging, while SIM technology requires complex modulation of the illumination source and can only improve resolution by twice the diffraction limit. Therefore, existing dual-modal imaging technologies suffer from drawbacks such as long imaging time, complex instrument structure, low resolution, and inability to perform synchronous real-time imaging.

[0012] The all-light field camera eliminates the need for multi-angle illumination and multi-frame acquisition. It achieves phase imaging by simultaneously acquiring intensity information from the real image plane and Fourier transform plane using only two two-dimensional array detectors. The advantage of GISC multicolor coded super-resolution microscopy technology lies in its ability to achieve super-resolution imaging without modulating the illumination source, possessing a single-frame super-resolution capability of 80nm, and completing multicolor imaging in a single exposure.

[0013] Therefore, it is necessary to propose a dual-modal microscopy imaging technology that integrates full-field camera technology and GISC multicolor coding super-resolution microscopy technology to solve the problems of slow imaging time, complex structure and low resolution of existing dual-modal imaging, and to achieve real-time imaging. Summary of the Invention

[0014] The purpose of this invention is to provide a dual-modal real-time microscopic imaging device and method, thereby solving the problems of low spatial resolution, inability to simultaneously perform real-time dual-modal imaging, and difficulty in multi-color real-time imaging in existing dual-modal phase and fluorescence imaging.

[0015] To achieve the above objectives, the present invention provides a dual-modal real-time microscopic imaging device, comprising: an illumination component configured to illuminate a sample to be tested; an imaging component configured to receive light signals from the sample to be tested and to emit scattered light signals and fluorescence signals through different optical paths; a full-field camera configured to receive scattered light signals emitted from the imaging component; a fluorescence imaging two-dimensional array detector and a GISC camera configured to switchably receive fluorescence signals emitted from the imaging component; and an image processing processor electrically connected to the full-field camera, the fluorescence imaging two-dimensional array detector, and the GISC camera.

[0016] The lighting assembly includes multiple excitation light sources, an optical fiber combiner connected to the multiple excitation light sources, and an optical fiber collimator connected to the optical fiber combiner via a single optical fiber.

[0017] The illumination component is configured to emit monochromatic light through one of a plurality of excitation light sources, or to emit polychromatic light through a plurality of excitation light sources in a sequential or simultaneous manner; and to combine the laser output of the excitation light source into a single optical fiber through an optical fiber combiner, and to collimate the sample to be tested through the optical fiber collimator.

[0018] The sample to be tested is mounted on a displacement stage, which is a two-dimensional displacement stage.

[0019] The full-light field camera is configured to split the scattered light signal using a beam splitter. One of the scattered light signals is directly detected by the two-dimensional array detector of the first full-light field camera on the imaging plane, and the other is detected by the two-dimensional array detector of the second full-light field camera on the back focal plane after being transformed by the imaging lens of the full-light field camera. The phase of the scattered light signal of the object is then obtained by reconstructing and recovering it using a Fourier iterative algorithm.

[0020] The GISC camera includes a random phase modulator placed at a first distance Z1 from the imaging surface of the fluorescence signal, and a two-dimensional array detector of the GISC camera placed at a second distance Z2 after the random phase modulator to record the fluorescence speckle signal. The first distance Z1 and the second distance Z2 satisfy the optimal condition for speckle detection. The GISC camera is configured to reconstruct a multicolor super-resolution fluorescence signal by performing correlation operations with the obtained fluorescence speckle signal and a preset calibration matrix.

[0021] The imaging component is configured to emit the scattered light signal into a first optical path, and the full-field camera is located in the first optical path; the imaging component is configured to switchably emit the fluorescence signal into a second optical path and a third optical path, the fluorescence imaging two-dimensional array detector is located in the second optical path, and the GISC camera is located in the third optical path.

[0022] The imaging assembly includes an objective lens, a dichroic mirror, and a switching mirror arranged sequentially along the optical path. The dichroic mirror is configured to reflect the scattered light signal in the optical signal and emit it into the first optical path, while allowing the fluorescence signal in the optical signal to pass through and emit into the second optical path. The switching mirror is configured to reflect the fluorescence signal propagating along the second optical path to the third optical path when rotated to the first position, and to allow the fluorescence signal propagating along the second optical path to continue propagating along the second optical path when rotated to the second position. The switching mirror is located on the second optical path.

[0023] The imaging component is configured to emit the scattered light signal into a first optical path, and the full-field camera is located on the first optical path; the imaging component is configured to emit the fluorescence signal into a second optical path, and the fluorescence imaging two-dimensional array detector and the GISC camera can be switched to be located on the second optical path.

[0024] The image processing processor has a data processing module; the image processing processor is configured as follows:

[0025] S1: Outputs signals acquired synchronously by the full-field camera and GISC camera or the full-field camera and fluorescence imaging two-dimensional array detector;

[0026] S2: Caches the collected data and transmits it to the data processing module of the image processing processor;

[0027] S3: The pre-built full-field phase recovery algorithm and GISC super-resolution algorithm in the data processing module are used to calculate and reconstruct the acquired signal to obtain the reconstruction result;

[0028] S4: Output the reconstruction result.

[0029] On the other hand, the present invention provides a method for dual-modal imaging of phase and fluorescence microscopy under monochromatic illumination, comprising:

[0030] A0: Construct a dual-modal real-time microscopic imaging device as described above;

[0031] A1: By using the illumination component to emit monochromatic light, the fluorescence imaging two-dimensional array detector is switched to receive the fluorescence signal emitted from the imaging component;

[0032] A2: Using an image processing processor, the full-field camera and the fluorescence imaging two-dimensional array detector are controlled to synchronously acquire signals, the acquired data is buffered, and the acquired data is reconstructed using a data processing module to obtain the reconstruction result; then the reconstruction result is output.

[0033] On the other hand, the present invention provides a method for super-resolution dual-modal imaging of phase and GISC fluorescence under monochromatic illumination, comprising:

[0034] B0: Construct a dual-modal real-time microscopic imaging device as described above;

[0035] B1: By using the illumination component to emit monochromatic light, the GISC camera is switched to receive the fluorescence signal emitted by the self-imaging component;

[0036] B2: Using an image processing processor, control the full-field camera and GISC camera to synchronously acquire signals, buffer the acquired data, and use the data processing module to calculate and reconstruct the acquired data to obtain the reconstruction result; then output the reconstruction result.

[0037] On the other hand, the present invention provides a method for super-resolution dual-modal imaging of full-color phase and GISC multicolor fluorescence under multicolor sequential illumination, comprising:

[0038] C0: Construct a dual-modal real-time microscopic imaging device as described above;

[0039] C1: By using the illumination component to emit monochromatic light of different wavelengths in sequence, the GISC camera is switched to receive the fluorescence signal emitted by the imaging component.

[0040] C2: Using an image processing processor, the system controls the full-field camera and the GISC camera to synchronously acquire signals in a time sequence, buffers the acquired data, and uses the data processing module to calculate and reconstruct the time sequence data to obtain a time sequence image as the reconstruction result; then outputs the reconstruction result; and synthesizes a full-color phase image and a multi-color fluorescence image based on the time sequence image.

[0041] On the other hand, the present invention provides a method for super-resolution dual-modal imaging of full-color phase and GISC multicolor fluorescence under simultaneous multicolor illumination, comprising:

[0042] D0: Construct a dual-modal real-time microscopic imaging device as described above; wherein, the first full-light field camera two-dimensional array detector and the second full-light field camera two-dimensional array detector in the full-light field camera of the dual-modal real-time microscopic imaging device are both RGB color array detectors, so that the full-light field camera constitutes a color full-light field camera.

[0043] D1: By using the illumination component to simultaneously emit multicolor light, the GISC camera is switched to receive the fluorescence signal emitted by the self-imaging component;

[0044] D2: Using an image processing processor, the color full-field camera and the GISC camera are controlled to synchronously acquire signals, and the acquired data is buffered. Using the image processing processor, the data output by the color full-field camera is decomposed into RGB three-color channels to obtain the decomposed three-color channel signals. The data processing module is used to calculate and reconstruct the decomposed three-color channel signals respectively to obtain the reconstruction results. The reconstruction results are then superimposed to generate a full-color phase image. Using the image processing processor, the data output by the GISC camera is calculated and reconstructed to obtain a multicolor fluorescence image. Thus, a full-color phase image and a multicolor fluorescence image are obtained.

[0045] The dual-modal real-time microscopic imaging device and method of this invention integrates full-field camera imaging technology and GISC multicolor coded super-resolution microscopy technology, employing a unified fiber-optic bundled collimation wide-field illumination method. Specifically, the phase imaging method uses double-sided constrained Fourier iterative phase retrieval, requiring only a single exposure to recover phase information. The fluorescence super-resolution imaging method uses GISC technology, achieving a spatial resolution of 80 nm under wide-field illumination. Therefore, by integrating full-field camera imaging technology and GISC super-resolution microscopy technology, this invention replaces existing dual-modal imaging methods that require modulation of the excitation light and multi-angle illumination, making real-time imaging of biological multicolor microscopy difficult. It features a simple design, compact structure, low cost, eliminates the need for excitation light modulation and multi-angle illumination, provides high-resolution imaging capability in a single frame, and offers better real-time performance.

[0046] Existing phase imaging methods often require more than twice the Nyquist sampling rate to recover the phase. The dual-modal real-time microscopic imaging method and apparatus of this invention utilizes a full-field camera module. This module splits the scattered signal using a beam splitter within the full-field camera, simultaneously acquiring intensity signals from both the real image plane and the Fourier surface. Through the dual-surface constraints of the real image plane and the Fourier surface, sufficient information can be obtained to solve for the phase and perform phase recovery. Phase object reconstruction imaging can be achieved with a sampling rate close to one Nyquist, resulting in a lower sampling rate for the full-field camera compared to traditional methods. Furthermore, the acquisition of signals from the real image plane and the Fourier surface through lens transformation not only creates dual-surface intensity constraints but also introduces prior constraints inherent in the imaging system itself (such as the resolving constraints of the real image plane due to the Fourier transform properties of the lens, the band-limiting constraints of the Fourier surface due to the truncation of the Fourier surface by the objective lens, and the bounded constraints imposed by the object's size on the real image plane).

[0047] Existing dual-modal fluorescence imaging often requires switching filters to achieve multicolor imaging. This invention uses an optical fiber combiner to couple light of different wavelengths to a single optical fiber, enabling simultaneous illumination of multiple colors. Furthermore, the GISC multicolor super-resolution imaging camera module can compress and encode information from the high-dimensional space of the light field into a two-dimensional array detector. This method allows for multicolor imaging in a single exposure, significantly reducing the sampling time required for multicolor imaging and providing single-frame multicolor imaging capability. This provides the best technical approach for real-time multicolor biological imaging.

[0048] In summary, the dual-modal real-time microscopic imaging device and method of the present invention integrate the phase recovery capability of full-field camera imaging technology and the super-resolution imaging capability of GISC super-resolution microscopy technology. Combined with multicolor illumination, it achieves a highly efficient fluorescence excitation method for single-exposure multicolor imaging. It has advantages such as effectively reducing the sampling requirements of the detector, simplifying the optical path, and improving the efficiency of imaging information acquisition. It provides a powerful imaging means for multicolor labeled and label-free dual-modal real-time biological (fluorescence, phase) imaging. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a dual-modal real-time microscopic imaging device according to an embodiment of the present invention.

[0050] Figure 2 This is a diagram showing the module composition of a GISC camera.

[0051] Figure 3 This is a schematic diagram of the phase and fluorescence microscopy dual-modal imaging method under monochromatic illumination.

[0052] Figure 4 This is a schematic diagram of the phase and GISC fluorescence super-resolution imaging method under monochromatic illumination.

[0053] Figure 5 This is a schematic diagram of the principle of the full-color phase and GISC fluorescence super-resolution dual-modal imaging method under time-series multicolor illumination.

[0054] Figure 6 This is a schematic diagram of the principle of the full-color phase and GISC fluorescence super-resolution dual-modal imaging method under multi-color simultaneous illumination. Detailed Implementation

[0055] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0056] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0057] First embodiment: Dual-modal real-time microscopic imaging device

[0058] like Figure 1 The diagram shown is a structural schematic of a dual-modal real-time microscopic imaging device according to an embodiment of the present invention. Figure 1 As shown, the dual-modal real-time microscopic imaging device includes an illumination component (which includes multiple laser light sources 1, an optical fiber combiner 2 connected to multiple excitation light sources, and an optical fiber collimator 3 connected to the optical fiber combiner 2 via a single optical fiber 2') for illuminating the sample 4 under test; an imaging component (which includes an objective lens 5, a dichroic mirror 6, a first reflecting mirror 81, and a switching mirror 9) for receiving light signals from the sample 4 under test and emitting scattered light signals and fluorescence signals through different optical paths; a full-field camera 7 for receiving scattered light signals emitted from the imaging component (specifically, the dichroic mirror 6 of the imaging component) and preferably receiving the scattered light signals through the imaging lens 111 of the imaging component; a fluorescence imaging two-dimensional array detector 10 and a GISC camera 12 for switching between receiving fluorescence signals emitted from the imaging component (specifically, the dichroic mirror 6 of the imaging component); and an image processing processor 13 electrically connected to the full-field camera 7, the fluorescence imaging two-dimensional array detector 10, and the GISC camera 12.

[0059] The illumination component is configured to emit monochromatic light through one of multiple excitation sources, or to emit multicolor light sequentially or simultaneously through multiple excitation sources. The laser output from the excitation sources is combined into a single optical fiber via an optical fiber combiner, and then collimated to illuminate the sample under test via the optical fiber collimator. Specifically, when the illumination component emits multicolor light, after the excitation light illuminates the sample under test 4, the excitation light forms scattered light on the sample 4. Simultaneously, the efficient multi-wavelength excitation method can simultaneously excite multicolor fluorescence. Combined with the beam splitting effect of the dichroic mirror described below, this enables the simultaneous collection of dual-mode signals of multicolor fluorescence and scattered light.

[0060] The illumination assembly includes multiple excitation light sources 1, an optical fiber combiner 2 connected to the multiple excitation light sources 1, and an optical fiber collimator 3 connected to the optical fiber combiner 2 via a single optical fiber 2'. The excitation light sources 1 include, but are not limited to, continuous or pulsed output coherent lasers with center wavelengths of 488nm, 532nm, 639nm, etc., with phase imaging and fluorescence imaging sharing the same light source. The excitation light source must satisfy certain spatial and temporal coherence requirements. Insufficient coherence will result in the light source used for phase imaging being insufficient to form a good diffraction pattern on the Fourier surface; therefore, a laser or LED with good coherence must be used as the excitation light source. The excitation light source 1 is configured to emit excitation light, which can be used to illuminate the sample 4 under test, generating a scattered light signal and simultaneously exciting the fluorescent label (such as fluorescent dye, fluorescent marker, autofluorescent protein, autofluorescent biomolecule, autofluorescent tissue, etc.) of the sample 4 to emit a fluorescence signal. The scattered light excited by the illumination source is used for label-free phase imaging.

[0061] The fiber combiner 2 is configured to output the laser from the excitation source 1 into a single fiber. The fiber collimator 3 is located at the end of the single fiber and is configured to irradiate the sample 4 to be tested. The single fiber can be a multimode fiber, a single-mode fiber, or a polarization-maintaining fiber. The output end of the single-mode fiber or the polarization-maintaining fiber is equipped with a suitable fiber collimator 3 according to the requirements of the bright field of view, so as to obtain illumination that meets the requirements of the illumination field of view.

[0062] The fiber collimator 3 is configured to collimate the outgoing light from a single fiber 2' and illuminate the sample 4 to be tested in the form of collimated light and wide-field illumination.

[0063] The sample 4 to be tested is mounted on the displacement stage 14. The displacement stage 14 is a two-dimensional displacement stage with a mounting position for the sample 4 to be tested, and the light source assembly is aligned with the sample 4 to be tested. The displacement stage 14 can be a micrometer-scale or nanometer-scale displacement stage, or a combination of both, to meet different experimental scenarios, depending on the requirements of the microscopic imaging field of view and resolution.

[0064] The imaging component is located on the side of the sample 4 away from the illumination component. It is configured to receive the light signal from the sample 4 through the objective lens 5 and emit the scattered light signal and the fluorescence signal from the light signal to different optical paths respectively. In this embodiment, the dichroic mirror 6 in the imaging component is configured to emit the scattered light signal to the first optical path, and the full-field camera 7 is located on the first optical path; the switching mirror 9 in the imaging component is configured to switchably emit the fluorescence signal to the second optical path and the third optical path, the fluorescence imaging two-dimensional array detector 10 is located on the second optical path, and the GISC camera 12 is located on the third optical path.

[0065] The imaging assembly includes, but is not limited to, an objective lens 5 and imaging lenses (such as a first imaging lens 111, a second imaging lens 112, and a third imaging lens 113) that are matched with the objective lens 5, as well as auxiliary optical elements such as a dichroic mirror 6, a filter, and a switching mirror 9. In this embodiment, the imaging assembly includes an objective lens 5, a dichroic mirror 6, a first reflecting mirror 81, and a switching mirror 9 arranged sequentially along the optical path. In some other embodiments, the first reflecting mirror 81 may be omitted. The dichroic mirror 6 is configured to reflect the scattered light signal in the optical signal and exit into the first optical path, and to allow the fluorescence signal in the optical signal to pass through and exit into the second optical path. The full-field camera 7 is located on the first optical path. The switching mirror 9 is configured to: when rotated to the first position, reflect the fluorescence signal propagating along the second optical path to the third optical path; and when rotated to the second position, allow the fluorescence signal propagating along the second optical path to continue propagating along the second optical path. Accordingly, the first reflector 81, the switching mirror 9, and the fluorescence imaging two-dimensional array detector 10 are all located in the second optical path, while the GISC camera 12 is located in the third optical path. Thus, when the switching mirror 9 is rotated to the second position, the fluorescence signal enters the GISC camera 12; when it is rotated to the first position, the fluorescence signal directly enters the fluorescence imaging two-dimensional array detector 10 for wide-field fluorescence microscopy imaging.

[0066] A first imaging lens 111 is disposed between the full-field camera 7 and the dichroic mirror 6 (i.e., in front of the full-field camera 7). A second imaging lens 112 is disposed between the dichroic mirror 6 and the fluorescence imaging two-dimensional array detector 10 (i.e., in front of the fluorescence imaging two-dimensional array detector 10). A second reflecting mirror 82 and a third imaging lens 113 are disposed between the dichroic mirror 6 and the GISC camera 12 (i.e., in front of the GISC camera 12). Furthermore, a bandpass filter may be disposed between the dichroic mirror 6 and the first imaging lens 111, the second imaging lens 112, and the third imaging lens 113.

[0067] In other embodiments, similar to this embodiment, the imaging component is configured to emit the scattered light signal into a first optical path, and the full-field camera 7 is located on the first optical path. The difference from this embodiment is that the imaging component can also be configured to emit the fluorescence signal into a second optical path, where the fluorescence imaging two-dimensional array detector 10 and the GISC camera 12 can be switched between each other on the second optical path; that is, one of the fluorescence imaging two-dimensional array detector 10 and the GISC camera 12 is removed from the second optical path, and the other is switched onto the second optical path. Accordingly, the imaging component includes an objective lens 5 and a dichroic mirror 6 arranged sequentially along the direction of the optical path.

[0068] Thus, the all-light field camera 7 receives the scattered light signal emitted from the first imaging component 111, and the fluorescence imaging two-dimensional array detector 10 and the GISC camera 12 can switch between receiving the fluorescence signal emitted from the second imaging component 112 or the third imaging component 113.

[0069] The fluorescence imaging two-dimensional array detector 10 can be a high sampling rate CCD camera, EMCCD camera, CMOS camera, sCMOS camera or full-color camera.

[0070] The full-light field camera 7 can be referred to in the patent document with application number 202111548124.X related to full-light field cameras. The full-light field camera 7 includes a full-light field camera beamsplitter, a first full-light field camera two-dimensional array detector arranged in sequence, and the full-light field camera beamsplitter, a full-light field camera imaging lens, and a second full-light field camera two-dimensional array detector arranged in sequence in another beam-splitting direction of the full-light field camera beamsplitter.

[0071] Therefore, the full-light field camera 7 is configured to split the scattered light signal using a beam splitter. One of the split scattered light signals is directly detected by the first full-light field camera's two-dimensional array detector on the imaging plane to acquire the real image plane intensity signal; the other signal is transformed by the full-light field camera's imaging lens and detected by the second full-light field camera's two-dimensional array detector at its back focal plane to acquire the Fourier surface intensity signal. Subsequently, the phase of the scattered light signal of the object is obtained through reconstruction and recovery using a Fourier iterative algorithm. Thus, by utilizing the Fourier transform principle of the lens, placing the first and second full-light field camera's two-dimensional array detectors on the real image plane and Fourier surface respectively, and simultaneously acquiring the intensity information from both planes, phase recovery is achieved using a full-light field phase retrieval algorithm. Due to the strong constraints of the real image plane and Fourier surface, single-frame phase retrieval imaging is achieved, thereby realizing real-time phase imaging.

[0072] Full-field phase retrieval algorithms for full-field cameras include, but are not limited to, the Gerchberg-Saxton (GS) algorithm, the Hybrid Input-Output (HIO) algorithm, and the Yang-Gu (YG) algorithm, among other Fourier iterative phase retrieval algorithms. The input data for a full-field phase retrieval algorithm consists of the intensity signals of the scattered signal on the real image plane and the intensity signals on the Fourier surface (i.e., double-sided constraint, intensity constraint acquired from both sides), and the output data is the phase of the object under test.

[0073] The two-sided prior constraints incorporated by the all-light-field phase retrieval algorithm include, but are not limited to: Fourier band-limiting constraints limited by the objective aperture, band-limiting constraints on the object's own size (i.e., the size of the sample image on the real image plane), and analytical constraints (Cauchy-Riemann equations, Shannon interpolation, Kramers-Kronig relations). These two-sided prior constraints are all inherent constraints of the physical system in this invention (image size of the real image plane, Fourier band-limiting, sparsity constraints, and analytical properties). These prior constraints provide inherent constraints on the physical system (image size, Fourier band-limiting, sparsity constraints, and analytical properties) for the phase retrieval algorithm. In addition to measuring the intensity of the real image plane and the Fourier plane, the physical design of the all-light-field camera can provide band-limiting constraints on the Fourier plane and analytical constraints on the real image plane. Band-limiting constraints can reduce the number of equations and ensure uniqueness at low sampling rates, while analytical constraints on the real image plane can accelerate the convergence process and further optimize the phase reconstruction quality.

[0074] like Figure 2 As shown, the GISC camera 12 includes a random phase modulator 121 placed at a first distance Z1 from the imaging plane of the fluorescence signal and a GISC camera two-dimensional array detector 122 placed at a second distance Z2 after the random phase modulator. Light entering the GISC camera 12 reaches the random phase modulator 121 after the first distance Z1 and reaches the GISC camera two-dimensional array detector 122 after the second distance Z2. The distances Z1 and Z2 should be adjusted to meet the optimization requirements of the speckle contrast after modulation. In this embodiment, the GISC camera 12 is a GISC multicolor super-resolution camera. Therefore, the GISC camera 12 uses the random phase modulator 121 to randomly compress and encode fluorescence signals of different wavelengths to obtain an encoded signal. The encoded signal is the fluorescence speckle signal formed after the random phase modulator 121 randomly modulates the incident light field. The GISC camera two-dimensional array detector 122 collects the encoded signal, that is, records the fluorescence speckle signal. The GISC camera is configured to reconstruct the multicolor super-resolution fluorescence signal by performing correlation operations with the obtained fluorescence speckle signal and a preset calibration matrix.

[0075] The GISC super-resolution algorithm involves performing correlation operations between the coded signal and the pre-calibrated system matrix to obtain the super-resolution imaging result. The system matrix is ​​a matrix formed by normalizing and vectorizing the speckle intensity distributions of different locations and wavelengths corresponding to the random codes pre-calibrated by the GISC system.

[0076] The correlation operations include, but are not limited to, sparse constraints, compressed sensing, least squares fitting, maximum likelihood estimation, and deep learning. The correlation operation requires a speckle field Y acquired during imaging, a pre-calibrated matrix A, and a multicolor super-resolution fluorescence image X to be reconstructed, all satisfying the matrix relationship Y = AX. The correlation algorithm uses the aforementioned sparse constraints, compressed sensing, and other algorithms to inversely solve for X from Y.

[0077] There are several methods for calibrating the system matrix, including:

[0078] ① Stage Calibration: Place a monochromatic fluorescent ball on the sample stage and move the stage according to the imaging accuracy. Each time the stage is moved, the GISC camera acquires a fluorescence speckle signal. After traversing all positions within the field of view, change the wavelength of the monochromatic fluorescent ball and repeat the above steps until the requirements for imaging polychromaticity are met. All acquired fluorescence speckle signals are normalized, vectorized, and matrixed to form a calibration matrix A.

[0079] ② Computational Imaging Calibration: A known phase distribution is obtained through the design and fabrication of a multicolor random phase modulator. Based on wave optics and geometric optics theories, a full-link simulation model of GISC microscopy is established to obtain the fluorescence light field distribution after random phase modulation. Then, through extensive high-precision simulation calculations, a high-precision calibration matrix is ​​constructed. This method can effectively reduce calibration errors caused by system and light source instability during actual calibration.

[0080] ③ Phase measurement and calibration: Quantitative phase imaging technology is used to perform high-precision measurement of the phase distribution of the phase modulator, reducing the phase error introduced by the manufacturing error, obtaining the accurate phase distribution of the random phase modulator, thereby correcting the GISC microscopic full-link simulation model to make it more consistent with the real system and improve the accuracy of the calibration matrix.

[0081] ④ Deep Learning Calibration: Using multicolor fluorescence signals as X and speckle field signals acquired by the GISC system as Y, a training set is established by collecting massive amounts of fluorescence imaging data. A functional mapping relationship between X and Y is established based on the GISC system parameters, i.e., Y = f(X|θ). By solving for θ using the backpropagation algorithm, the functional relationship between the multicolor fluorescence signals and the speckle field can be determined, thus completing the calibration.

[0082] The GISC super-resolution algorithm and the full-field phase retrieval algorithm can run in the image processing processor 13. The image processing processor 13 may include various miniaturized, parallel-capable high-performance computing devices such as computers, FPGA chips, or microcontrollers. The image processing processor 13 has a data processing module, which may be a built-in Fourier iteration module or an on-chip multi-core MCU within the FPGA. The data processing module is configured to execute the pre-stored full-field phase retrieval algorithm and the GISC super-resolution algorithm to reconstruct the image that needs to be reconstructed. It should be noted that the wide-field fluorescence imaging results acquired by the fluorescence imaging two-dimensional array detector 10 do not require algorithmic image reconstruction; what you see is what you get. Therefore, the image processing processor 13 does not need to reconstruct them.

[0083] The specific execution flow of the image processing processor 13 is as follows:

[0084] Step S1: Output the signals acquired synchronously by the full-field camera 7 and the GISC camera 12, or the full-field camera 7 and the fluorescence imaging two-dimensional array detector 10;

[0085] Step S2: (Image processing processor 13) Buffer the acquired data and transmit it to the data processing module of the image processing processor 13;

[0086] Step S3: The acquired signal is reconstructed using either the pre-built full-field phase retrieval algorithm or the GISC super-resolution algorithm in the data processing module to obtain the reconstruction result. It should be noted that the wide-field fluorescence imaging results acquired by the fluorescence imaging two-dimensional array detector 10 do not require algorithm reconstruction; the intensity distribution detected by the detector is the desired result. Therefore, if the wide-field fluorescence imaging method is not needed, step S3 is not required.

[0087] Step S4: Output the reconstruction result to achieve real-time dual-modal imaging. In this embodiment, the reconstruction result is output to a display device.

[0088] Therefore, a multicolor or monochromatic coherent excitation source 1 is coupled to a single optical fiber using an optical fiber combiner 2. An optical fiber collimator 3 is integrated at the output end of the single optical fiber to design an illumination field for illuminating the sample. After the excitation light illuminates the sample 4, it generates scattered light, simultaneously exciting the fluorescence signal generated by the multicolor fluorescent label in the sample. Subsequently, the scattered light and fluorescence signal are simultaneously collected by the objective lens 5. A dichroic mirror 6 separates the scattered light from the fluorescence signal; the scattered signal is reflected by the dichroic mirror 6, while the fluorescence signal passes through it. The scattered signal is imaged onto a full-field camera 7 via a first imaging lens 111 for phase imaging. The fluorescence signal is imaged onto a GISC camera 12 via a third imaging lens 113 for single-frame multicolor super-resolution imaging. In addition, the fluorescence signal can also be imaged onto a two-dimensional fluorescence imaging array detector 10 via a switching mirror 9 and a second imaging lens 112 for conventional wide-field fluorescence microscopy.

[0089] The second embodiment is a dual-modal imaging method using phase and fluorescence microscopy under monochromatic illumination.

[0090] like Figure 3 As shown, the dual-modal imaging method of phase and fluorescence microscopy under monochromatic illumination includes the following steps:

[0091] Step A0: Construct the dual-modal real-time microscopic imaging device described above;

[0092] As described above, the dual-modal real-time microscopic imaging device includes an illumination assembly (including an excitation source 1, an optical fiber 2', and an optical fiber collimator 3) for illuminating the sample 4; an imaging assembly (including an objective lens 5, a dichroic mirror 6, a switching mirror 9, a first imaging lens 111, a second imaging lens 112, a third imaging lens 113, and a first reflecting mirror 81) for receiving light signals from the sample 4; a full-field camera 7 for receiving scattered light signals emitted from the imaging assembly; a fluorescence imaging two-dimensional array detector 10 for switching between receiving fluorescence signals emitted from the imaging assembly; and an image processing processor 13 electrically connected to the full-field camera 7, the fluorescence imaging two-dimensional array detector 10, and the full-field camera 7. The illumination assembly includes an excitation source 1, a single optical fiber 2' connected to the excitation source 1, and an optical fiber collimator 3. However, since the illumination assembly emits monochromatic light, and since it is a single-wavelength excitation light, only the single excitation source 1 needs to be turned on. Therefore, compared to… Figure 1 , Figure 3 Only one excitation source 1 is shown.

[0093] In this embodiment, the fluorescence imaging two-dimensional array detector 10 is an sCMOS camera. In order to detect weak fluorescence signals, a quantum-efficient and sensitive sCMOS is required.

[0094] In this embodiment, the imaging component includes an objective lens 5, a dichroic mirror 6, and a first reflecting mirror 81 arranged sequentially along the optical path. The objective lens 5 is 100× and 0.8NA.

[0095] In this embodiment, the imaging components used include a first imaging lens 111 in front of the full light field camera 7 and a second imaging lens 112 in front of the fluorescence imaging two-dimensional array detector 10. The focal length of the first imaging lens 111 and the second imaging lens 112 is 180mm.

[0096] In this embodiment, the full-light field camera 7 includes a full-light field camera imaging lens with a focal length of 100mm. Both the first and second full-light field camera two-dimensional array detectors used in the full-light field camera 7 can be 2048×2048 CMOS array detectors with a pixel size of 6.45μm×6.45μm.

[0097] Step A1: Using the illumination component to emit monochromatic light, the fluorescence imaging two-dimensional array detector of the dual-modal real-time microscopy device is switched to receive the fluorescence signal emitted from the imaging component in order to perform conventional wide-field fluorescence microscopy.

[0098] In this embodiment, the illumination component includes a single-on excitation light source 1, an optical fiber 2' connected to the excitation light source 1, and an optical fiber collimator 3. When the illumination component emits monochromatic light, the excitation light source 1 with the corresponding wavelength is selected and turned on according to the excitation wavelength of the dye molecules of the object under test.

[0099] Step A2: Using the image processing processor 13, control the full-field camera 7 and the fluorescence imaging two-dimensional array detector 10 to synchronously acquire signals, buffer the acquired data, and use the data processing module to calculate and reconstruct the acquired signals to obtain the reconstruction results; then output the reconstruction results to realize dual-modal real-time imaging.

[0100] In this embodiment, the image processing processor 13 is an image processing processor developed based on FPGA. It is connected to the full light field camera and sCMOS camera via USB 3.0 interface to receive the data acquired by the camera. After calculation, the reconstructed result is output to the display device via DP 1.2 interface.

[0101] Thus, the excitation light from the single excitation source 1 is coupled to the optical fiber 2' and collimated by the optical fiber collimator 3, then illuminates the sample 4 in a near-collimated and wide-field illumination manner. The scattered light and fluorescence generated by the sample are collected by the objective lens 5 and split into two paths by the dichroic mirror 6. One path goes to the full-field camera 7 to acquire real image plane and Fourier surface information, and the other path enters the sCMOS camera 10 for fluorescence microscopy imaging. The acquired signals are input into the graphics processing unit 13 for processing via USB 3.0 transmission, realizing real-time phase and fluorescence microscopy dual-modal imaging under monochromatic illumination.

[0102] The third embodiment is a dual-modal super-resolution imaging method using phase and GICS fluorescence under monochromatic illumination.

[0103] like Figure 4 As shown, the method for phase and fluorescence super-resolution dual-modal microscopy imaging under monochromatic illumination includes the following steps:

[0104] Step B0: Construct the dual-modal real-time microscopic imaging device described above;

[0105] As described above, the dual-modal real-time microscopic imaging device includes an illumination assembly (including an excitation source 1, an optical fiber 2', and an optical fiber collimator 3) configured to illuminate the sample 4; an imaging assembly (including an objective lens 5, a dichroic mirror 6, a first imaging lens 111, a second imaging lens 112, a third imaging lens 113, a first reflecting mirror 81, a second reflecting mirror 82, and a switching mirror 9) configured to receive light signals from the sample 4; a full-field camera 7 configured to receive scattered light signals emitted from the imaging assembly; a fluorescence imaging two-dimensional array detector 10 and a GISC camera 12 configured to switchably receive fluorescence signals emitted from the imaging assembly; and an image processing processor 13 electrically connected to the full-field camera 7, the fluorescence imaging two-dimensional array detector 10, and the GISC camera 12. The illumination assembly includes a single activated excitation source 1, a single optical fiber 2', and a beam combiner 3. Since the illumination assembly emits light of a single wavelength, therefore... Figure 4 Only one excitation source 1 is shown.

[0106] In this embodiment, the imaging component includes an objective lens 5, a dichroic mirror 6, and a first reflecting mirror 81 arranged sequentially along the optical path. The objective lens 5 is 100× and 0.8NA.

[0107] In this embodiment, the imaging components used include a first imaging lens 111 in front of the full light field camera 7 and a third imaging lens 113 in front of the GISC camera 12, with a focal length of 180mm for both the first imaging lens 111 and the third imaging lens 113.

[0108] In this embodiment, the full-light field camera 7 includes a Fourier imaging lens with a focal length of 100mm. Both the first and second full-light field camera two-dimensional array detectors used in the full-light field camera 7 can be 2048×2048 CMOS array detectors with a pixel size of 6.45μm×6.45μm.

[0109] Step B1: Using the illumination component to emit monochromatic light, the GISC camera 12 of the dual-modal real-time microscopic imaging device is switched to receive the fluorescence signal emitted by the imaging component; correspondingly, the fluorescence imaging two-dimensional array detector 10 does not receive the fluorescence signal at this time.

[0110] In this embodiment, the illumination component includes a single-on excitation light source 1, an optical fiber 2', and an optical fiber collimator 3. When the illumination component emits monochromatic light, a laser light source 1 with the corresponding wavelength is selected from multiple excitation light sources 1 to be turned on according to the excitation wavelength of the dye molecules of the object under test.

[0111] Step B2: Using the image processing processor 13, control the full light field camera 7 and the GISC camera 12 to synchronously acquire signals, buffer the acquired data, and use the data processing module to calculate and reconstruct the acquired data to obtain the reconstruction result; then output the reconstruction result to realize dual-modal real-time imaging.

[0112] In this embodiment, the image processing processor 13 is an image processing processor developed based on FPGA. It is connected to the full light field camera, GISC super-resolution camera and sCMOS camera via USB 3.0 interface to receive the data acquired by the camera. After calculation, the reconstructed result is output to the display device via DP 1.2 interface.

[0113] Because the wavelength of the scattered signal (phase information reconstructed by the all-light field camera) is the same as the excitation wavelength, in the case of time-series detection, the wavelength of this phase information can be determined according to the different laser time-series excitation sequences, without requiring the all-light field camera to have the ability to distinguish colors.

[0114] Thus, the single-wavelength excitation source 1 is collimated by the fiber collimator 3 through the fiber optic 2' and illuminates the sample 4 in a near-collimated and wide-field illumination manner. The scattered light and fluorescence generated by the sample are collected by the objective lens 5 and split into two paths by the dichroic mirror 6. One path goes to the full-field camera 7 to acquire real image plane and Fourier surface information, and the other path goes to the GISC camera 12. The acquired signal is transmitted via USB 3.0 and input to the control and image processing processor 13, which is based on the FPGA on-chip multi-core MCU, for processing, realizing real-time phase and GISC fluorescence super-resolution dual-modal imaging.

[0115] Fourth embodiment: Full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging method under multicolor sequential illumination

[0116] like Figure 5 As shown, the full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging method under multicolor sequential illumination includes the following steps:

[0117] Step C0: Construct the dual-modal real-time microscopic imaging device described above;

[0118] As described above, the dual-modal real-time microscopic imaging device includes an illumination component for illuminating the sample 4 to be tested, an imaging component (including an excitation source 1, an optical fiber combiner 2, an optical fiber 2', an optical fiber collimator 3, and a timing controller 15) for receiving light signals from the sample 4 to be tested, a full-field camera 7 for receiving scattered light signals emitted from the self-imaging component (including an objective lens 5, a dichroic mirror 6, a switching mirror 9, a first imaging lens 111, a second imaging lens 112, a third imaging lens 113, a first reflecting mirror 81, and a second reflecting mirror 82), a fluorescence imaging two-dimensional array detector 10 and a GISC camera 12 for switching between each other to receive fluorescence signals emitted from the self-imaging component, and an image processing processor 13 electrically connected to the full-field camera 7, the fluorescence imaging two-dimensional array detector 10, and the GISC camera 12.

[0119] In this embodiment, the fluorescence imaging two-dimensional array detector 10 is an sCMOS camera, and the fluorescence imaging two-dimensional array detector 10 is not used in this mode.

[0120] In this embodiment, the imaging assembly includes an objective lens 5, a dichroic mirror 6, and a first reflecting mirror 81 arranged sequentially along the optical path. The objective lens 5 is 100× and 0.8NA.

[0121] In this embodiment, the imaging components used include a first imaging lens 111 in front of the full light field camera 7 and a third imaging lens 113 in front of the GISC camera 12, with a focal length of 180mm for both the first imaging lens 111 and the third imaging lens 113.

[0122] In this embodiment, the full-light field camera 7 includes a full-light field camera imaging lens with a focal length of 100mm. Both the first and second full-light field camera two-dimensional array detectors used in the full-light field camera 7 can be 2048×2048 CMOS array detectors with a pixel size of 6.45μm×6.45μm.

[0123] Step C1: Under the control of the timing controller, the illumination component emits monochromatic light in sequence to emit light of different wavelengths each time, and the GISC camera 12 is switched to receive the fluorescence signal emitted by the self-imaging component 113; correspondingly, the fluorescence imaging two-dimensional array detector 10 does not receive the fluorescence signal at this time.

[0124] The illumination assembly includes multiple excitation light sources 1 and an optical fiber combiner 2 connected to the multiple excitation light sources 1. The optical fiber combiner 2 couples multiple optical fibers to a single optical fiber, and the end of the single optical fiber is connected to an optical fiber collimator 3. Since the illumination assembly emits multi-color light sequentially, it also includes a timing controller 15 connected to the multiple excitation light sources 1 and configured to turn on one excitation light source 1 at a time according to the sequence. The turned-on excitation light sources 1 include laser sources with center wavelengths of 488nm, 532nm, and 639nm. The sequential laser excitation uses an STM32-based timing controller, which controls the switching of the excitation light sources 1 with pulse signal output.

[0125] Step C2: Using the image processing processor 13, control the full-field camera 7 and the GISC camera 12 to synchronously acquire signals in a time sequence, buffer the acquired data, and use the data processing module to calculate and reconstruct the acquired data to obtain a time sequence image as the reconstruction result; then output the reconstruction result, and synthesize a full-color phase and multi-color fluorescence image based on the time sequence image.

[0126] In this embodiment, the full-field camera 7 can simultaneously image the transmittance and phase of the object under test 4. It detects the transmittance and Fourier surface intensity of blue (488nm), green (532nm), and red (639nm) in the excitation sequence, and then reconstructs them separately. The reconstruction result is a time-series image, which includes the time-series acquired real image intensity image and the corresponding phase image reconstructed using the algorithm. The reconstructed light field information (i.e., the reconstruction result) is then superimposed to form a full-color phase map.

[0127] In this embodiment, the image processing processor 13 is an image processing processor developed based on FPGA. It is connected to the full light field camera, GISC super-resolution camera and sCMOS camera via USB 3.0 interface to receive the data acquired by the camera. After calculation, the reconstructed result is output to the display device via DP 1.2 interface.

[0128] Thus, the timing controller 15 sequentially turns on the excitation light source 1 of a single wavelength each time. The laser light excited each time is coupled to a single optical fiber through the fiber combiner 2. The excitation light in the single optical fiber is collimated by the fiber collimator 3 and then illuminates the sample 4 under test in a wide field. The scattered light and fluorescence generated by the sample are collected by the objective lens 5 and split into two paths by the dichroic mirror 6. One path goes to the full-field camera 7 to collect information on the real image plane and Fourier surface, and the other path goes to the GISC camera 12 for acquisition. The acquired signal is transmitted via USB 3.0 and input to the control and image processing processor 13, which is based on the FPGA on-chip multi-core MCU, for processing, so as to realize full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging, namely full-color phase super-resolution imaging and multicolor super-resolution fluorescence imaging.

[0129] Fifth embodiment: Full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging method under simultaneous multicolor illumination

[0130] like Figure 6 As shown, the method for super-resolution dual-modal imaging of full-color phase and GISC multicolor fluorescence under simultaneous multicolor illumination includes the following steps:

[0131] Step D0: Construct the dual-modal real-time microscopic imaging device described above;

[0132] As described above, the dual-modal real-time microscopic imaging device includes an illumination component (including an excitation source 1, an optical fiber combiner 2, and an optical fiber collimator 3) configured to illuminate the sample 4 under test; an imaging component (including an objective lens 5, a dichroic mirror 6, a switching mirror 9, a first imaging lens 111, a second imaging lens 112, a third imaging lens 113, a first reflecting mirror 81, and a second reflecting mirror 82) configured to receive light signals from the sample 4 under test; a full-field camera 7 configured to receive scattered light signals emitted from the imaging component; a fluorescence imaging two-dimensional array detector 10 and a GISC camera 12 configured to switch between each other to receive fluorescence signals emitted from the imaging component; and an image processing processor 13 electrically connected to the full-field camera 7, the fluorescence imaging two-dimensional array detector 10, and the GISC camera 12.

[0133] In this embodiment, the fluorescence imaging two-dimensional array detector 10 is an sCMOS camera, and the fluorescence imaging two-dimensional array detector 10 is not used.

[0134] In this embodiment, the imaging assembly includes an objective lens 5, a dichroic mirror 6, and a first reflecting mirror 81 arranged sequentially along the optical path. The objective lens 5 is 100× and 0.8NA.

[0135] In this embodiment, the imaging components used include a first imaging lens 111 in front of the full light field camera 7 and a third imaging lens 113 in front of the GISC camera 12, with a focal length of 180mm for both the first imaging lens 111 and the third imaging lens 113.

[0136] In this embodiment, the full-light field camera 7 is a full-color full-light field camera, which includes a full-light field camera imaging lens with a focal length of 100mm. The first and second full-light field camera two-dimensional array detectors used in the full-light field camera 7 are both RGB color array detectors, thus constituting a color full-light field camera. Furthermore, both the first and second full-light field camera two-dimensional array detectors can be 2688×2200 CMOS array detectors with a pixel size of 4.54μm×4.54μm.

[0137] Step D1: Using the illumination component to simultaneously emit multicolor light, the GISC camera 12 is switched to receive the fluorescence signal emitted by the third imaging lens 113 of the self-imaging component; correspondingly, the fluorescence imaging two-dimensional array detector 10 does not receive the fluorescence signal at this time.

[0138] The illumination assembly includes multiple excitation light sources 1 and an optical fiber combiner 2 connected to the multiple excitation light sources 1. The optical fiber combiner 2 couples multiple optical fibers to a single optical fiber 2', and the end of the single optical fiber 2' is connected to an optical fiber collimator 3. Since the illumination assembly simultaneously emits multicolor light, it is configured to simultaneously activate multiple excitation light sources 1. The activated excitation light sources 1 include laser sources with center wavelengths of 488nm, 532nm, and 639nm.

[0139] Step D2: Using the image processing processor 13, control the full-light field camera 7 (i.e., the color full-light field camera) and the GISC camera 12 to synchronously acquire signals and buffer the acquired data; using the image processing processor 13, decompose the data output by the color full-light field camera into RGB three-color channels to obtain the decomposed three-color channel signals, and use the data processing module to calculate and reconstruct the decomposed three-color channel signals respectively to obtain the reconstruction results, and re-superimpose the reconstruction results to generate a full-color phase image; using the image processing processor 13, calculate and reconstruct the data output by the GISC camera to obtain a multi-color fluorescence image; thus, a full-color phase image and a multi-color fluorescence image are obtained, realizing dual-modal real-time imaging.

[0140] In this embodiment, the full-field camera 7 can simultaneously image the transmittance and phase of the object under test 3. After the transmittance of blue (488nm), green (532nm), and red (639nm) can be detected simultaneously, the intensity collected by the real image plane and the Fourier plane is decomposed into channels according to the three colors RGB in the subsequent calculation process, and the phase information is reconstructed respectively. Then, the multi-color light fields obtained after decomposition and reconstruction are superimposed to obtain the full-color phase map.

[0141] In this embodiment, the image processing processor 13 is an image processing processor developed based on FPGA. It is connected to the full light field camera 7, GISC camera 12, and sCMOS camera (i.e., fluorescence imaging two-dimensional array detector 10) via USB 3.0 interface to receive camera-acquired data. After calculation, the reconstructed result is output to the display device via DP 1.2 interface.

[0142] Thus, multiple excitation light sources 1 are activated simultaneously, and the lasers excited simultaneously are coupled to a single optical fiber 2' through an optical fiber combiner 2. The excitation light in the single optical fiber 2' is collimated by an optical fiber collimator 3 and then illuminates the sample 4 under test in a wide field. The scattered light and fluorescence generated by the sample 4 are collected by the objective lens 5 and split into two paths by a dichroic mirror 6. One path goes to the full-field camera 7 to collect information on the real image plane and Fourier surface, and the other path goes to the GISC camera 12 for acquisition. The acquired signals are transmitted via USB 3.0 and input to the control and image processing processor 13, which is based on the FPGA on-chip multi-core MCU, for processing, realizing full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging.

[0143] In summary, dual-modal imaging includes, but is not limited to, four common methods: phase and fluorescence microscopy dual-modal imaging under monochromatic illumination, phase and GISC fluorescence super-resolution dual-modal imaging under monochromatic illumination, full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging under time-series multicolor illumination, and full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging under multicolor illumination. Dual-modal imaging refers to the dual-modal real-time microscopy imaging device of this invention having two imaging modes operating simultaneously.

[0144] The phase and fluorescence microscopy dual-modal imaging mode under monochromatic illumination can excite the fluorescence and scattering signals of the sample with a single laser, collect them through the objective lens, and separate them with a dichroic mirror to enter the full-field camera and two-dimensional array detector for dual-modal imaging. In this mode, the phase and GISC fluorescence super-resolution imaging dual-modal imaging under monochromatic illumination can be switched by adjusting the rotating mirror.

[0145] The dual-modal imaging mode of full-color phase and GISC fluorescence super-resolution under time-sequential multicolor illumination can control the excitation sequence of the laser through a timing controller. The fluorescence and scattering signals of the sample are sequentially excited by the time-sequential laser, collected by the objective lens, and separated by a dichroic mirror before entering the full-field camera and the GISC multicolor super-resolution camera for dual-modal imaging. Full-color phase imaging, achieved by the full-field camera, allows imaging of the transmittance and phase of the object at multiple wavelengths.

[0146] Simultaneously, under multicolor illumination, the full-color phase and GISC fluorescence super-resolution dual-modal imaging modes can couple multiple laser beams of different wavelengths into a single laser beam using an optical fiber bundler. This laser beam excites the fluorescence and scattering signals of the sample, which are then collected by the objective lens and separated using a dichroic mirror before entering the full-color full-field camera and the GISC multicolor super-resolution camera for dual-modal imaging. A full-color full-field camera refers to replacing the two-dimensional array detector in a traditional full-field camera with a full-color two-dimensional array detector.

[0147] During the switching process, the GISC camera 12 does not need to move. Since the imaging lens of the full-field camera is a telephoto lens, the wavelength is not sensitive to the full-field camera 7. Therefore, neither camera needs to move its lens position; only the objective lens 5 needs to locate the focal plane.

[0148] Phase microscopy achieved in this way, compared to Fourier layer imaging or coherent diffraction imaging, does not require overlapping spatial Fourier spectral information or oversampling to obtain sufficient information to recover the phase. This reduces redundant information, reduces the amount of data required, and shortens the experimental sampling time to meet the needs of real-time imaging.

[0149] Fluorescence multicolor super-resolution imaging achieved in this way does not require complex electromechanical control systems such as switching filters. It utilizes random encoding and compression of high-dimensional information of the light field, and this method has the capability of single-frame multicolor imaging.

[0150] Although both full-field camera imaging technology and GISC multicolor coded super-resolution microscopy technology are existing technologies, there are difficulties in integrating the two. The difficulties in integration technology lie in:

[0151] 1. High Integration of Illumination Methods. Compared to traditional dual-modal imaging, this device integrates the different illumination methods of dual-modal imaging into an independent unit. This invention highly integrates the illumination methods of phase imaging and fluorescence imaging, using fiber combiners and fiber collimators as key integration components. This allows multi-color illumination for multi-color fluorescence imaging to be integrated into a single fiber perpendicularly illuminating the sample. This device can greatly improve imaging efficiency, enabling simultaneous multi-color illumination in a single exposure, which highly matches the characteristic of GISC cameras that can perform multi-color imaging in a single exposure.

[0152] 2. Single-frame imaging capability. Traditional dual-modal imaging requires multi-angle illumination and multiple frame acquisitions to meet the constraints for solving phase information. This invention, however, utilizes a full-field camera with dual-surface acquisition constraints to meet the phase information requirements even at low sampling rates. Traditional dual-modal super-resolution imaging requires multi-angle moiré modulation using SIM technology, thus necessitating multiple frame acquisitions. This invention, however, employs a GISC super-resolution algorithm combining intensity correlation and compressed sensing to perform super-resolution imaging on single-frame acquired fluorescence signals.

[0153] 3. Multicolor Imaging Capability. Traditional dual-modal imaging has not yet reported on achieving multicolor imaging, and this invention promises to achieve a breakthrough in this area. Multicolor super-resolution relies on the compression coding capability of a random phase modulator for multi-wavelength signals. Multicolor phase imaging technology relies on multicolor illumination and the multicolor acquisition of real image surface intensity from a full-field camera.

[0154] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Various variations can be made to the above embodiments of the present invention. All simple and equivalent changes and modifications made in accordance with the claims and description of this application fall within the protection scope of the claims of this patent. All aspects not described in detail in this invention are conventional technical content.

Claims

1. A dual-modal real-time microscopic imaging device, characterized in that, include: An illumination assembly, configured to illuminate the sample to be tested; An imaging component is configured to receive light signals from the sample under test and emit scattered light signals and fluorescence signals through different optical paths; A full-field light camera, configured to receive scattered light signals emitted from the imaging component; A two-dimensional array detector for fluorescence imaging and a GISC camera, which are configured to switch between receiving fluorescence signals emitted from the imaging component; as well as The image processing processor is electrically connected to the full-field camera, the fluorescence imaging two-dimensional array detector, and the GISC camera. The illumination assembly includes multiple excitation sources, an optical fiber combiner connected to the multiple excitation sources, and an optical fiber collimator connected to the optical fiber combiner via a single optical fiber; phase imaging and fluorescence imaging share the same excitation source. The imaging assembly includes an objective lens, a dichroic mirror, and a switching mirror arranged sequentially along the optical path. The dichroic mirror is configured to reflect the scattered light signal in the optical signal and exit into the first optical path, while allowing the fluorescence signal in the optical signal to pass through and exit into the second optical path. The switching mirror is configured to reflect the fluorescence signal propagating along the second optical path to the third optical path when rotated to the first position, and to allow the fluorescence signal propagating along the second optical path to continue propagating along the second optical path when rotated to the second position. The switching mirror is located on the second optical path. The lighting component is configured to emit monochromatic light by turning on one of a plurality of excitation light sources, or to emit polychromatic light by a plurality of excitation light sources in a sequential or simultaneous manner. The laser output from the excitation source is combined into a single optical fiber through an optical fiber combiner and then collimated to illuminate the sample under test through the optical fiber collimator. Dual-modal imaging includes: phase and fluorescence microscopy dual-modal imaging under monochromatic illumination, phase and GISC fluorescence super-resolution dual-modal imaging under monochromatic illumination, full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging under time-series multicolor illumination, and full-color phase and GISC multicolor fluorescence super-resolution dual-modal imaging under multicolor illumination.

2. The dual-modal real-time microscopic imaging device according to claim 1, characterized in that, The sample to be tested is mounted on a displacement stage, which is a two-dimensional displacement stage.

3. The dual-modal real-time microscopic imaging device according to claim 1, characterized in that, The full-light field camera is configured to split the scattered light signal using a beam splitter. One of the scattered light signals is directly detected by the two-dimensional array detector of the first full-light field camera on the imaging plane, and the other is detected by the two-dimensional array detector of the second full-light field camera on the back focal plane after being transformed by the imaging lens of the full-light field camera. The phase of the scattered light signal of the object is then obtained by reconstructing and recovering it using a Fourier iterative algorithm.

4. The dual-modal real-time microscopic imaging device according to claim 1, characterized in that, The GISC camera includes a random phase modulator placed at a first distance Z1 from the imaging surface of the fluorescence signal, and a two-dimensional array detector of the GISC camera placed at a second distance Z2 after the random phase modulator to record the fluorescence speckle signal. The first distance Z1 and the second distance Z2 satisfy the optimal condition for speckle detection. The GISC camera is configured to reconstruct a multicolor super-resolution fluorescence signal by performing correlation operations with the obtained fluorescence speckle signal and a preset calibration matrix.

5. The dual-modal real-time microscopic imaging device according to claim 1, characterized in that, The imaging component is configured to emit the scattered light signal into a first optical path, and the full-field camera is located in the first optical path; the imaging component is configured to switchably emit the fluorescence signal into a second optical path and a third optical path, the fluorescence imaging two-dimensional array detector is located in the second optical path, and the GISC camera is located in the third optical path.

6. The dual-modal real-time microscopic imaging device according to claim 1, characterized in that, The imaging component is configured to emit the scattered light signal into a first optical path, and the full-field camera is located on the first optical path; the imaging component is configured to emit the fluorescence signal into a second optical path, and the fluorescence imaging two-dimensional array detector and the GISC camera can be switched to be located on the second optical path.

7. The dual-modal real-time microscopic imaging device according to claim 1, characterized in that, The image processing processor has a data processing module; The image processing processor is configured as follows: Step S1: Output the signals acquired simultaneously by the full-field camera and the GISC camera, or by the full-field camera and the fluorescence imaging two-dimensional array detector; Step S2: Cache the collected data and transfer it to the data processing module of the image processing processor; Step S3: Use the pre-built full-field phase retrieval algorithm and GISC super-resolution algorithm in the data processing module to calculate and reconstruct the acquired signal to obtain the reconstruction result; Step S4: Output the reconstruction result.

8. A method for dual-modal imaging using phase and fluorescence microscopy under monochromatic illumination, characterized in that, include: Step A0: Construct the dual-modal real-time microscopic imaging device according to any one of claims 1-7; Step A1: Use the illumination component to emit monochromatic light and switch the fluorescence imaging two-dimensional array detector to receive the fluorescence signal emitted from the imaging component; Step A2: Using the image processing processor, control the full-field camera and the fluorescence imaging two-dimensional array detector to synchronously acquire signals, buffer the acquired data, and use the data processing module to calculate and reconstruct the acquired data to obtain the reconstruction result; then output the reconstruction result.

9. A method for super-resolution dual-modal imaging of phase and GISC fluorescence under monochromatic illumination, characterized in that, include: Step B0: Construct the dual-modal real-time microscopic imaging device according to any one of claims 1-7; Step B1: Use the illumination component to emit monochromatic light and switch the GISC camera to receive the fluorescence signal emitted by the imaging component; Step B2: Using the image processing processor, control the all-light field camera and the GISC camera to synchronously acquire signals, buffer the acquired data, and use the data processing module to calculate and reconstruct the acquired data to obtain the reconstruction result; The reconstruction result is then output.

10. A method for super-resolution dual-modal imaging of full-color phase and GISC multicolor fluorescence under multicolor sequential illumination, characterized in that, include: Step C0: Construct the dual-modal real-time microscopic imaging device according to any one of claims 1-7; Step C1: Using the illumination component to emit monochromatic light of different wavelengths in sequence, switch the GISC camera to receive the fluorescence signal emitted by the imaging component; Step C2: Using an image processing processor, control the full-field camera and the GISC camera to synchronously acquire signals in a time sequence, buffer the acquired data, and use the data processing module to calculate and reconstruct the time sequence data to obtain a time sequence image as the reconstruction result; then output the reconstruction result; synthesize a full-color phase image and a multi-color fluorescence image based on the time sequence image.

11. A method for super-resolution dual-modal imaging of full-color phase and GISC multicolor fluorescence under simultaneous multicolor illumination, characterized in that, include: Step D0: Construct a dual-modal real-time microscopic imaging device according to any one of claims 1-7; wherein, the first full-light field camera two-dimensional array detector and the second full-light field camera two-dimensional array detector in the full-light field camera of the dual-modal real-time microscopic imaging device are both RGB color array detectors, so that the full-light field camera constitutes a color full-light field camera. Step D1: Simultaneously emit multicolor light using the illumination component, and switch the GISC camera to receive the fluorescence signal emitted by the self-imaging component; Step D2: Using an image processing processor, control the color full-field camera and the GISC camera to synchronously acquire signals and buffer the acquired data; using the image processing processor, decompose the data output by the color full-field camera into RGB three-color channels to obtain the decomposed three-color channel signals, and use the data processing module to calculate and reconstruct the decomposed three-color channel signals respectively to obtain the reconstruction results, and re-superimpose the reconstruction results to generate a full-color phase image; using the image processing processor, calculate and reconstruct the data output by the GISC camera to obtain a multicolor fluorescence image; thus, the full-color phase image and the multicolor fluorescence image are obtained.

Images