Multi-modal microscopic imaging system and method

Abstract

The invention relates to a multi-modal microscopic imaging system and method, and the system comprises a multi-modal imaging module which is used for transmitting a multi-modal laser beam; the scanning system comprises a scanning device and an objective lens; the controller is used for controlling the scanning device to move to a specified pose and controlling the scanning device to conduct the multi-mode laser beam to a target area of the sample to be detected through the objective lens for scanning under the specified pose; wherein a multi-mode optical signal and an ultrasonic signal are returned after surface excitation of the to-be-detected sample; the transducer assembly is positioned between the sample to be detected and the objective lens and is used for collecting ultrasonic signals; and the controller is also used for processing the collected multi-mode optical signal and ultrasonic signal returned by the multi-mode imaging module to generate a scanning result image. According to the invention, the problem of time-space consistency caused by time-sharing detection in traditional multi-mode microscopic imaging is solved.

Description

Technical Field

[0001] The present application relates to the field of microscopic imaging, and in particular to a multimodal microscopic imaging system and method. Background Art

[0002] In the field of microscopic imaging, single-modality imaging techniques (such as fluorescence microscopy, photoacoustic imaging, and optical coherence tomography) struggle to simultaneously meet the multidimensional imaging needs of biological samples due to their inherent technical limitations. However, each method offers unique imaging information that complements each other. For example, photoacoustic imaging, based on the photoacoustic effect, provides optical absorption information within tissue structures. It is particularly suitable for detecting the distribution of blood vessels and pigmented tissues (such as melanin), enabling deep tissue exploration. Fluorescence microscopy, using fluorescent signals, provides specific molecular-level information, particularly advantageous for labeling specific molecules or cells, and is suitable for high-resolution localized imaging. Optical coherence tomography provides non-invasive, high-resolution cross-sectional images of tissue structure, particularly suitable for imaging soft and transparent tissues, enabling in-depth visualization of tissue microstructure. To overcome the technical bottlenecks of single-modality microscopy, multimodal microscopy integrates multiple imaging methods to achieve complementary advantages. However, current multimodal fusion, involving multiple advanced imaging technologies, presents a high level of system complexity. In many cases, independent observations of each modality are required, followed by subsequent image alignment and fusion, which increases the complexity of analysis. Furthermore, because each mode is collected in a time-sharing manner, it is difficult to study the real-time changing process.

[0003] Currently, no effective solution has been proposed to the problem that related technologies cannot ensure the spatiotemporal consistency of multimodal signals. Summary of the Invention

[0004] The embodiments of the present application provide a multimodal microscopic imaging system and method to at least solve the problem in the related art that the temporal and spatial consistency of multimodal signals cannot be guaranteed.

[0005] In a first aspect, an embodiment of the present application provides a multimodal microscopic imaging system, the device comprising: a multimodal imaging module, a transducer assembly, a scanning system, and a controller;

[0006] The multimodal imaging module is used to emit a multimodal laser beam;

[0007] The scanning system includes a scanning device and an objective lens;

[0008] The controller is used to control the scanning device to move to a specified position and control the scanning device to transmit the multimodal laser beam through the objective lens to a target area of ​​the sample to be tested for scanning in the specified position; wherein, after being excited by the surface of the sample to be tested, a multimodal optical signal and an ultrasonic signal are returned;

[0009] The transducer assembly is located between the sample to be tested and the objective lens, and is used to collect the ultrasonic signal;

[0010] The controller is further configured to process the collected multimodal optical signal and the ultrasonic signal returned by the multimodal imaging module to generate a scanning result image.

[0011] In some embodiments, the multimodal imaging module includes a fluorescence / normal wide-field microscopy imaging module, a fluorescence confocal scanning microscopy imaging module, a photoacoustic scanning microscopy imaging module, and an optical coherence tomography microscopy imaging module.

[0012] In some embodiments, the transducer assembly is a hollow ring transducer assembly;

[0013] The hollow ring-shaped transducer assembly includes a window and a hollow ring-shaped photoacoustic transducer; the hollow ring-shaped photoacoustic transducer assembly is arranged in a water tank; a first photoacoustic coupling liquid is injected into the water tank; the window is located at the liquid surface of the photoacoustic coupling liquid; the hollow ring-shaped photoacoustic transducer is used to collect the ultrasonic signal generated by the photoacoustic scanning microscopy imaging module to excite the sample to be tested; or,

[0014] The transducer assembly is a prism-coupled transducer assembly;

[0015] The prism-coupled transducer assembly comprises a prism group, an acoustic coupling reflection medium, and a wide-field photoacoustic transducer; the lower surface of the prism group is in a second photoacoustic coupling liquid, and the upper surface of the prism group is in air; the prism group comprises a triangular prism and an rhombic prism; the triangular prism is located between the objective lens and the sample to be measured; the acoustic coupling reflection medium is located on the contact surface between the triangular prism and the rhombic prism, and is used to reflect the ultrasonic signal to the wide-field photoacoustic transducer;

[0016] The wide-field photoacoustic transducer is mounted on the rhombic prism and is used to collect the ultrasonic signal generated by the photoacoustic scanning microscopy imaging module when the sample to be tested is excited.

[0017] In some embodiments, the working modes of the fluorescence / normal wide-field microscopy imaging module include normal mode and fluorescence mode;

[0018] The fluorescence / normal wide-field microscopy module is used to switch the working light source of the fluorescence / normal wide-field microscopy module to a normal light source and switch the beam splitter in the fluorescence / normal wide-field microscopy module to a beam splitting cube in the normal mode;

[0019] The controller is further configured to, when the first laser beam emitted by the ordinary light source is transmitted to the sample to be measured via the beam splitter cube, instruct the scanning device to observe the sample in real time, and adjust the three-dimensional translation stage carrying the sample to be measured until the observed image is clear;

[0020] The fluorescence / normal wide-field microscopy imaging module is further configured to, in the fluorescence mode, switch the working light source to a fluorescence light source and switch the beam splitter to a dichroic mirror;

[0021] The controller is further configured to adjust the three-dimensional translation stage to move the target area into the field of view of the objective lens when the second laser beam emitted by the fluorescent light source is transmitted to the sample to be measured via the dichroic mirror.

[0022] In some embodiments, the third laser beam emitted by the fluorescence confocal scanning microscopy imaging module generates a fluorescence signal after being excited by the sample to be tested, and returns to the fluorescence confocal scanning microscopy imaging module;

[0023] The fourth laser beam emitted by the photoacoustic scanning microscopy module generates the ultrasonic signal after being excited by the sample to be tested and returns to the photoacoustic scanning microscopy module;

[0024] The fifth laser beam emitted by the optical coherence tomography microscopy imaging module is excited by the sample to be tested and returns an optical coherence tomography signal to the optical coherence tomography microscopy imaging module;

[0025] The controller is further used to collect data of the fluorescence signal, the ultrasound signal and the optical coherence tomography signal;

[0026] The controller is further configured to process the fluorescence signal, the ultrasound signal, and the optical coherence tomography signal to obtain multimodal information; and generate the scan result image based on the multimodal information.

[0027] In some embodiments, the multimodal microscopic imaging system further comprises a data acquisition card;

[0028] The fluorescence signal is returned to the fluorescence confocal scanning microscopy imaging module and then sent to the data acquisition card, and then transmitted to the controller;

[0029] The ultrasonic signal is collected by the transducer assembly and sent to the data acquisition card, and then transmitted to the controller.

[0030] In some embodiments, the controller is further configured to collect the fluorescence signal intensity within a preset time period at each scanning collection point, average the fluorescence signal intensities, and obtain fluorescence intensity characteristic information of the scanning collection point;

[0031] The controller is further configured to collect a spectral signal for each of the scanning acquisition points based on the optical coherence tomography signal, and perform Fourier transform on the spectral signal to obtain optical coherence tomography depth information at the scanning acquisition point;

[0032] The controller is further configured to collect the ultrasonic signal intensity within a preset time period at each scanning collection point for the ultrasonic signal, and obtain ultrasonic intensity distribution information at the scanning collection point.

[0033] In some embodiments, the optical coherence tomography microscopy module includes:

[0034] a fiber coupler, configured to split the fifth laser beam into two paths, one path to the sample arm and the other path to the reference arm;

[0035] The sample arm is used to transmit the fifth laser beam to the sample to be measured, and return the sample arm optical signal generated by the excitation of the sample to be measured to the linear array camera spectrometer via the optical fiber coupler;

[0036] The reference arm is used to transmit the fifth laser beam to the reference arm reflector, and reflect the reference arm optical signal to the linear array camera spectrometer via the fiber coupler;

[0037] The linear array camera spectrometer is used to receive the sample arm light signal and the reference arm light signal, combine the sample arm light signal and the reference arm light signal into the optical coherence tomography signal, and transmit the signal to the controller for optical coherence tomography imaging data processing.

[0038] In some embodiments, the fluorescence confocal scanning microscopy imaging module includes a first-band fluorescence confocal scanning microscopy imaging module and a second-band fluorescence confocal scanning microscopy imaging module; the multimodal microscopy imaging system further includes a first spectroscopic element, a second spectroscopic element, a third spectroscopic element, and a reflective element;

[0039] The first beam splitter element is configured to reflect the first-band laser beam emitted by the first-band fluorescence confocal scanning microscopy imaging module and transmit the fourth laser beam emitted by the photoacoustic scanning microscopy imaging module to the reflective element;

[0040] The second beam splitter element is configured to reflect the second-band laser beam emitted by the second-band fluorescence confocal scanning microscopy imaging module and transmit the fifth laser beam emitted by the optical coherence tomography microscopy imaging module to the third beam splitter element;

[0041] The reflecting element is used to reflect the first wavelength band laser beam and the fourth laser beam to the third beam splitting element;

[0042] The third beam splitter element is used to reflect the first-band laser beam and the fourth laser beam reflected by the reflective element to the scanning device, and transmit the second-band laser beam and the fifth laser beam combined by the second beam splitter element to the scanning device.

[0043] In a second aspect, an embodiment of the present application provides a multimodal microscopic imaging method, the method comprising:

[0044] controlling the multimodal imaging module to emit a multimodal laser beam;

[0045] Controlling a scanning device in a scanning system to move to a specified position, and controlling the scanning device to transmit the multimodal laser beam to a target area of ​​a sample to be tested through an objective lens in the scanning system in the specified position for scanning; wherein, after being excited by the surface of the sample to be tested, a multimodal light signal and an ultrasonic signal are returned, and the ultrasonic signal is collected by a transducer assembly, and the transducer assembly is located between the sample to be tested and the objective lens;

[0046] The collected multimodal optical signal and the ultrasonic signal returned by the multimodal imaging module are processed to generate a scanning result image.

[0047] Compared with the related art, the multimodal microscopic imaging system and method provided in the embodiment of the present application, wherein the system includes: a multimodal imaging module for emitting a multimodal laser beam; a scanning system including a scanning device and an objective lens; a controller for controlling the scanning device to move to a specified position, and controlling the scanning device to transmit the multimodal laser beam to the target area of ​​the sample to be tested through the objective lens for scanning in the specified position; wherein, after being excited by the surface of the sample to be tested, a multimodal light signal and an ultrasonic signal are returned; a transducer assembly is located between the sample to be tested and the objective lens, and is used to collect the ultrasonic signal; the controller is also used to process the collected multimodal light signal and the ultrasonic signal returned by the multimodal imaging module to generate a scan result image. Based on this, the same target area is synchronously observed by the multimodal imaging module, which solves the problem of spatiotemporal consistency caused by time-sharing detection in traditional multimodal microscopic imaging.

[0048] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0050] Figure 1 is a hardware structure block diagram of a terminal of a multimodal microscopic imaging system according to an embodiment of the present application;

[0051] Figure 2 is a structural block diagram of a multimodal microscopic imaging system according to an embodiment of the present application;

[0052] Figure 3 is a schematic diagram of a multimodal microscopic imaging system according to an embodiment of the present application;

[0053] Figure 4 1 is a schematic structural diagram of a hollow ring transducer assembly according to an embodiment of the present application;

[0054] Figure 5 1 is a schematic structural diagram of a prism-coupled transducer assembly according to an embodiment of the present application;

[0055] Figure 6 Schematic diagram of the fluorescence / normal wide-field microscopy module, the fluorescence scanning confocal microscopy module, the photoacoustic scanning microscopy module, and the optical coherence tomography microscopy module in this embodiment;

[0056] Figure 7 Schematic diagram of one implementation of the sample stage in this embodiment;

[0057] Figure 8 4 is a flow chart of a multimodal microscopic imaging method according to an embodiment of the present application.

[0058] Description of reference numerals:

[0059] Figure 1 : 112, processor; 114, memory; 116, transmission equipment; 118, input and output devices.

[0060] Figure 2 :201, multimodal imaging module; 202, scanning system; 203, controller; 204, transducer assembly.

[0061] Figure 3:1. 3D translation stage; 2. Sample; 3. Water tank assembly; 4. Objective lens; 5. Reflector; 6. Beam splitter; 7. Fluorescence / normal widefield microscopy module; 8. Field lens; 9. Scanning lens; 10. Computer; 11. Scanning device; 12. Dichroic mirror; 13. Reflector; 14. Fluorescence scanning confocal microscopy module in the visible light band; 15. Dichroic mirror; 16. Photoacoustic scanning microscopy module; 17. Optical phase Dry-layer tomography microscopy module; 18. Dichroic mirror; 19. Near-infrared fluorescence scanning confocal microscopy module; A. Hollow ring transducer assembly; B. Prism-coupled transducer assembly; 301. Distilled water; 302. Thin film; A01. Wide-band window; A02. Hollow ring photoacoustic transducer; B01. Triangular prism; B02. Silicone oil; B03. Parallelogram prism; B04. Wide-field photoacoustic transducer.

[0062] Figure 4 : 4, objective lens; A01, broadband window; A02, hollow photoacoustic transducer; A03, window fixing structure; A04, sealing ring; A05, retaining ring; A06, outer sleeve; A07, hollow photoacoustic transducer fixing part; A08, sealing ring; A09 inner sleeve; A10, exhaust hole; A11, five-axis adjustment frame; A12, one-dimensional translation stage; A13, fixing structure.

[0063] Figure 5 : 4, objective lens; B01, triangular prism; B02, silicone oil; B03, rhombic prism; B04, wide-field photoacoustic transducer; B05, prism pair fixing structure; B06, gantry structure; B07, gantry base structure; B08, adjusting screw; B09, spring.

[0064] Figure 6: 4. Objective lens; 7. Fluorescence / normal widefield microscopy module; 10. Computer; 14. Visible light band fluorescence scanning confocal microscopy module; 16. Photoacoustic scanning microscopy module; 17. Optical coherence tomography microscopy module; 20. Control card; 21. Data acquisition card; 701. Fluorescence light source (fluorescence mode) or normal illumination light source (normal mode); 702. Condenser lens; 703. Filter; 704. Condenser lens; 705. Dichroic mirror (fluorescence mode) or beam splitter cube (normal mode); 706. Imaging lens; 707. Filter; 708. Industrial or scientific camera; 709. Darkfield illumination light source; 1401. Fluorescence excitation laser; 1402. Filter; 1403. Dichroic mirror; 1404. Filter; 1405. Focusing lens; 1406. Pinhole; 1407. Photodetector; 1601. Pulsed laser; 1602. Adjustable neutral density filter; 1701. Broadband laser; 1702. Fiber jumper; 1703. Fiber coupler; 1704. Fiber jumper; 1705. Fiber collimator; 1706. Fiber jumper; 1707. Fiber collimator; 1708. Reference arm compensation lens; 1709. Reflector; 1710. Fiber jumper; 1711. Linear array camera spectrometer.

[0065] Figure 7 : 3. Water tank assembly; 101. Z-axis translation stage; 102. Y-axis translation stage; 103. X-axis translation stage; 104. Sample placement platform; 105. Column; 301. Distilled water; 302. Film; 303. Water tank inner layer structure; 304. Water tank outer layer structure; 305. Water tank base structure. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means and should not be understood as the contents disclosed in the present application being insufficient.

[0067] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0068] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "an," "the," and similar expressions used herein do not denote limitations on quantity and may refer to either the singular or the plural. The terms "comprise," "include," "have," and any variations thereof, used herein, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules (units) is not limited to the listed steps or units but may also include steps or units not listed, or may include other steps or units inherent to the process, method, product, or apparatus. The terms "connected," "connected," "coupled," and similar expressions used herein are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used herein, "plurality" means greater than or equal to two. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" may mean: A exists alone; A and B exist simultaneously; or B exists alone. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0069] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 FIG is a hardware structure block diagram of a terminal of a multimodal microscopic imaging system according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown) a processor 112 (the processor 112 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 114 for storing data. Optionally, the terminal may also include a transmission device 116 and an input / output device 118 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1Different configurations shown.

[0070] The memory 114 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the multimodal microscopy imaging system in the embodiments of the present application. The processor 112 executes the computer programs stored in the memory 114 to perform various functional applications and data processing, thereby implementing the above-mentioned method. The memory 114 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 114 may further include memory remotely located relative to the processor 112, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0071] Transmission device 116 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 116 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 116 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0072] This embodiment provides a multimodal microscopic imaging system, Figure 2 is a structural block diagram of a multimodal microscopic imaging system according to an embodiment of the present application, such as Figure 2 As shown, the apparatus includes: a multimodal imaging module 201 , a scanning system 202 , a controller 203 and a transducer assembly 204 .

[0073] The multimodal imaging module 201 integrates multiple microscopic imaging modalities and can collaboratively emit multimodal laser beams: the scanning system 202 can be composed of a two-dimensional galvanometer scanning device and a long working distance objective lens, wherein the scanning device is controlled by the controller 203 to achieve two-dimensional deflection of the laser beam, and cooperates with the scanning lens and the field lens to form a telecentric scanning system, so that the laser focus forms a distortion-free raster scanning trajectory on the sample surface. The objective lens adopts a high numerical aperture design, taking into account both imaging resolution and working distance, ensuring that the laser beam is focused on the target area of ​​the sample with the minimum divergence angle, while efficiently collecting the returned light signal. The system supports two modes: galvanometer scanning and translation stage scanning: galvanometer scanning is suitable for small-scale rapid imaging, while translation stage scanning is used for large-area distortion-free scenes. The two are switched by the controller 203 to adapt to different experimental needs.

[0074] The controller 203, with a computer at its core, implements full-process control through a control card and a data acquisition card. First, it controls the movement of the galvanometer mirror to a specified position according to a preset scanning path (e.g., raster-style progressive scanning), synchronously triggering the laser and detector of the multimodal imaging module 201. The specified position refers to the target state achieved along the preset scanning path, achieved by the controller 203 through precise control of the spatial position and deflection angle of the scanning device. The scanning device, in a series of specified positions along the preset scanning path, enables the laser beam to focus along a preset trajectory to a specified location on the sample surface. Second, after the laser beam is transmitted through the objective lens to the sample surface, stimulating multimodal optical signals (fluorescence signals, optical coherence tomography signals) and ultrasonic signals (collected by the transducer), the data acquisition card and the linear array camera are coordinated in real time to synchronously capture the signals.

[0075] Transducer assembly 204, located between the sample to be measured and the objective lens, offers two configurations for ultrasonic signal acquisition: a hollow ring transducer assembly comprising a wide-band window and a hollow piezoelectric ceramic transducer. The laser beam is incident on the sample through the window and the hollow region of the transducer, while the ultrasonic signal is collected by the piezoelectric ceramic on the outside of the transducer. An exhaust pipe and a multi-axis adjustment mechanism ensure the optical flatness of the water coupling layer and the optimized transducer posture. A prism-coupled transducer assembly, using a triangular prism and a parallelogram prism in conjunction with a silicone oil layer, achieves total internal reflection transmission of the laser beam. The ultrasonic signal, after reflection from the silicone oil, is received by the wide-field transducer. Both designs ensure coaxiality between ultrasonic signal acquisition and optical signal transmission, avoiding temporal and spatial errors caused by signal path delays.

[0076] Controller 203 performs differentiated processing on the returned multimodal signals: time-domain averaging of the fluorescence signal for noise reduction, recording of the ultrasound signal's time-domain waveform, and Fourier transform inversion of depth information from the optical coherence tomography signal. Finally, based on the scanning system's coordinate mapping, controller 203 fuses these three signals to generate a multimodal scan image containing tissue structure, function, and molecular information, resolving the spatiotemporal registration challenges of traditional time-sharing imaging.

[0077] Figure 3Schematic diagram of a multimodal microscopic imaging system according to an embodiment of the present application. The multimodal imaging module 201 may include a fluorescence / normal widefield microscopic imaging module 7, a visible light fluorescence scanning confocal microscopic imaging module 14, a photoacoustic scanning microscopic imaging module 16, an optical coherence tomography microscopic imaging module 17, and a near-infrared fluorescence scanning confocal microscopic imaging module 19. The fluorescence / normal widefield microscopic imaging module 7 may be represented as FL-WF, the visible light fluorescence scanning confocal microscopic imaging module 14 and the near-infrared fluorescence scanning confocal microscopic imaging module 19 may both be represented as FL-LSCM, the photoacoustic scanning microscopic imaging module 16 may be represented as PAM, and the optical coherence tomography microscopic imaging module 17 may be represented as OCT. Scanning system 202 includes a scanning device 11, a telecentric lens assembly, and an objective lens 4. Scanning device 11 is used to scan the laser beam, preferably using a two-dimensional galvanometer. The telecentric lens assembly includes a scanning lens 9 and a field lens 8. The scanning lens 9 and field lens 8 form a 4f system positioned between scanning device 11 and objective lens 4. The scanning device 11 and objective lens 4 are positioned at the front and rear focal planes of the 4f system, respectively. Controller 203 includes computer 10. Transducer assembly 204 includes either a hollow ring transducer assembly A or a prism-coupled transducer assembly B.

[0078] Through the above-mentioned embodiment, a multimodal microscopic imaging system is provided, which includes at least a multimodal imaging module 201, a scanning system 202, a controller 203 and a transducer assembly 204. Through this device, multimodal synchronous scanning imaging of the same target area is achieved, which solves the position alignment and real-time problems caused by traditional time-sharing detection of each modality. In addition, the design of the transducer assembly realizes coaxial transmission of ultrasonic signals and optical signals, ensures the temporal and spatial consistency of multimodal optical signals and ultrasonic signals, and ultimately efficiently generates a scanning result image that integrates multimodal information.

[0079] In some embodiments, the multimodal imaging module includes a fluorescence / normal wide-field microscopy imaging module, a fluorescence confocal scanning microscopy imaging module, a photoacoustic scanning microscopy imaging module, and an optical coherence tomography microscopy imaging module.

[0080] The fluorescence / normal wide-field microscopy module camera generally uses a CMOS / CCD scientific research camera or industrial camera. Depending on the observation band, a black-and-white camera or a color camera can be flexibly selected. The operating mode can be divided into normal mode and fluorescence mode for real-time observation of samples. The fluorescence confocal scanning microscopy module uses a laser with a specific wavelength band, which is reflected by a dichroic mirror and then filtered through a pinhole to achieve high-resolution three-dimensional fluorescence imaging. The photoacoustic scanning microscopy module uses a pulsed laser to excite the sample to produce a photoacoustic effect. Its laser beam is combined with other modes to penetrate the tissue. The optical coherence tomography microscopy module uses a broadband fiber laser and constructs an interference optical path through a 2×2 fiber coupler. After the four major modes of laser beam are combined by a beam splitter, they are transmitted through the same optical path to ensure the spatial coaxiality and synchronous excitation capability of the multimodal signal.

[0081] In the above embodiment, rapid real-time observation is achieved through the fluorescence / conventional wide-field microscopy module, the fluorescence confocal scanning microscopy module provides high-resolution three-dimensional fluorescence imaging, the photoacoustic scanning microscopy module obtains ultrasonic signals of the optical absorption characteristics of the tissue, and the optical coherence tomography microscopy module achieves high-resolution visualization of the tissue tomographic structure. The four modules work together to achieve synchronous imaging of the four modalities of fluorescence wide-field, fluorescence confocal, photoacoustic and optical coherence tomography of the same target area, solving the position alignment and real-time problems of traditional time-sharing detection, and the shared optical scanning system improves scanning efficiency and flexibility.

[0082] In some embodiments, the transducer assembly is a hollow ring transducer assembly;

[0083] The hollow ring-shaped transducer assembly includes a window and a hollow ring-shaped photoacoustic transducer; the hollow ring-shaped photoacoustic transducer assembly is arranged in a water tank; a first photoacoustic coupling liquid is injected into the water tank; the window is located at the liquid surface of the photoacoustic coupling liquid; the hollow ring-shaped photoacoustic transducer is used to collect the ultrasonic signal generated by the photoacoustic scanning microscopy imaging module to excite the sample to be tested; or,

[0084] The transducer assembly is a prism-coupled transducer assembly;

[0085] The prism-coupled transducer assembly comprises a prism group, an acoustic coupling reflection medium, and a wide-field photoacoustic transducer; the lower surface of the prism group is in a second photoacoustic coupling liquid, and the upper surface of the prism group is in air; the prism group comprises a triangular prism and an rhombic prism; the triangular prism is located between the objective lens and the sample to be measured; the acoustic coupling reflection medium is located on the contact surface between the triangular prism and the rhombic prism, and is used to reflect the ultrasonic signal to the wide-field photoacoustic transducer;

[0086] The wide-field photoacoustic transducer is mounted on the rhombic prism and is used to collect the ultrasonic signal generated by the photoacoustic scanning microscopy imaging module when the sample to be tested is excited.

[0087] When the transducer assembly is a hollow ring transducer assembly, it includes a window and a hollow ring photoacoustic transducer. The hollow ring photoacoustic transducer assembly is placed in a water tank, into which a first photoacoustic coupling liquid is injected. The window is located at the surface of the photoacoustic coupling liquid. By contacting the water, the contact surface is optically level, avoiding optical chromatic aberration caused by surface tension. The hollow ring photoacoustic transducer is used to collect ultrasonic signals generated by the sample under test when the photoacoustic scanning microscopy module excites the sample. Its design, with a light-transmitting center and piezoelectric ceramic outer surface, can efficiently collect ultrasonic signals while the laser beam passes through the window and photoacoustic coupling liquid and enters the sample. Figure 3 In the hollow ring transducer assembly A, the broadband window A01 is placed on the surface of distilled water 301, with the water in contact with the window. The hollow ring photoacoustic transducer A02 is placed below the broadband window A01. The laser beam passes through the broadband window A01, the distilled water 301 (the hollow portion of the hollow ring photoacoustic transducer A02), and the film 302, before being incident on sample 2 (the mouse) for optical scanning. The multimodal optical signal returns along the original path, while the ultrasonic signal is detected by the hollow ring photoacoustic transducer A02.

[0088] Figure 4 This is a structural schematic diagram of a hollow ring-shaped transducer assembly according to an embodiment of the present application, which includes an objective lens 4, a broadband window A01, a hollow photoacoustic transducer A02, a window fixing structure A03, a sealing ring A04, a retaining ring A05, an outer sleeve A06, a hollow photoacoustic transducer fixing part A07, a sealing ring A08, an inner sleeve A09, an exhaust hole A10, a five-axis adjustment frame A11, a one-dimensional translation stage A12 and a fixing structure A13.

[0089] Specifically, the broadband window A01 and window fixture A03 were glued together and then placed from above into the inner sleeve A09. A sealing ring A04 was placed between the window fixture A03 and the retaining ring A05, and then the retaining ring A05 was screwed in place. The hollow photoacoustic transducer A02 was placed from below into the inner sleeve A09 and secured with side screws. A sealing ring A08 was placed between the hollow photoacoustic transducer A02 and the hollow photoacoustic transducer fixture A07, and the hollow photoacoustic transducer fixture A07 was screwed to the inner sleeve A09. The inner sleeve A09 was placed into the outer sleeve A06 and secured with side screws. During the experiment, distilled water filled the gap between the broadband window A01 and the hollow photoacoustic transducer A02. Excess air inside was removed through the vent A10. The outer sleeve A06 was placed from below into the five-axis adjustment frame A11 and secured with screws. Connect and secure the one-dimensional translation stage A12 to the fixed structure A13, and then secure the five-axis adjustment frame A11 to the one-dimensional translation stage A12. The broadband window A01, hollow photoacoustic transducer A02, and objective lens 4 remain coaxial and concentric. Fixed structure A13 is used to connect to the external platform.

[0090] When the transducer assembly is a prism-coupled transducer assembly, it includes a prism assembly consisting of a triangular prism and an rhombus prism, an acoustic coupling reflective medium (such as silicone oil) located at the contact surface of the two prisms, and a wide-field photoacoustic transducer mounted on the rhombus prism. The lower surface of the prism assembly is immersed in a second photoacoustic coupling liquid, which is injected into a water tank. The upper surface of the prism assembly is exposed to air. The triangular prism is arranged between the objective lens and the sample to be measured to form an optical path. The laser beam sequentially passes through the triangular prism, the silicone oil layer, the rhombus prism, and the photoacoustic coupling liquid to be incident on the sample. The acoustic coupling reflective medium utilizes its optical transparency and acoustic reflection properties to reflect the ultrasonic signal generated by the sample excitation to the wide-field photoacoustic transducer. The transducer achieves efficient collection of ultrasonic signals by adhering to the surface of the rhombus prism. At the same time, the total reflection transmission mechanism of the prism assembly ensures coaxial transmission of the laser beam and the ultrasonic signal. Figure 3 In the prism-coupled transducer assembly B, the laser beam passes through triangular prism B01, silicone oil B02, parallelogram prism B03, distilled water 301, and film 302, before being incident on sample 2 for optical scanning. The multimodal optical signal returns along the original path, while the ultrasonic signal is reflected by silicone oil B02, then by parallelogram prism B03, and is received by wide-field photoacoustic transducer B04.

[0091] Figure 5This is a structural schematic diagram of a prism-pair coupled transducer assembly according to an embodiment of the present application. The prism-pair coupled transducer assembly includes an objective lens 4, a triangular prism B01, silicone oil B02, an rhombus prism B03, a wide-field photoacoustic transducer B04, a prism pair fixing structure B05, a gantry structure B06, a gantry base structure B07, an adjusting screw B08, and a spring B09.

[0092] Specifically, the triangular prism B01 and the rhombus prism B03 are combined with silicone oil B02 to form a prism pair, which is placed in the prism pair fixing structure B05 and fixed by the front fastening screws. The wide-field photoacoustic transducer B04 is placed in the slot of the prism pair fixing structure B05 from above, and the transducer contacts the rhombus prism B03, and is then fixed by the lateral fastening screws. The prism pair fixing structure B05 is fixed to the center position of the gantry structure B06 by screws. After the gantry structure B06 is connected to the gantry base structure B07, it is fixed to the external platform by the B08 adjustment screw. The B09 spring is always in a compressed state. Adjusting the B08 adjustment screw can achieve pitch adjustment and Z-direction adjustment of the prism pair.

[0093] In the above embodiments, coaxial transmission and synchronous acquisition of ultrasonic signals and optical signals are achieved by adopting two designs: a hollow ring transducer assembly and a prism-coupled transducer assembly. The hollow ring transducer assembly contacts the photoacoustic coupling liquid through a window to ensure the optical level of the liquid surface, thereby avoiding optical chromatic aberration caused by surface tension. The structural design of the hollow ring photoacoustic transducer, in which light is transmitted in the middle and acoustic signals are collected on the outer side, allows efficient collection of ultrasonic signals while the laser beam passes through the window to enter the sample. The prism-coupled transducer assembly uses a triangular prism and an rhombus prism in combination with an acoustic coupling reflective medium (such as silicone oil), utilizes prism total reflection to transmit the laser beam, and reflects the ultrasonic signal to the wide-field photoacoustic transducer. Both designs solve the problems of optical coupling and acoustic signal transmission during ultrasonic signal acquisition, thereby ensuring the spatiotemporal consistency of ultrasonic signals and multimodal optical signals in multimodal imaging.

[0094] In some embodiments, the working modes of the fluorescence / normal wide-field microscopy imaging module include normal mode and fluorescence mode;

[0095] The fluorescence / normal wide-field microscopy module is used to switch the working light source of the fluorescence / normal wide-field microscopy module to a normal light source and switch the beam splitter in the fluorescence / normal wide-field microscopy module to a beam splitting cube in the normal mode;

[0096] The controller is further configured to, when the first laser beam emitted by the ordinary light source is transmitted to the sample to be measured via the beam splitter cube, instruct the scanning device to observe the sample in real time, and adjust the three-dimensional translation stage carrying the sample to be measured until the observed image is clear;

[0097] The fluorescence / normal wide-field microscopy imaging module is further configured to, in the fluorescence mode, switch the working light source to a fluorescence light source and switch the beam splitter to a dichroic mirror;

[0098] The controller is further configured to adjust the three-dimensional translation stage to move the target area into the field of view of the objective lens when the second laser beam emitted by the fluorescent light source is transmitted to the sample to be measured via the dichroic mirror.

[0099] Specifically, the fluorescence / normal widefield microscopy module operates in two modes: normal mode and fluorescence mode. In normal mode, the module switches the working light source to a normal light source such as an LED or laser, and switches the beam splitter to a beamsplitter cube. After the first laser beam emitted by the normal light source is transmitted through the beamsplitter cube to the sample to be measured, the controller instructs the scanning device to perform real-time brightfield or darkfield observation of the sample. Simultaneously, the module adjusts the three-dimensional translation stage (including a combination of one or more of the following devices: a piezoelectric translation stage, an air bearing translation stage, a mechanical motorized translation stage, or a manual translation stage) carrying the sample, fine-tuning the X, Y, and Z axis positions to achieve a clear image. In fluorescence mode, the module switches the working light source to a fluorescent light source such as an LED or laser of a specific wavelength, and switches the beamsplitter to a dichroic mirror. After the second laser beam emitted by the fluorescent light source is transmitted through the dichroic mirror to the sample to be measured, the controller adjusts the three-dimensional translation stage to precisely move the target area to the center of the objective lens' field of view, and cooperates with the filter set to filter out stray light to achieve the excitation and preliminary positioning of the sample's fluorescence signal. Figure 3 , which can be understood as placing sample 2 on the 3D translation stage 1 and placing the water tank assembly 3 above the sample 2, so that the film 302 is in close contact with the sample 2. The 3D translation stage 1 is moved to position the sample 2 below the objective lens 4. The sample is observed in real time using the fluorescence / normal widefield microscopy imaging module 7 in normal mode. The 3D translation stage 1 is repeatedly adjusted until the observed image is clear. The fluorescence / normal widefield microscopy imaging module 7 is switched to fluorescence mode, and the 3D translation stage 1 is adjusted to move the area of ​​sample 2 to be observed into the field of view of the objective lens 4.

[0100] In the above embodiment, multi-scene observation and precise positioning of the sample are achieved by switching between the normal mode and the fluorescence mode of the fluorescence / normal wide-field microscopy imaging module: in the normal mode, switching to the normal light source and the beam splitter cube allows real-time observation of the sample through bright field or dark field illumination, and the image is made clear by adjusting the three-dimensional translation stage, which is suitable for preliminary observation and overall positioning of the sample; in the fluorescence mode, switching to the fluorescent light source and the dichroic mirror, using a specific wavelength laser to excite the sample fluorescence, combined with the three-dimensional translation stage to move the target area to the field of view of the objective lens, providing precise target positioning for subsequent multi-modal high-precision scanning. The flexible switching between the two modes improves the system's adaptability to different observation needs and operational efficiency.

[0101] In some embodiments, the third laser beam emitted by the fluorescence confocal scanning microscopy imaging module generates a fluorescence signal after being excited by the sample to be tested, and returns to the fluorescence confocal scanning microscopy imaging module;

[0102] The fourth laser beam emitted by the photoacoustic scanning microscopy module generates the ultrasonic signal after being excited by the sample to be tested and returns to the photoacoustic scanning microscopy module;

[0103] The fifth laser beam emitted by the optical coherence tomography microscopy imaging module is excited by the sample to be tested and returns an optical coherence tomography signal to the optical coherence tomography microscopy imaging module;

[0104] The controller is further used to collect data of the fluorescence signal, the ultrasound signal and the optical coherence tomography signal;

[0105] The controller is further configured to process the fluorescence signal, the ultrasound signal, and the optical coherence tomography signal to obtain multimodal information; and generate the scan result image based on the multimodal information.

[0106] Specifically, the fluorescence confocal scanning microscopy module emits a third laser beam through a laser of a specific wavelength, which is reflected by a dichroic mirror and focused by a scanning system onto the sample to be tested. The fluorescence signal generated by the excitation returns along the original path, passes through the dichroic mirror, is filtered by a filter set and a pinhole spatial filter, and is converted into an electrical signal by a photodetector and transmitted to a controller.

[0107] The fourth pulse laser beam emitted by the photoacoustic scanning microscopy module excites the sample to produce a photoacoustic effect. The generated ultrasonic signal is collected by the transducer assembly (hollow ring or prism pair coupling structure), converted into an electrical signal through the piezoelectric effect, and then transmitted to the controller;

[0108] The fifth broad-spectrum laser beam emitted by the optical coherence tomography microscopy module is divided into a sample arm and a reference arm by a fiber coupler. The reflected light from the sample and the reference arm form interference fringes in the spectrometer, which are collected by a linear array camera as optical coherence tomography signals.

[0109] The controller synchronously triggers the acquisition of three types of signals through the control card: the fluorescence signal is de-noised by time-domain averaging through the data acquisition card, the ultrasound signal directly records the time-domain waveform, and the optical coherence tomography signal (OCT) is Fourier transformed to invert the depth information. Finally, the multimodal data is fused based on the scanning coordinate mapping to generate a scanning result image that includes molecular markers (fluorescence), optical absorption characteristics (ultrasound) and tissue tomographic structure (OCT).

[0110] Figure 6Schematic diagram of the fluorescence / normal widefield microscopy module, the fluorescence scanning confocal microscopy module, the photoacoustic scanning microscopy module and the optical coherence tomography microscopy module in this embodiment. Figure 3 and Figure 6 , it can be understood that the computer 10 controls the control card 20 to output the scanning waveform to the scanning device 11 ( Figure 3 ), and simultaneously outputs a synchronous trigger signal to the data acquisition card 21 and the optical coherence tomography microscopy imaging module 17 for triggering acquisition. In the visible light band fluorescence scanning confocal microscopy imaging module 14, the fluorescence excitation laser 1401 emits a laser beam, which is filtered by the filter 1402 and reflected by the dichroic mirror 1403 and then enters the subsequent optical path (such as Figure 3 As shown). After the sample is excited, the fluorescence returns along the original path, passes through the dichroic mirror 1403, and then the filter 1404 filters out the light of the unnecessary band other than the fluorescence. Subsequently, the focusing lens 1405 focuses the fluorescence, and the focus is spatially filtered through the pinhole 1406 to achieve a confocal effect. Finally, the photodetector 1407 collects the light signal, and the data acquisition card 21 is responsible for collecting and transmitting it to the computer 10. At the same time, after receiving the trigger signal from the control card 20, the pulse laser 1601 outputs a pulsed laser, and the light intensity is adjusted by the adjustable neutral density filter 1602 before entering the subsequent optical path (as shown). Figure 3 As shown). In addition, in the optical coherence tomography microscopy module 17, the laser beam output by the broadband laser 1701 is transmitted to the fiber coupler 1703 by the fiber jumper 1702, and then split into two light paths, which are output to the sample arm by the fiber jumper 1704 and to the reference arm by the fiber jumper 1706. The laser output by the fiber jumper 1704 is coupled by the fiber collimator 1705 and then output to the subsequent optical path (such as Figure 3 (as shown). Light output from fiber jumper 1706 is coupled through fiber collimator 1707 and output to the free space of the reference arm. Subsequently, the laser light passes through reference arm compensation lens 1708 and is focused onto reflector 1709. It then returns along the same path, passes through fiber coupler 1703, enters fiber jumper 1710, and is ultimately output to linear array camera spectrometer 1711. Similarly, light reflected from the sample also returns along the same path and ultimately enters linear array camera spectrometer 1711. The laser light returning from the sample arm and reference arm forms interference fringes in linear array camera spectrometer 1711, which are recorded and captured by the camera and then transmitted to computer 10. Computer 10 generates a scan result image based on the captured information.

[0111] In the above embodiment, the third, fourth, and fifth laser beams are respectively emitted to the sample to be tested through the fluorescence confocal scanning microscopy module, the photoacoustic scanning microscopy module, and the optical coherence tomography microscopy module. After the generated fluorescence signals, ultrasonic signals, and optical coherence tomography signals are returned to each module, the controller synchronously collects and processes the three types of signals, realizing the real-time fusion of multimodal information such as molecular labeling (fluorescence), optical absorption characteristics (photoacoustics), and tissue tomographic structure (OCT) of the same target area, solving the spatiotemporal registration problem of traditional time-sharing detection, and providing an efficient and accurate technical solution for the joint analysis of the structure, function, and molecular levels of biological samples.

[0112] In some embodiments, the multimodal microscopic imaging system further comprises a data acquisition card;

[0113] The fluorescence signal is returned to the fluorescence confocal scanning microscopy imaging module and then sent to the data acquisition card, and then transmitted to the controller;

[0114] The ultrasonic signal is collected by the transducer assembly and sent to the data acquisition card, and then transmitted to the controller.

[0115] Specifically, the multimodal microscopy system also includes a high-speed data acquisition card for capturing and transmitting fluorescence signals and ultrasonic signals in real time: when the fluorescence signal generated by the fluorescence confocal scanning microscopy module excites the sample and returns, it is converted into an electrical signal by the photodetector and sent to the data acquisition card, and then transmitted to the controller after analog-to-digital conversion for time-domain averaging noise reduction processing; the ultrasonic signal generated by the photoacoustic scanning microscopy module excites the sample and is collected by a transducer component (such as a hollow ring or prism pair coupling structure), converted into an electrical signal through the piezoelectric effect, and then sent to the data acquisition card, and then transmitted to the controller after analog-to-digital conversion to be saved as a time-domain waveform.

[0116] In the above embodiment, by setting up a data acquisition card, high-speed synchronous acquisition and precise transmission of fluorescence signals and ultrasonic signals are achieved. The high-speed sampling capability of the data acquisition card ensures the synchronous acquisition and precise transmission of the two types of signals, providing hardware support for the spatiotemporal consistency and data fusion accuracy of multimodal imaging.

[0117] In some embodiments, the controller is further configured to collect the fluorescence signal intensity within a preset time period at each scanning collection point, average the fluorescence signal intensities, and obtain fluorescence intensity characteristic information of the scanning collection point;

[0118] The controller is further configured to collect a spectral signal for each of the scanning acquisition points based on the optical coherence tomography signal, and perform Fourier transform on the spectral signal to obtain optical coherence tomography depth information at the scanning acquisition point;

[0119] The controller is further configured to collect the ultrasonic signal intensity within a preset time period at each scanning collection point for the ultrasonic signal, and obtain ultrasonic intensity distribution information at the scanning collection point.

[0120] Specifically, the controller uses a differentiated algorithm to process multimodal signals: for fluorescence signals, the signal intensity within a preset time period is continuously collected for each scanning acquisition point, random noise is eliminated through a time domain averaging algorithm, and the fluorescence intensity characteristic information of the point is obtained after averaging the intensity values, which is used to characterize the distribution characteristics of the sample molecular markers; for optical coherence tomography signals, spectral interference fringes are collected for each scanning acquisition point, and the correspondence between spectral frequency and depth is inverted through a Fourier transform algorithm to obtain the optical coherence tomography depth information of the point, thereby realizing the tomographic analysis of the tissue microstructure; for ultrasonic signals, the signal intensity within a preset time period is continuously collected for each scanning acquisition point, and directly recorded as a time domain waveform to obtain the ultrasonic intensity distribution information of the point, which is used to reflect the spatial distribution of the optical absorption characteristics of the tissue. The three types of algorithms work together to ensure the accurate extraction and physical significance representation of multimodal data.

[0121] In the above embodiment, the controller performs differentiated processing of multimodal signals to achieve accurate analysis and physical property characterization of fluorescence, optical coherence tomography, and ultrasound signals. The three types of algorithms work together to ensure high-precision extraction and biological significance of multimodal data, providing a reliable information basis for subsequent image fusion and biomedical analysis.

[0122] In some embodiments, the optical coherence tomography microscopy module includes:

[0123] a fiber coupler, configured to split the fifth laser beam into two paths, one path to the sample arm and the other path to the reference arm;

[0124] The sample arm is used to transmit the fifth laser beam to the sample to be measured, and return the sample arm optical signal generated by the excitation of the sample to be measured to the linear array camera spectrometer via the optical fiber coupler;

[0125] The reference arm is used to transmit the fifth laser beam to the reference arm reflector, and reflect the reference arm optical signal to the linear array camera spectrometer via the fiber coupler;

[0126] The linear array camera spectrometer is used to receive the sample arm light signal and the reference arm light signal, combine the sample arm light signal and the reference arm light signal into the optical coherence tomography signal, and transmit the signal to the controller for optical coherence tomography imaging data processing.

[0127] Specifically, the optical coherence tomography microscopy module includes a fiber coupler, a sample arm, a reference arm, and a linear array camera spectrometer. Figure 6Fiber coupler 1703 is used to split the fifth laser beam emitted by broadband laser 1701 into two paths: one path is transmitted to the sample arm, and the other path is transmitted to the reference arm. The sample arm transmits the fifth laser beam to the sample under test via fiber jumper 1704 and fiber collimator 1705. The sample arm optical signal generated by the sample under test returns to fiber coupler 1703 along the original path. The reference arm transmits the fifth laser beam to reference arm reflector 1709 via fiber jumper 1706, fiber collimator 1707, and reference arm compensation lens 1708. The reflected reference arm optical signal also returns to fiber coupler 1703. Linear camera spectrometer 1711 receives the sample and reference arm optical signals output by fiber coupler 1703, combines them to form an optical coherence tomography signal, and transmits it to computer 10 for optical coherence tomography data processing. By Fourier transforming the spectral signal, the reflectivity distribution at each tissue depth is inverted, enabling tomographic imaging of tissue microstructure.

[0128] In the above embodiment, the fifth laser beam is divided into a sample arm and a reference arm through an optical fiber coupler. The sample arm transmits the laser to the sample to be tested and returns the sample arm light signal. The reference arm transmits the laser to the reflector and returns the reference arm light signal. The linear array camera spectrometer receives and combines the two types of light signals to form an optical coherence tomography signal, which is transmitted to the controller for Fourier transform processing, thereby realizing high-resolution tomographic imaging of the tissue microstructure of the sample to be tested, and can accurately obtain the reflectivity distribution information at each depth, providing key data support for the tissue layered structure for multimodal imaging.

[0129] In some embodiments, the fluorescence confocal scanning microscopy imaging module includes a first-band fluorescence confocal scanning microscopy imaging module and a second-band fluorescence confocal scanning microscopy imaging module; the multimodal microscopy imaging system further includes a first spectroscopic element, a second spectroscopic element, a third spectroscopic element, and a reflective element;

[0130] The first beam splitter element is configured to reflect the first-band laser beam emitted by the first-band fluorescence confocal scanning microscopy imaging module and transmit the fourth laser beam emitted by the photoacoustic scanning microscopy imaging module to the reflective element;

[0131] The second beam splitter element is configured to reflect the second-band laser beam emitted by the second-band fluorescence confocal scanning microscopy imaging module and transmit the fifth laser beam emitted by the optical coherence tomography microscopy imaging module to the third beam splitter element;

[0132] The reflecting element is used to reflect the first wavelength band laser beam and the fourth laser beam to the third beam splitting element;

[0133] The third beam splitter element is used to reflect the first-band laser beam and the fourth laser beam reflected by the reflective element to the scanning device, and transmit the second-band laser beam and the fifth laser beam combined by the second beam splitter element to the scanning device.

[0134] Specifically, the fluorescence confocal scanning microscopy imaging module includes a visible light band fluorescence scanning confocal microscopy imaging module and a near-infrared band fluorescence scanning confocal microscopy imaging module. The multimodal microscopy imaging system also integrates a first spectroscopic element, a second spectroscopic element, a third spectroscopic element and a reflective element to realize multi-beam combining. The first beam splitter uses a dichroic mirror structure. Based on its wavelength-selective reflection characteristics, it can efficiently reflect the first-band laser beam emitted by the visible-band fluorescence scanning confocal microscopy imaging module, while transmitting the fourth laser beam emitted by the photoacoustic scanning microscopy imaging module. The two beams are then directed to the third beam splitter through a reflective element (such as a reflector). The second beam splitter is also a dichroic mirror, designed to reflect the second-band laser beam (such as the near-infrared band) emitted by the near-infrared fluorescence scanning confocal microscopy imaging module, and transmit the fifth laser beam of the optical coherence tomography microscopy imaging module to the third beam splitter. The third beam splitter serves as the core beam combining element. It achieves wavelength-selective reflection through multi-layer coating, reflecting the combined light of the first-band laser beam and the fourth laser beam to the scanning device, while transmitting the combined light of the second-band laser beam and the fifth laser beam. Finally, the scanning device (such as a two-dimensional galvanometer) drives the multimodal laser beam to focus on the sample to be tested through the objective lens, realizing the common optical path transmission of multi-band fluorescence, photoacoustic, and OCT signals. Figure 3This can be understood as turning on the fluorescence scanning confocal microscopy module (14 or 19) based on the desired fluorescence wavelength band. 14 is for the visible light band, and 19 is for the near-infrared band. Their structures are identical, differing only in the wavelength bands used. The excitation laser light emitted by the visible light fluorescence scanning confocal microscopy module 14 is reflected by dichroic mirror 15 (first beam splitter), then by reflective mirror 13 (reflective element), and finally by dichroic mirror 12 (third beam splitter), before entering the scanning device 11. The excitation laser light emitted by the near-infrared fluorescence scanning confocal microscopy module 19 is reflected by dichroic mirror 18 (second beam splitter), then transmitted through dichroic mirror 12 (third beam splitter), and finally enters the scanning device 11. Turn on the photoacoustic scanning microscopy module 16. The emitted laser beam passes through the dichroic mirror 15 (first beam splitter) and is then combined with the excitation laser beam emitted by the visible-band fluorescence scanning confocal microscopy module 14. Both beams are reflected by the reflector 13 (reflection element) and the dichroic mirror 12 (third beam splitter) before entering the scanning device 11. Turn on the light source of the optical coherence tomography microscopy module 17. The emitted laser beam passes through the dichroic mirror 18 (second beam splitter) and is then combined with the excitation laser beam emitted by the near-infrared-band fluorescence scanning confocal microscopy module 19. Both beams are then transmitted through the dichroic mirror 12 (third beam splitter) and combined with the laser beam emitted by the photoacoustic scanning microscopy module 16 and the laser beam emitted by the visible-band fluorescence scanning confocal microscopy module 14, and both beams enter the scanning device 11. The combined laser beam is then focused onto the sample surface by a telecentric scanning system consisting of the scanning lens 9, the field lens 8, and the objective lens 4. Then, the computer 10 controls the scanning device 11 to drive the laser focus to complete a two-dimensional scan at the focal plane of the objective lens 4. The scanning method adopts a raster-type general two-dimensional scan.

[0135] In the above embodiment, through the synergistic effect of the first spectroscopic element, the second spectroscopic element, the third spectroscopic element and the reflective element, the wavelength-selective combination and common optical path transmission of the first-band fluorescence confocal laser beam, the second-band fluorescence confocal laser beam, the photoacoustic scanning laser beam and the optical coherence tomography laser beam are achieved, so that the multimodal laser beam is focused onto the sample to be measured through the same scanning device and objective lens, solving the system complexity and spatial alignment problems caused by the independent optical paths required for laser beams of different bands in traditional multimodal imaging, realizing the synchronous excitation and acquisition of multimodal signals, and improving the imaging efficiency and spatiotemporal alignment accuracy.

[0136] Figure 7This is a schematic diagram of one implementation of the sample stage in this embodiment; the Z-axis translation stage 101 is located at the bottom, and the Y-axis translation stage 102 and the X-axis translation stage 103 are stacked upward in sequence. Four columns 105 are fixed to the sample placement platform 104 by fastening screws on the side. Above the sample placement platform 104 is a water tank assembly 3, in which distilled water 301 is placed in the water tank, and a film 302 is clamped between the inner layer structure 303 and the outer layer structure 304 of the water tank, and fixed together to the water tank base structure 305. The water tank base structure 305 is connected to the column 105, and by adjusting the fastening screws therebetween, the distance between the water tank assembly 3 and the sample placement platform 104 can be adjusted to achieve a close fit between the film 302 and the sample.

[0137] This embodiment provides a multimodal microscopic imaging method, Figure 8 is a flow chart of a multimodal microscopic imaging method according to an embodiment of the present application, such as Figure 8 As shown, the process includes the following steps:

[0138] Step S801: Control the multimodal imaging module to emit a multimodal laser beam.

[0139] Step S802, controlling the scanning device in the scanning system to move to a specified posture, and controlling the scanning device to transmit the multimodal laser beam through the objective lens in the scanning system to the target area of ​​the sample to be tested for scanning in the specified posture; wherein, after being excited by the surface of the sample to be tested, a multimodal light signal and an ultrasonic signal are returned, and the ultrasonic signal is collected by a transducer assembly, and the transducer assembly is located between the sample to be tested and the objective lens.

[0140] Step S803 : Processing the collected multimodal optical signal and the ultrasonic signal returned by the multimodal imaging module to generate a scanning result image.

[0141] The specific process is as follows: place the sample to be tested on the sample stage; place the water tank above the sample to be tested, with the film at the bottom of the water tank in contact with the sample, and fill the water tank with distilled water; select dark field or bright field illumination mode, turn on the fluorescence / ordinary wide-field microscopy module camera, and switch the module to normal mode; adjust the three-dimensional translation stage, move the sample under the photoacoustic transducer assembly so that the assembly is submerged in water, observe the fluorescence / ordinary wide-field microscopy module camera, and fine-tune the z-axis translation stage until the image is clear; switch the fluorescence / ordinary wide-field microscopy module to fluorescence mode, adjust the xyz-axis translation stage until it is positioned in the observation area; start the fluorescence confocal scanning microscopy module, photoacoustic scanning microscopy module and optical coherence tomography microscopy module for scanning imaging. According to a preset scanning path (such as raster-style progressive scanning), the galvanometer is controlled to move to a specified position, synchronously triggering the laser and detector of the multimodal imaging module. After the laser beam is transmitted through the objective lens to the sample surface, stimulating multimodal optical signals (fluorescence signals, optical coherence tomography signals) and ultrasound signals (collected by the transducer), the data acquisition card and the linear array camera are coordinated in real time to capture the signals. The controller performs differential processing on the returned multimodal signals: time-domain averaging is used to reduce noise on the fluorescence signal, time-domain waveforms are recorded on the ultrasound signal, and depth information is inverted through Fourier transform on the optical coherence tomography signal. Finally, based on the coordinate mapping of the scanning system, the controller fuses these three signals to generate a multimodal scan result image that contains tissue structure, function, and molecular information.

[0142] The sample to be tested is typically biological tissue, including tissue, organs, or whole bodies of experimental animals such as mice, rats, and rabbits. A thin film at the bottom of the water tank contacts the sample, and water is typically added to ensure efficient ultrasonic signal transmission. Switching the module to normal mode involves switching the light source and beam splitter within the module. In fluorescence mode, the light source is typically a specific wavelength, used to excite the sample to produce fluorescence, and is often used in conjunction with a filter. The beam splitter is typically a dichroic mirror, with the excitation light and fluorescence corresponding to the dichroic mirror's transmission and reflection positions, respectively, or the two can be interchanged. In normal mode, the light source is typically broadband. In darkfield illumination, the beam splitter can be eliminated. In brightfield illumination, the beam splitter typically uses a non-polarizing or polarizing beam splitter cube. The transducer assembly is located between the sample to be tested and the objective lens, offering two configurations for acquiring ultrasonic signals. This ensures coaxiality between ultrasonic signal acquisition and optical signal transmission, avoiding spatiotemporal errors caused by signal path delays. Scanning imaging involves the computer initiating a scanning program, and the scanning system performing a two-dimensional scan. The fluorescence confocal scanning microscopy module, the photoacoustic scanning microscopy module, and the optical coherence tomography microscopy module share the same scanning system. The general process of two-dimensional scanning is as follows:

[0143] (A) The computer controls the scanning system to locate to the specified starting position;

[0144] (B) Acquisition signals of each modality: fluorescence and ultrasound signals are collected by a data acquisition card, and optical coherence tomography signals are collected by a linear array camera spectrometer;

[0145] (C) Data processing: Fluorescence signals were averaged, ultrasound signals were not processed, and optical coherence tomography signals were Fourier transformed.

[0146] (D) The computer controls the scanning system to locate to the next specified position along the pre-designed scanning path;

[0147] (E) Repeat steps AD until all specified poses are scanned.

[0148] Scanning systems can be primarily categorized into three types: optical scanning, sample scanning, and hybrid scanning, a combination of the two. Optical scanning involves the sample remaining stationary while the scanning system drives the laser movement to achieve scanning, typically using devices such as galvanometers or rotating mirrors. Sample scanning involves the laser beam remaining stationary while the scanning system drives the sample movement to achieve scanning, typically using a translation stage. Hybrid scanning involves the coordinated movement of the two to achieve scanning, typically requiring the use of specialized algorithms. The scanning path is typically raster scanning, but can also be specially designed based on specific circumstances.

[0149] Through the above steps, multimodal synchronous imaging of the same target area, including fluorescence wide-field microscopy, fluorescence confocal scanning microscopy, photoacoustic scanning microscopy, and optical coherence tomography microscopy, is achieved, solving the position alignment and real-time problems of traditional time-sharing detection of each modality. The design of the transducer assembly located between the sample to be tested and the objective lens ensures the coaxial transmission and spatiotemporal consistency of the ultrasonic signal and the multimodal optical signal, and ultimately efficiently generates a scanning result image that integrates multimodal information.

[0150] In addition, in conjunction with the multimodal imaging module method in the above embodiments, the present application may provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any of the multimodal imaging module methods in the above embodiments.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0152] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0153] Those skilled in the art should understand that the various technical features of the above-described embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multimodal microscopic imaging system, characterized in that: include: multimodal imaging modules, transducer assemblies, scanning systems, and controllers; The multimodal imaging module is used to emit a multimodal laser beam; The scanning system includes a scanning device and an objective lens; The controller is used to control the scanning device to move to a specified position and control the scanning device to transmit the multimodal laser beam through the objective lens to a target area of ​​the sample to be tested for scanning in the specified position; wherein, after being excited by the surface of the sample to be tested, a multimodal optical signal and an ultrasonic signal are returned; The transducer assembly is located between the sample to be tested and the objective lens, and is used to collect the ultrasonic signal; The controller is further configured to process the collected multimodal optical signal and the ultrasonic signal returned by the multimodal imaging module to generate a scanning result image.

2. The multimodal microscopic imaging system according to claim 1, characterized in that: The multimodal imaging module includes a fluorescence / conventional wide-field microscopy imaging module, a fluorescence confocal scanning microscopy imaging module, a photoacoustic scanning microscopy imaging module and an optical coherence tomography microscopy imaging module.

3. The multimodal microscopic imaging system according to claim 1, characterized in that: The transducer assembly is a hollow ring-shaped transducer assembly; The hollow ring transducer assembly includes a window and a hollow ring photoacoustic transducer; the hollow ring photoacoustic transducer assembly is disposed in a water tank; a first photoacoustic coupling liquid is injected into the water tank; the window is located at the liquid level of the first photoacoustic coupling liquid; the hollow ring photoacoustic transducer is used to collect the ultrasonic signal; or, The transducer assembly is a prism-coupled transducer assembly; The prism-coupled transducer assembly comprises a prism group, an acoustic coupling reflection medium, and a wide-field photoacoustic transducer; the lower surface of the prism group is in a second photoacoustic coupling liquid, and the upper surface of the prism group is in air; the prism group comprises a triangular prism and an rhombic prism; the triangular prism is located between the objective lens and the sample to be measured; the acoustic coupling reflection medium is located on the contact surface between the triangular prism and the rhombic prism, and is used to reflect the ultrasonic signal to the wide-field photoacoustic transducer; The wide-field photoacoustic transducer is mounted on the rhombus prism and is used to collect the ultrasonic signal.

4. The multimodal microscopic imaging system according to claim 2, wherein: The working modes of the fluorescence / normal wide-field microscopy imaging module include normal mode and fluorescence mode; The fluorescence / normal wide-field microscopy module is used to switch the working light source of the fluorescence / normal wide-field microscopy module to a normal light source and switch the beam splitter in the fluorescence / normal wide-field microscopy module to a beam splitting cube in the normal mode; The controller is further configured to, when the first laser beam emitted by the ordinary light source is transmitted to the sample to be measured via the beam splitter cube, instruct the scanning device to observe the sample in real time, and adjust the three-dimensional translation stage carrying the sample to be measured until the observed image is clear; The fluorescence / normal wide-field microscopy imaging module is further configured to, in the fluorescence mode, switch the working light source to a fluorescence light source and switch the beam splitter to a dichroic mirror; The controller is further configured to adjust the three-dimensional translation stage to move the target area into the field of view of the objective lens when the second laser beam emitted by the fluorescent light source is transmitted to the sample to be measured via the dichroic mirror.

5. The multimodal microscopic imaging system according to claim 2, characterized in that: The third laser beam emitted by the fluorescence confocal scanning microscopy imaging module generates a fluorescence signal after being excited by the sample to be tested, and returns to the fluorescence confocal scanning microscopy imaging module; The fourth laser beam emitted by the photoacoustic scanning microscopy module generates the ultrasonic signal after being excited by the sample to be tested and returns to the photoacoustic scanning microscopy module; The fifth laser beam emitted by the optical coherence tomography microscopy imaging module is excited by the sample to be tested and returns an optical coherence tomography signal to the optical coherence tomography microscopy imaging module; The controller is further used to collect data of the fluorescence signal, the ultrasound signal and the optical coherence tomography signal; The controller is further configured to process the fluorescence signal, the ultrasound signal, and the optical coherence tomography signal to obtain multimodal information; and generate the scan result image based on the multimodal information.

6. The multimodal microscopic imaging system according to claim 5, characterized in that: The multimodal microscopic imaging system further comprises a data acquisition card; The fluorescence signal is returned to the fluorescence confocal scanning microscopy imaging module and then sent to the data acquisition card, and then transmitted to the controller; The ultrasonic signal is collected by the transducer assembly and sent to the data acquisition card, and then transmitted to the controller.

7. The multimodal microscopic imaging system according to claim 5, characterized in that: The controller is further configured to collect the fluorescence signal intensity within a preset time period at each scanning and collecting point, average the fluorescence signal intensities, and obtain fluorescence intensity characteristic information of the scanning and collecting point; The controller is further configured to collect a spectral signal for each of the scanning acquisition points based on the optical coherence tomography signal, and perform Fourier transform on the spectral signal to obtain optical coherence tomography depth information at the scanning acquisition point; The controller is further configured to collect the ultrasonic signal intensity within a preset time period at each scanning collection point for the ultrasonic signal, and obtain ultrasonic intensity distribution information at the scanning collection point.

8. The multimodal microscopic imaging system according to claim 5, characterized in that: The optical coherence tomography microscopy module includes: a fiber coupler, configured to split the fifth laser beam into two paths, one path to the sample arm and the other path to the reference arm; The sample arm is used to transmit the fifth laser beam to the sample to be measured, and return the sample arm optical signal generated by the excitation of the sample to be measured to the linear array camera spectrometer via the optical fiber coupler; The reference arm is used to transmit the fifth laser beam to the reference arm reflector, and reflect the reference arm optical signal to the linear array camera spectrometer via the fiber coupler; The linear array camera spectrometer is used to receive the sample arm light signal and the reference arm light signal, combine the sample arm light signal and the reference arm light signal into the optical coherence tomography signal, and transmit the signal to the controller for optical coherence tomography imaging data processing.

9. The multimodal microscopic imaging system according to claim 2, characterized in that: The fluorescence confocal scanning microscopy imaging module includes a first-band fluorescence confocal scanning microscopy imaging module and a second-band fluorescence confocal scanning microscopy imaging module; the multimodal microscopy imaging system also includes a first spectroscopic element, a second spectroscopic element, a third spectroscopic element and a reflective element; The first beam splitter element is configured to reflect the first-band laser beam emitted by the first-band fluorescence confocal scanning microscopy imaging module and transmit the fourth laser beam emitted by the photoacoustic scanning microscopy imaging module to the reflective element; The second beam splitter element is configured to reflect the second-band laser beam emitted by the second-band fluorescence confocal scanning microscopy imaging module and transmit the fifth laser beam emitted by the optical coherence tomography microscopy imaging module to the third beam splitter element; The reflecting element is used to reflect the first wavelength band laser beam and the fourth laser beam to the third beam splitting element; The third beam splitter element is used to reflect the first-band laser beam and the fourth laser beam reflected by the reflective element to the scanning device, and transmit the second-band laser beam and the fifth laser beam combined by the second beam splitter element to the scanning device.

10. A multimodal microscopic imaging method, characterized in that: include: controlling the multimodal imaging module to emit a multimodal laser beam; Controlling a scanning device in a scanning system to move to a specified position, and controlling the scanning device to transmit the multimodal laser beam to a target area of ​​a sample to be tested through an objective lens in the scanning system in the specified position for scanning; wherein, after being excited by the surface of the sample to be tested, a multimodal light signal and an ultrasonic signal are returned, and the ultrasonic signal is collected by a transducer assembly, and the transducer assembly is located between the sample to be tested and the objective lens; The collected multimodal optical signal and the ultrasonic signal returned by the multimodal imaging module are processed to generate a scanning result image.

Images