A multimodal imaging device

Abstract

This invention relates to a multimodal imaging device. The multimodal imaging device includes: a Raman spectroscopy analysis module for obtaining Raman spectral information of a target object at a first sampling position using excitation light; an optical coherence tomography (OCT) module for obtaining an image of the target object's tissue structure at a second sampling position using imaging probe light; and a co-localization module for controlling the first sampling position of the excitation light in the Raman spectroscopy analysis module and / or the second sampling position in the OCT module, based on a determined region of interest of the target object, such that the first sampling position and the second sampling position are spatially co-located within the region of interest. The multimodal imaging device of this invention can obtain high spatial resolution image information and high sensitivity and specificity Raman spectral information from OCT imaging in a co-localization manner.

Description

[0001] This application is a divisional application of patent application No. 202111381679.X, filed on November 22, 2021, entitled "A Multimodal Imaging Device". Technical Field

[0002] The present invention relates to optical imaging devices for detection / diagnosis, particularly multimodal imaging devices, and more particularly to endoscopes based on Raman spectroscopy and optical coherence tomography multimodal imaging. Background Technology

[0003] Early screening and detection of cancer, as well as postoperative follow-up, are important means to improve patient survival rates, and endoscopic imaging devices are important imaging diagnostic tools for early cancer screening and detection, as well as postoperative follow-up.

[0004] Optical coherence tomography (OCT) is an observational technique that uses the scattered light from physiological tissues to coherently image them. OCT boasts high spatial resolution (~10 μm), allowing for real-time, non-invasive detection of tissue scattering changes and providing two-dimensional or three-dimensional micrometer-scale tissue structure / morphological information, visualizing infiltration. However, OCT's accuracy in early cancer diagnosis is not high. For example, the sensitivity and specificity of OCT for diagnosing cervical intraepithelial neoplasia are only 88% and 69%, respectively.

[0005] Raman spectroscopy is an analytical technique that utilizes the Raman scattering effect of molecules to obtain information about molecular vibrations and rotations. Raman spectroscopy is correlated with molecular chemical bond information, allowing the identification of different molecular types and the assessment of relative concentration peaks based on varying intensities. Raman spectroscopy offers higher accuracy and specificity in tumor diagnosis than other methods. For example, Raman spectroscopy has a sensitivity of 93.5% and a specificity of 97.8% for diagnosing intraepithelial neoplasia (CIN), a precancerous lesion of the cervix; its sensitivity and specificity for diagnosing early-stage gliomas are as high as 93% and 91%, respectively. In contrast, magnetic resonance imaging (MRI) has a sensitivity and specificity of only 88% and 54%, respectively. However, Raman spectroscopy endoscopy cannot provide wide-field imaging modes like white light endoscopy, narrow-band endoscopy, optical coherence tomography (OCT), autofluorescence imaging (AF), or confocal endoscopy, and therefore cannot visualize suspicious lesion areas during endoscopic examinations.

[0006] Therefore, to improve the efficiency and accuracy of diagnosis / screening, it is desirable to obtain both tissue structure imaging information (optical coherence tomography) and molecular structure information (Raman spectroscopy) with high diagnostic sensitivity and specificity. However, simply obtaining both types of information is insufficient for diagnosis and screening. Only when both types of information originate from the same spatially co-located region (i.e., from essentially the same spatial location) can they be combined to provide meaningful auxiliary information for cancer / tumor diagnosis / screening. If the two types of information originate from spatially mismatched (non-co-located) regions, they represent different information from different regions (although there may be some overlap). In this case, combining the two types of information is not advisable, as it leads to a decrease in the accuracy of spatial information, which contradicts the goal of improving accuracy through combining the two types of information.

[0007] Therefore, it is necessary to obtain spatially colocalized tissue structure imaging information and Raman spectral information. The higher the degree of spatial colocalization of these two types of information, the more beneficial it is to combine the two types of information, thereby improving the accuracy and efficiency of diagnosis / screening.

[0008] Furthermore, the mismatch between the speed of optical coherence tomography (OCT) (e.g., above 100 frames / second) and the speed of Raman spectroscopy (2-5 Hz) prevents them from being used together efficiently. Clearly, due to the slow speed of Raman spectroscopy, even combining OCT with Raman spectroscopy still requires a relatively long time to obtain comprehensive information. Therefore, there is a desire to obtain tissue structure imaging and Raman spectral information at faster speeds, such as by using OCT and Raman spectroscopy in a synergistic manner to improve the efficiency, accuracy, and specificity of cancer screening.

[0009] Finally, it is also desirable that the probe size of such devices be small enough (e.g., at least less than 10 mm) to enable integration with existing endoscopic systems (e.g., white light endoscopes or narrow-band endoscopes). Summary of the Invention

[0010] To address the above-mentioned technical problems, this invention provides a multimodal imaging device. The multimodal imaging device of this invention includes a Raman spectroscopy analysis module, an optical coherence tomography (OCT) module, and a co-localization module. The multimodal imaging device of this invention utilizes the co-localization module to enable the Raman spectroscopy analysis module and the OCT module to image and detect the target object within the same co-localized region. Furthermore, the Raman spectroscopy analysis module and the OCT module in the multimodal imaging device of this invention operate collaboratively, thereby obtaining the diagnostic evidence needed for cancer screening with high efficiency, high accuracy, and high specificity. The design of the multimodal imaging device of this invention also allows for the fabrication of a probe with a smaller size, facilitating integration into existing endoscopic systems.

[0011] According to an embodiment of the present invention, a multimodal imaging device is provided, comprising:

[0012] The Raman spectroscopy analysis module is used to obtain Raman spectral information of the target object at the first sampling position using excitation light;

[0013] An optical coherence tomography module is used to acquire an image of the tissue structure of a target object at a second sampling location using an imaging probe beam; and

[0014] The co-location module is used to control the first sampling position of the excitation light in the Raman spectroscopy analysis module and / or the second sampling position in the optical coherence tomography module according to the determined region of interest of the target object, so that the first sampling position and the second sampling position achieve spatial co-location within the region of interest.

[0015] Optionally, the multimodal imaging device includes a probe having a housing and a detection window, and is used to detect a target object, wherein excitation light from a Raman spectroscopy analysis module and imaging detection light from an optical coherence tomography module are coupled in the probe.

[0016] Optionally, the Raman spectroscopy analysis module includes: a first light source, a first beam splitter, a first coupling objective, a first optical fiber, a spectrometer, a first lens group, and a first dichroic mirror;

[0017] The first beam splitter is used to transmit the excitation light from the first light source and reflect the scattered signal light from the Raman spectrum of the target object. The spectrometer is used to receive the scattered signal light from the Raman spectrum of the target object reflected by the first beam splitter. The first coupling objective is used to receive the outgoing light from the first beam splitter or from the co-positioning module. The first optical fiber is used to receive the outgoing light from the first coupling objective. The first lens group is used to receive the outgoing light from the first optical fiber. The first dichroic mirror is used to receive and transmit the outgoing light from the first lens group.

[0018] Optionally, a grating and a receiving lens are disposed between the spectrometer and the first beam splitter. The grating is used to split the reflected light from the first beam splitter, the receiving lens is used to receive the emitted light from the grating, and the spectrometer is used to receive the emitted light from the receiving lens.

[0019] Optionally, the first lens group includes a first collecting lens.

[0020] Optionally, the first optical fiber includes a multi-core optical fiber, wherein a central core group consisting of at least one core of the central portion of the multi-core optical fiber is used to transmit excitation light of the Raman spectrum from the first light source, and a plurality of peripheral core groups consisting of groups of at least one core of the multi-core optical fiber surrounding the central portion are used to transmit scattered signal light of the Raman spectrum.

[0021] Optionally, the peripheral fiber core groups are symmetrically distributed around the central fiber core group.

[0022] Optionally, the central fiber core assembly is provided with a bandpass filter at its end near the target object, and the peripheral fiber core assembly is provided with a notch filter and / or a long-pass filter at its end near the target object.

[0023] Alternatively, the cross-sections of the central core assembly and the peripheral core assembly are substantially circular.

[0024] Optionally, the optical coherence tomography module includes a second light source, a beam splitter, an interferometer, a second optical fiber, a scanning submodule, a detector, a second lens group, and a reflector;

[0025] The second light source, interferometer, detector, and scanning submodule are optically coupled to the beam splitter; the scanning submodule is optically coupled to the beam splitter via a second fiber optic cable.

[0026] A portion of the second optical fiber passes through the scanning submodule;

[0027] The second lens group is used to receive the outgoing light from the second optical fiber, the reflector is used to reflect the outgoing light from the second lens group, and the first dichroic mirror is used to reflect the reflected light from the reflector, thereby coupling the light from the first lens group with the light from the second lens group.

[0028] Optionally, the second light source and / or interferometer and / or detector are optically coupled to the beam splitter via a coupling fiber.

[0029] Optionally, the first lens group and the second lens group are arranged in parallel inside the probe, and the first dichroic mirror, the scanning submodule, the reflector and at least a portion of the second optical fiber are located inside the probe.

[0030] Optionally, the scanning submodule is used to control the position of the imaging probe light of the second light source to obtain the position of the tissue structure image of the target object by controlling the position of the second optical fiber.

[0031] Optionally, the scanning submodule includes a piezoelectric ceramic tube.

[0032] Optionally, the second lens group includes a second focusing lens and a diffraction lens, with the diffraction lens positioned between the second focusing lens and the reflecting mirror.

[0033] Optionally, a circulator is provided between the detector and the beam splitter.

[0034] Optionally, the second optical fiber and / or the coupling optical fiber may include single-mode optical fiber.

[0035] Optionally, the co-location module is located in the excitation light incident light path of the Raman spectroscopy analysis module.

[0036] Optionally, the co-positioning module is located between the first beam splitter and the first coupling objective.

[0037] Optionally, the co-positioning module has a switchable first mode and a second mode;

[0038] In the first mode, the co-localization module does not change the first sampling position;

[0039] In the second mode, the co-location module is used to control the first sampling position.

[0040] Optionally, the co-positioning module includes a first flip mirror, a second flip mirror, a first scanning galvanometer, and a second scanning galvanometer; the first flip mirror and the second flip mirror are disposed between the first beam splitter and the first coupling objective.

[0041] The first and second flip mirrors are used to control whether the mirror surfaces of the first and second flip mirrors are parallel or non-parallel to the optical path between the first beam splitter and the first coupling objective by rotating about an axis orthogonal to the optical path between the first beam splitter and the first coupling objective; the first and second scanning galvanometers are used to control the first sampling position by rotating about different axes.

[0042] In the first mode, the mirror surfaces of the first and second flip mirrors are parallel to the optical path between the first beam splitter and the first coupling objective; and

[0043] In the second mode, the mirror surfaces of the first and second flip mirrors are not parallel to the optical path between the first beam splitter and the first coupling objective.

[0044] Optionally, in the second mode, the first flip mirror is used to receive and reflect the light transmitted through the first beam splitter, the first scanning mirror is used to receive and reflect the reflected light from the first flip mirror, the second scanning mirror is used to receive and reflect the reflected light from the first scanning mirror, the second flip mirror is used to receive and reflect the reflected light from the second scanning mirror, and the first coupling objective is used to receive the reflected light from the second flip mirror.

[0045] Optionally, the first and second scanning mirrors include a Galvo mirror, a MEMS-driven reflector, or a resonant mirror.

[0046] Optionally, the multimodal imaging device in one embodiment of the present invention further includes: a detection lens and a detection optical fiber;

[0047] The Raman spectroscopy analysis module includes: a first light source, a first beam splitter, a first coupling objective, a spectrometer, and a second dichroic mirror;

[0048] The optical coherence tomography module includes: a second light source, a beam splitter, an interferometer, a remote scanning submodule, and a detector;

[0049] The first beam splitter, the co-positioning module, the second dichroic mirror, and the first coupling objective are arranged sequentially along the transmission direction of the emitted light from the first light source.

[0050] The first beam splitter is used to transmit excitation light from the first light source and reflect scattered signal light from the Raman spectrum of the target object. The spectrometer is used to receive the scattered signal light from the Raman spectrum of the target object reflected by the first beam splitter. The second dichroic mirror is used to transmit the outgoing light from the co-positioning module or the outgoing light from the first beam splitter and reflect the imaging probe light from the second light source, thereby coupling the two.

[0051] The first coupling objective is used to receive the coupled light from the second dichroic mirror;

[0052] The probe fiber is used to receive the outgoing light from the first coupling objective;

[0053] The probe lens is used to receive the outgoing light from the probe fiber;

[0054] The remote scanning submodule is positioned between the second dichroic mirror and the beam splitter and is used to receive and reflect imaging probe light transmitted from the second light source through the beam splitter;

[0055] The second light source, interferometer, and detector are optically coupled to the beam splitter.

[0056] Optionally, the second light source and / or interferometer and / or detector are optically coupled to the beam splitter via a coupled fiber optic cable.

[0057] Optionally, the multimodal imaging device includes a probe, and a detection lens and at least a portion of the detection optical fiber are disposed inside the probe.

[0058] Optionally, the co-positioning module has a switchable first mode and a second mode;

[0059] In the first mode, the co-localization module does not change the first sampling position;

[0060] In the second mode, the co-location module is used to control the first sampling position.

[0061] Optionally, the co-positioning module includes a first flip mirror, a second flip mirror, a first scanning galvanometer, and a second scanning galvanometer; the first flip mirror and the second flip mirror are disposed between the first beam splitter and the second dichroic mirror.

[0062] The first and second flip mirrors are used to control whether the mirror surfaces of the first and second flip mirrors are parallel or non-parallel to the optical path between the first beam splitter and the second dichroic mirror by rotating about an axis orthogonal to the optical path between the first beam splitter and the second dichroic mirror; the first and second scanning galvanometers are used to control the first sampling position by rotating about different axes.

[0063] In the first mode, the mirror surfaces of the first and second flip mirrors are parallel to the optical path between the first beam splitter and the second dichroic mirror; and

[0064] In the second mode, the mirror surfaces of the first and second flip mirrors are not parallel to the light path between the first beam splitter and the second dichroic mirror.

[0065] Optionally, in the second mode, the first flip mirror is used to receive and reflect the light transmitted through the first beam splitter, the first scanning mirror is used to receive and reflect the reflected light from the first flip mirror, the second scanning mirror is used to receive and reflect the reflected light from the first scanning mirror, the second flip mirror is used to receive and reflect the reflected light from the second scanning mirror, and the second dichroic mirror is used to receive and transmit the reflected light from the second flip mirror.

[0066] Optionally, the first and second scanning mirrors include a Galvo mirror, a MEMS-driven reflector, or a resonant mirror.

[0067] Optionally, the remote scanning submodule is used to control the imaging probe light of the second light source to obtain the position of the tissue structure image of the target object by rotating about at least two axes.

[0068] Optionally, the probe fiber may include a multi-core fiber.

[0069] A central core assembly, consisting of at least one core from the central portion of a multi-core optical fiber, is used to transmit imaging probe light from a second light source and light from the target object for obtaining an image of the target object's tissue structure.

[0070] Multiple peripheral core groups, each consisting of at least one core of a multi-core optical fiber surrounding the central portion, are used to transmit excitation light from the Raman spectrum of a first light source and scattered signal light from the Raman spectrum, respectively.

[0071] Optionally, the peripheral fiber core group for transmitting the excitation light of the Raman spectrum and the peripheral fiber core group for transmitting the scattered signal light of the Raman spectrum in the multimodal imaging device are arranged alternately.

[0072] Optionally, the central fiber core assembly and the peripheral fiber core assembly for transmitting the excitation light of the Raman spectrum are provided with bandpass filters at their ends near the target object, and the peripheral fiber core assembly for transmitting the scattered signal light of the Raman spectrum is provided with notch filters and / or long-pass filters at its ends near the target object.

[0073] Alternatively, the cross-sections of the central core assembly and the peripheral core assembly are substantially circular.

[0074] Optionally, the cross-sectional area of ​​the central core group is greater than the cross-sectional area of ​​the single peripheral core group for transmitting the excitation light of the Raman spectrum and the cross-sectional area of ​​the single peripheral core group for transmitting the scattered signal light of the Raman spectrum.

[0075] Optionally, the remote scanning submodule includes a remote scanning galvanometer.

[0076] Optionally, the remote scanning mirror includes a MEMS-driven reflector, a Galvo mirror, or a resonant mirror.

[0077] Optionally, the detection lens includes a detection focusing lens.

[0078] Optionally, a grating and a receiving lens are disposed between the spectrometer and the first beam splitter. The grating is used to split the reflected light from the first beam splitter, the receiving lens is used to receive the emitted light from the grating, and the spectrometer is used to receive the emitted light from the receiving lens.

[0079] Optionally, the first light source and the second light source are respectively provided with a first light source switch and a second light source switch, and an optional intermediate reflector is provided between the first light source and the first beam splitter to reflect the excitation light from the first light source to the first beam splitter.

[0080] Optionally, the co-localization module is used to move the first sampling position to substantially coincide with the position of the region of interest.

[0081] Optionally, the first lens group is used to ensure that the size of the excitation light from the Raman spectroscopy analysis module at the first sampling position is substantially the same as the size of the region of interest.

[0082] Optionally, the co-localization module is used to move the first sampling position to a position that substantially overlaps with the region of interest.

[0083] Optionally, the co-positioning module is used to synchronously control the first sampling position and the second sampling position with the remote scanning submodule so that the first sampling position and the second sampling position substantially overlap.

[0084] Optionally, the multimodal imaging device of the present invention further includes:

[0085] The image processing module is used to fuse Raman spectral information from the first sampling location and tissue structure images from the second sampling location, in order to generate fused multimodal information of the region of interest.

[0086] Optionally, the multimodal imaging device of the present invention is an endoscope.

[0087] Optionally, the diameter of the probe of the multimodal imaging device of the present invention is 2-10 mm.

[0088] Optionally, the diameter of the probe of the multimodal imaging device of the present invention is 2-5 mm.

[0089] Optionally, the region of interest of the target object is determined from an image of the target object acquired by an imaging device different from the multimodal imaging device, from spectral information acquired by a Raman spectroscopy analysis module, or from an image acquired by an optical coherence tomography module.

[0090] Optionally, the imaging device, different from the multimodal imaging device, includes a white light endoscope module and / or a narrowband imaging module.

[0091] Optionally, the region of interest is a medically relevant region.

[0092] The multimodal imaging device according to the present invention utilizes a colocalization module to enable the Raman spectroscopy analysis module and the optical coherence tomography module to detect / inspect the same area, thereby obtaining spatially colocalized tissue structure images and Raman spectral information.

[0093] Furthermore, the multimodal imaging device of the present invention utilizes a co-localization module to enable the Raman spectroscopy analysis module and the optical coherence tomography (OCT) module to operate collaboratively in a highly efficient manner. In cancer diagnosis, it can simultaneously leverage the accuracy and specificity of the Raman spectroscopy analysis module and the high spatial resolution of the OCT module in obtaining tissue structure information in two or three dimensions, while avoiding the problem of Raman spectroscopy detection being slower than OCT. In this case, spatial co-localization of Raman spectroscopy analysis and OCT is also achieved.

[0094] Finally, the probe of the present invention can be made in a small size, especially the probe in another embodiment of the present invention described above can even be made in a small size of 2-5 mm, which allows the imaging device of the present invention to be easily integrated into existing endoscope systems. Attached Figure Description

[0095] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0096] Figure 1 A block diagram of a multimodal imaging device according to an embodiment of the present invention is shown, wherein the portion within the dashed box corresponds to the multimodal imaging device according to an embodiment of the present invention.

[0097] Figure 2 A schematic diagram of a multimodal imaging apparatus according to one embodiment of the present invention is shown.

[0098] Figure 3 An alternative fiber arrangement in a multi-core optical fiber is shown in a multimodal imaging apparatus according to one embodiment of the present invention.

[0099] Figure 4a This paper illustrates a co-location detection process that actually occurs during the operation of a multimodal imaging device according to one embodiment of the present invention.

[0100] Figure 4b This illustrates another co-localization detection process that actually occurs during the operation of a multimodal imaging device according to one embodiment of the present invention.

[0101] Figure 5 A schematic diagram of a multimodal imaging apparatus according to another embodiment of the present invention is shown.

[0102] Figure 6 An alternative fiber arrangement in a multi-core optical fiber is shown in a multimodal imaging apparatus according to another embodiment of the present invention.

[0103] Figure 7a This illustrates a co-location detection process that actually occurs during the operation of a multimodal imaging device according to another embodiment of the present invention.

[0104] Figure 7b This illustrates another co-localization detection process that actually occurs during the operation of a multimodal imaging device according to another embodiment of the present invention.

[0105] Explanation of reference numerals in the attached figures:

[0106] 100: Raman Spectroscopy Analysis Module

[0107] 101: First Light Source

[0108] 102: First beam splitter

[0109] 103: First coupling objective lens

[0110] 104: First optical fiber

[0111] 105: First lens group

[0112] 115: First collecting lens

[0113] 106: First dichroic mirror

[0114] 106': Second dichroic mirror

[0115] 107: Raster

[0116] 108: Receiving lens

[0117] 109: Spectrometer

[0118] 110: Intermediate reflector

[0119] SW1: First light source switch

[0120] 200: Optical Coherence Tomography Module

[0121] 201: Second Light Source

[0122] 202: Beam splitter

[0123] 203: Interferometer

[0124] 204: Coupled fiber

[0125] 204': Second optical fiber

[0126] 205: Scanning Submodule

[0127] 205': Remote scanning submodule

[0128] 206: Detector

[0129] 207: Second lens group

[0130] 217: Second focusing lens

[0131] 227: Diffractive lens

[0132] 208: Reflector

[0133] 209: Circulator

[0134] SW2: Second light source switch

[0135] 300: Co-positioning module

[0136] 301: First Flip Mirror

[0137] 302: Second Flip Mirror

[0138] 303: First scanning galvanometer

[0139] 304: Second scanning galvanometer

[0140] 400: Probe

[0141] 401: Casing

[0142] 402: Detection Window

[0143] 501: Detection Fiber

[0144] 502: Detector Lens

[0145] A1: Area of ​​Concern

[0146] A1': Sampling area

[0147] NBI: Narrowband Imaging Module

[0148] WLR: White Light Endoscopy Module

[0149] BP: Bandpass filter

[0150] NF: Notch filter

[0151] BP-oct: A bandpass filter located at the end of the central fiber core assembly near the target object for transmitting imaging probe light.

[0152] BP-rs: A bandpass filter located at the end of the outer core assembly used to transmit excitation light for Raman spectroscopy, near the target object. Detailed Implementation

[0153] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely one embodiment of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0154] In this specification and accompanying drawings, substantially the same or similar steps and elements are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0155] In existing technologies, Raman spectroscopy and optical coherence tomography (OCT) are known to be used for the diagnosis / screening of cancer / tumors. However, the information obtained by Raman spectroscopy and OCT needs to be spatially colocalized; that is, Raman spectroscopy and OCT should detect / investigate the same area in space. Otherwise, obtaining spatially inconsistent Raman spectroscopy and OCT information is detrimental to improving the accuracy (e.g., spatial accuracy) and efficiency of diagnosis / screening.

[0156] On the other hand, Raman spectroscopy detection is slow, and obtaining Raman spectral information and optical coherence tomography simultaneously may take too much time.

[0157] Finally, there is no existing technology that provides a multimodal imaging device based on Raman spectroscopy and optical coherence tomography that can use a smaller probe.

[0158] To address the aforementioned technical problems, this invention provides a multimodal imaging device. In addition to a Raman spectroscopy analysis module and an optical coherence tomography (OCT) module, the multimodal imaging device also includes a co-localization module. This co-localization module can control the sampling position of the Raman spectroscopy analysis module and / or the OCT module for detecting the target object. Therefore, by controlling the sampling position of the target object, the multimodal imaging device of this invention can achieve spatial co-localization detection of Raman spectroscopy and OCT.

[0159] On the other hand, the multimodal imaging device of the present invention can also reduce the area that needs to be detected by Raman spectroscopy analysis, that is, it is not necessary to perform Raman spectroscopy analysis on the entire region. For example, only the first sampling position of the Raman spectrum needs to be controlled by the co-localization module to analyze the region of interest in the image obtained by the optical coherence tomography module. This largely avoids the disadvantage of slow detection speed of Raman spectroscopy analysis, while still utilizing the advantages of high accuracy and high specificity of Raman spectroscopy analysis, thereby improving the overall detection efficiency in a synergistic manner. Obviously, in this method, both types of spatial co-localization information are also obtained.

[0160] Finally, the multimodal imaging device of the present invention incorporates a focusing lens in the probe, which reduces the size of the probe and facilitates its integration into existing endoscope systems.

[0161] The operating device provided by the present invention will now be described in detail with reference to the accompanying drawings.

[0162] Figure 1 A schematic diagram of a multimodal imaging apparatus according to an embodiment of the present invention is shown.

[0163] Reference Figure 1The multimodal imaging device may include a Raman spectroscopy analysis module 100, an optical coherence tomography module 200, a co-localization module 300, and a probe 400. Figure 1 The solid lines schematically illustrate the optical paths of the excitation light from the Raman spectrum of the Raman spectroscopy analysis module 100 and the imaging probe light from the optical coherence tomography module 200, connecting the various modules (including the probe). The excitation light and imaging probe light of the Raman spectrum are coupled within the probe to detect and identify the target object. Based on... Figure 1 In one embodiment, the region of interest of the target object (e.g., a possible lesion area in the patient's body) is obtained by an imaging device (not shown, e.g., a white light endoscope module and / or narrow band imaging module of an endoscope) that is different from the multimodal imaging device.

[0164] In one embodiment, for example, in the image of the target object obtained by the endoscope white light endoscope module and / or narrow band imaging module, a predetermined area can be determined as the area of ​​interest for the target object through manual operation by the doctor.

[0165] In another embodiment, the processing module of the multimodal imaging device can also determine a predetermined region as the region of interest of the target object in the image of the target object obtained by the endoscope white light endoscope module and / or narrowband imaging module through a predetermined image processing algorithm.

[0166] The co-localization module 300 controls the first sampling position of the excitation light in the Raman spectroscopy analysis module 100 to move it to substantially coincide with the region of interest, based on the obtained region of interest. The optical coherence tomography (OCT) module 200 performs imaging detection on the first sampling position (i.e., the region of interest) to achieve spatial co-localization of the first sampling position and the second sampling position (from the OCT module 200) within the region of interest. However, the present invention is not limited to this specific embodiment. For example, although not shown, the co-localization module 300 of the present invention can also control the second sampling position of the OCT module 200 to move it to substantially coincide with the region of interest, and use the Raman spectroscopy analysis module 100 to analyze and detect the sampling position, thereby achieving spatial co-localization within the region of interest. Furthermore, embodiments of the present invention may also include simultaneously and / or synchronously controlling the first and second sampling positions to scan and analyze the region of interest.

[0167] According to Figure 1In another embodiment, the optical coherence tomography module 200 is used to obtain an image of the tissue structure of the target object and determine the region of interest of the target object using an imaging probe light. The colocalization module 300 is used to control the (first) sampling position of the excitation light in the Raman spectroscopy analysis module 100 according to the determined region of interest, thereby obtaining Raman spectral information at different positions in the region of interest. However, the present invention is not limited thereto. For example, although not shown, the Raman spectroscopy analysis module 100 can also be used to determine the region of interest of the target object and control the second sampling position of the optical coherence tomography module 200 to scan and image within the region of interest, thereby obtaining an image of the tissue structure and axial information within the region of interest. Obviously, whether controlling the first sampling position of the Raman spectroscopy analysis module 100 or controlling the second sampling position of the optical coherence tomography module 200, the present invention can obtain spatially colocalized Raman spectral information and tissue structure images by using the colocalization module 300. Those skilled in the art can adopt corresponding implementation methods based on actual conditions.

[0168] Figure 1 The relationships between the various modules are merely illustrative and exemplary, and do not limit the specific control methods and connection relationships of the co-positioning module 300.

[0169] pass Figure 1 As can be seen, the co-localization module 300 of the present invention enables the operator to control the first sampling position of the excitation light of the Raman spectrum, thereby making the first sampling position substantially coincide with the region of interest, and then making the second sampling position cover (e.g., by scanning) the first sampling position, thus obtaining spatially co-localized Raman spectral information and tissue structure / morphology images. Furthermore, in a variant embodiment, the optical coherence tomography module 200 can be used to quickly obtain tissue structure images and determine the region of interest of the target object, and then further guide the sampling position of the excitation light of the Raman spectrum to achieve highly accurate and specific diagnostic information for high-risk locations of the target object.

[0170] Figure 2 A schematic diagram of a multimodal imaging apparatus according to one embodiment of the present invention is shown.

[0171] Specifically, see Figure 2The first light source 101 of the Raman spectroscopy analysis module 100 uses a Raman excitation source with a wavelength of 785 nm. As the first light source 101, any light source known in the art suitable for Raman spectroscopy can be selected; the 785 nm Raman excitation source in this embodiment is merely an example. A first beam splitter 102, a co-positioning module 300, a first coupling objective lens 103, a first optical fiber 104, a first lens group 105, and a first dichroic mirror 106 are arranged along the excitation light emission path of the Raman spectrum. The first lens group 105 consists of a first collecting lens 115. The Raman spectroscopy analysis module 100 also includes a grating 107 for splitting the reflected light from the first beam splitter 102, a receiving lens 108 for receiving the emitted light from the grating 107, and a spectrometer 109 for receiving the emitted light from the receiving lens 108.

[0172] By adjusting the parameters of the first collecting lens 115, the size of the Raman spectrum spot on the target object can be controlled. This spot size can be adjusted within the range of 5 μm to 1 mm (diameter) as needed. In a preferred embodiment, the first collecting lens 115 ensures that the spot size of the excitation light from the Raman spectrum of the first light source 101 is substantially the same as the size of the region of interest. Alternatively, in an alternative embodiment, the first lens group may also use a focusing lens, including a focusing lens with high dispersion and / or high numerical aperture. For example, the effective focal length of the focusing lens can be 2-3 mm, the working distance can be 1 mm, and the numerical aperture N / A = 0.5. High dispersion lenses can increase the axial field of view, and high numerical aperture is beneficial for improving resolution and imaging signal-to-noise ratio. In this embodiment of the invention, it is preferred to use the first collecting lens 115 and ensure that the spot size of the excitation light from the Raman spectrum of the first light source 101 is consistent with the size of the region of interest, as this saves time in obtaining Raman spectral information.

[0173] Using both high-dispersion lenses and gratings for beam splitting allows for the simultaneous acquisition of Raman spectral information at different depths of the target object, thus improving detection speed.

[0174] Figure 2A schematic, enlarged cross-section of the first optical fiber 104 near the target object is shown. Inside the probe 400, a central core group, consisting of at least one core from the central portion of a multi-core optical fiber, transmits excitation light from the Raman spectrum of the first light source 101. Multiple peripheral core groups, each consisting of at least one core from the central portion of the multi-core optical fiber, transmit the scattered signal light from the Raman spectrum. A band-pass filter (BP, denoted as BP) is provided at the end of the central core group near the target object, and a notch-filter (NF, denoted as NF) is provided at the end of the peripheral core groups near the target object. In other words, the excitation light from the Raman spectrum excites the Raman spectrum of the target object through the band-pass filter (BP), and the Raman scattered light signal is filtered out for background noise by the notch-filter (NF) and long-pass filter (not shown). This arrangement advantageously improves the signal-to-noise ratio.

[0175] For acquiring Raman spectral information, this invention is not limited to the specific fiber arrangement described above. (See attached drawings of this invention.) Figure 3 Several other fiber optic deployment methods are given in the document.

[0176] However, the preferred fiber arrangement of the present invention is shown in the accompanying drawings. Figure 2 The arrangement shown is preferred in this invention, which is centrally symmetric and thus advantageous for obtaining better Raman spectral signals than other arrangements (e.g., arrangements with lower symmetry). For example, it has been found that using the accompanying drawings of this invention... Figure 2 The arrangement shown allows for a higher signal-to-noise ratio compared to the following arrangement: the optical fiber for transmitting the scattered signal of the Raman spectrum is arranged asymmetrically relative to the optical fiber for transmitting the excitation light of the Raman spectrum, for example, there are gaps around the optical fiber for transmitting the excitation light of the Raman spectrum where no optical fiber for transmitting the scattered signal of the Raman spectrum is located (in the case where a portion of the outer surface of the optical fiber for transmitting the excitation light of the Raman spectrum is in contact with the fiber wall), for example including... Figure 3 (The arrangement shown).

[0177] The optical coherence tomography (OCT) module 200 includes a second light source 201, a beam splitter 202, an interferometer 203, a coupling fiber 204, a second fiber 204', a scanning submodule 205, a detector 206, a second lens group 207, a mirror 208, and a circulator 209. As the second light source, any light source known in the art for optical coherence tomography can be used. As an example, the second light source 201 uses a swept-frequency light source including a wavelength of 1325 nm, and the detector 206 is a balanced photodetector. The second lens group 207 consists of a second focusing lens 217 and a diffraction lens 227. The scanning submodule 205 is a piezoelectric ceramic tube.

[0178] By using the second focusing lens 217, the spot size of the incident light emitted from the second light source 201 on the target object can be controlled. This spot size can be adjusted within the range of 5 μm to 1 mm (diameter) as needed. Furthermore, the focusing lens that can be used includes a focusing lens with high dispersion and / or high numerical aperture. For example, the effective focal length of the focusing lens that can be used can be 2-3 mm, the working distance can be 1 mm, and the numerical aperture N / A = 0.5. A high dispersion lens can increase the axial field of view, and a high numerical aperture is beneficial for improving resolution and imaging signal-to-noise ratio. Additionally, in this embodiment of the invention, it has been found that the diffraction lens 227 can compensate for dispersion, improve wavelength bandwidth, and enhance resolution, thereby improving image quality.

[0179] Piezoelectric ceramic tubes are known scanning devices in the art. In the scanning submodule 205 of this invention, the optical fiber and the piezoelectric ceramic tube are coaxially fixed. When the piezoelectric ceramic tube is pressurized, it undergoes piezoelectric deformation, causing the single-mode fiber head to bend. Two-dimensional scanning is achieved when the applied voltage to the two pairs (four-axis) electrodes of the piezoelectric ceramic tube along the x and y axes is a modulation voltage. The scanning amplitude reaches its maximum when the vibration frequency of the axis of the piezoelectric ceramic tube resonates with the self-frequency of a single single-mode fiber. Therefore, the scanning area size and scanning speed are controlled by controlling the amplitude and frequency of the voltage applied to the piezoelectric ceramic tube. When the modulation voltage applied to the x and y axes is an orthogonal sinusoidal signal, a spiral scanning trajectory is generated. Using piezoelectric ceramic tube scanning can achieve rapid 3D-OCT imaging of three-dimensional tissue structures.

[0180] Preferably, the device of the present invention includes an acquisition system based on a high-speed digital-to-analog converter and an FPGA in the detector 206 section, which can realize the reconstruction and display of the organizational structure image of the video stream. The FPGA system converts the probe light interference signal into a sample structure grayscale image and an attenuation coefficient grayscale image, and transmits them to the host computer for display.

[0181] The co-positioning module 300 includes a first flip mirror 301 and a second flip mirror 302 disposed between the first beam splitter 102 and the first coupling objective lens 103, and a first scanning galvanometer 303 and a second scanning galvanometer 304 paired with the two flip mirrors.

[0182] The probe 400 includes a housing 401 and a detection window 402.

[0183] A portion of the first optical fiber 104, the first lens group 105, and the first dichroic mirror 106 are disposed inside the probe 400; a portion of the second optical fiber 204', the scanning submodule 205, the second lens group 207, and the reflector 208 are disposed inside the probe 400.

[0184] The probe design of this invention allows the inner diameter of the probe to be reduced to 2-10mm (far smaller than the 5cm size used in traditional optical path designs in the prior art), which is beneficial for integration into the working channel of existing endoscope systems, reduces the damage that may be caused by endoscopic probes, and is beneficial for clinical use.

[0185] The following describes the operation of a multimodal imaging device according to one embodiment of the present invention.

[0186] The excitation light from the first light source 101 passes sequentially through the first beam splitter 102 and the co-positioning module 300, and after being collimated by the first coupling objective lens 103, enters the first optical fiber 104. The outgoing light from the first optical fiber 104 passes through the first collecting lens 115; wherein the first optical fiber 104 is a multi-core optical fiber. The parameters of the first collecting lens 115 can be selected to control the spot size of the Raman spectroscopy detection light. In one embodiment of the present invention, the spot diameter can be adjusted within the range of 5 micrometers to 1 millimeter. The outgoing light from the first collecting lens 115 is then coupled with the imaging detection light from the optical coherence tomography module 200 after passing through the first dichroic mirror 106, and then passes through the detection window 402 to detect the target object. The Raman spectral scattered light from the target object returns along a light path that is substantially the same as the excitation light, is reflected by the first beam splitter 102, and is incident on the grating 107 and split. The outgoing light from the grating 107 passes through the receiving lens 108 and is detected by the spectrometer 109.

[0187] Imaging probe light from the second light source 201 passes through beam splitter 202 and then enters probe 400 via second optical fiber 204'. The outgoing light from second optical fiber 204' in probe 400 enters second focusing lens 217 and then passes through diffraction lens 227 before being incident on mirror 208. The parameters of second focusing lens 217 can be selected to control the spot size of the imaging probe light. In one embodiment of the invention, the spot diameter can be adjusted within the range of 5 micrometers to 1 millimeter. It has been found that using diffraction lens 227 in the second lens group 207 improves imaging resolution. Outgoing light from mirror 208 is incident on first dichroic mirror 106, thereby coupling with Raman spectral excitation light passing through first dichroic mirror 106, and then imaging the target object through probe window 402. In this embodiment, second optical fiber 204' is coaxially fixed with piezoelectric ceramic tube, which includes two pairs of axes totaling four, arranged in a square around second optical fiber 204'. Applying a voltage to the piezoelectric ceramic tube induces piezoelectric deformation, which in turn causes the fiber optic head to bend. Two-dimensional scanning is achieved when the applied voltage to the two pairs (quad-axis) electrodes of the piezoelectric ceramic tube along the x-axis and y-axis is a modulation voltage. The scattered light from the sample returns along a path that is essentially the same as the incident light, and is detected by the detector 206 after passing through the beam splitter 202, interferometer 203, and circulator 209.

[0188] In this embodiment, the co-positioning module 300 can switch between a first mode and a second mode by adjusting the angles of the first flip mirror 301 and the second flip mirror 302 relative to the optical path 103 between the first beam splitter 102 and the first coupling objective lens.

[0189] In the first mode, when the mirror surfaces of the first flip mirror 301 and the second flip mirror 302 are parallel to the optical path between the first beam splitter 102 and the first coupling objective 103, the presence of the co-positioning module 300 does not affect the incident direction of the optical path between the first beam splitter 102 and the first coupling objective 103, thus not affecting the first sampling position of the Raman spectral excitation light on the target object.

[0190] In the second mode, when the mirror surfaces of the first flip mirror 301 and the second flip mirror 302 are not parallel to the optical path between the first beam splitter 102 and the first coupling objective lens 103, for example... Figure 2When the angle is set as shown, the first flip mirror 301 is used to receive and reflect the light transmitted through the first beam splitter 102, the first scanning mirror 303 is used to receive and reflect the reflected light from the first flip mirror 301, the second scanning mirror 304 is used to receive and reflect the reflected light from the first scanning mirror 303, the second flip mirror 302 is used to receive and reflect the reflected light from the second scanning mirror 304, and the first coupling objective 103 is used to receive the reflected light from the second flip mirror. The co-positioning module 300 of the second mode affects the excitation light of the Raman spectrum. For example, rotating the first scanning mirror 303 and / or the second scanning mirror 304 around a preset axis will cause the emitted light from the first beam splitter 102 to deviate from the direction of the original optical path (i.e., the direction of the optical path in the first mode) at a certain angle, which will cause the position of the incident light of the first coupling objective 103 to also change. This results in changing the sampling position of the excitation light of the Raman spectrum on the target object. In this embodiment, the first scanning mirror 303 and the second scanning mirror 304 can rotate along mutually orthogonal axes, respectively.

[0191] However, the present invention is not limited thereto. Those skilled in the art can, based on the above disclosure, set the spatial orientation of the axes of the first scanning mirror 303 and the second scanning mirror 304 according to actual conditions or needs. An axis with a certain orientation in a given coordinate system can be selected, thereby influencing the position / angle / shape of the excitation light in the Raman spectrum in different ways by rotating the first scanning mirror 303 and the second scanning mirror 304. In this embodiment, the first scanning mirror 303 and the second scanning mirror 304 use MEMS-driven mirrors. However, the present invention is not limited thereto. Those skilled in the art can, based on the above disclosure, use other optical elements with equivalent functions, such as Galvo mirrors, according to actual conditions or needs.

[0192] The multimodal imaging device of this embodiment of the present invention has a variety of operating modes, of which only a few are listed below by way of example.

[0193] For the first operating mode, the co-location detection process can be found in [reference needed]. Figure 4aIn this operating mode, the second focusing lens 217 makes the spot size of the optical coherence tomography (OCT) module 200 on the target object smaller than the size of the first sampling position. In step S1, the target object is imaged using NBI or WLR to identify the region of interest A1 (the shaded area). This identification process can be performed by an operator or doctor, or by an algorithm. In a preferred embodiment, the region of interest (tumor or suspected tumor site) is automatically identified using a deep learning model (CNN) image segmentation algorithm. In step S2, the colocalization module 300 moves the first sampling position of the Raman spectroscopy analysis module 100 to substantially coincide with the region of interest A1, and performs Raman spectroscopy analysis on the region; Raman spectral information can reflect the structural information of specific molecules, such as lipids, proteins, etc., and the signal intensity is related to concentration; therefore, selecting specific molecules associated with cancerous lesions is very helpful for cancer screening and detection. Next, the OCT module 200 uses its scanning submodule 205 to perform scanning imaging within the first sampling position to obtain an image of the tissue structure spatially colocalized with the first sampling position. In step S3, the obtained spatial co-localization multimodal information is analyzed. Preferably, the analysis is performed using a Long Short-Term Memory (LSTM) algorithm fusion method. Thus, the multimodal imaging device of the present invention utilizes the co-localization module 300 to control the first sampling position of the Raman spectroscopy analysis module 100, making it substantially coincide with the region of interest, and then uses the optical coherence tomography (OCT) module 200 to perform scanning imaging within this region. This allows for the acquisition of spatially co-localized Raman spectral information and tissue structure image information. Due to the high spatial consistency, the present invention obtains multimodal information at a truly precise location. This helps improve the correlation between Raman spectral information and tissue structure image information, thereby improving the accuracy and efficiency of diagnosis / screening. It should be noted that the present invention is not limited to this specific embodiment. For example, although not shown, the co-localization module 300 of the present invention can also control the second sampling position of the OCT module 200, moving it to substantially coincide with the region of interest A1, and using the Raman spectroscopy analysis module 100 to analyze and detect the sampling position, thereby achieving spatial co-localization within the region of interest.

[0194] As for another operating mode, its co-location detection process can be found in [reference needed]. Figure 4bIn this method, the first lens group 105 in the Raman spectroscopy analysis module 100 uses a focusing lens, and the size of the first sampling position is smaller than the spot size of the optical coherence tomography (OCT) module 200 on the target object. In this method, the OCT module 200 is used to obtain an image of the tissue structure of the target object using imaging probe light, and to determine the region of interest of the target object. The co-localization module 300 is used to control the first sampling position of the excitation light in the Raman spectroscopy analysis module 100 according to the determined region of interest of the target object, thereby obtaining Raman spectral information at different positions within the region of interest. In step S1, a 3D-OCT image is obtained after spiral scanning through the piezoelectric ceramic tube of the scanning submodule 205. In step S2, the region of interest A1 (the shaded area) is identified. This identification process can be performed by an operator or doctor, or by an algorithm. In a preferred embodiment, the region of interest (tumor or suspected tumor site) is automatically identified using a deep learning model (CNN) image segmentation algorithm, and the Raman spectroscopy sampling position is located and navigated using the co-localization module. In step S3, the sampling position A1' (shaded area) of the Raman spectroscopy analysis module is controlled by the co-localization module 300 to obtain Raman spectral information in the region of interest A1. Raman spectral information can reflect the structural information of specific molecules, such as lipids and proteins, and the signal intensity is related to concentration. Therefore, selecting specific molecules associated with cancerous lesions is very helpful for cancer screening and detection. Optionally, the Long Short-Term Memory (LSTM) algorithm can be used to fuse multimodal information (OCT tissue structure images and co-localized Raman spectral data) to further improve the accuracy of cancer diagnosis. Thus, the co-localization module 300 of the present invention achieves highly efficient collaborative operation of Raman spectroscopy analysis and optical coherence tomography by controlling the sampling position of the Raman spectral excitation light in a specific region of interest, enabling high-accuracy and high-specificity cancer / tumor screening and diagnosis. As described above, the present invention is not limited to this. Alternatively, suitable lenses and operating methods can be used to obtain the region of interest A1 using the Raman spectroscopy analysis module 100, and the second sampling position of the optical coherence tomography (OCT) module 200 can be controlled to scan within the region of interest A1 to obtain spatially co-located Raman spectral information and tissue structure image information. The advantage of this method is that, based on the region of interest obtained from two-dimensional Raman spectral information, 3D-OCT imaging can be further used to obtain corresponding axial tissue structure image information. Those skilled in the art can select specific implementation methods based on different areas of interest, based on the above disclosure.

[0195] The multimodal imaging device according to the above embodiments of the present invention can be used, for example, in rigid endoscopy examination scenarios such as glioma, gastrointestinal tumors, head and neck surgical tumors, and thyroid tumors to achieve detection of the transverse plane cutting edge of tissue.

[0196] Figure 5 A schematic diagram of a multimodal imaging apparatus according to another embodiment of the present invention is shown.

[0197] Drawings of this invention Figure 5 Another embodiment of the multimodal imaging device involved is described in the accompanying drawings of this invention. Figure 2 The most important difference in the multimodal imaging device described in the embodiments is that the excitation light of the Raman spectroscopy module and the imaging probe light of the optical coherence tomography (OCT) module are coupled using a dichroic mirror before entering the probe, and a single lens group and the same fiber bundle are used together within the probe, without distinguishing between the first lens group for the Raman spectroscopy module and the second lens group for the OCT module. (This is from an embodiment of the invention.) Figure 5 Another implementation of the multimodal imaging device involved in the invention has the advantage of being able to further reduce the size of the probe, even to 2-5 mm, which is more conducive to integration into the working channel of existing endoscopy systems, reducing the damage that may be caused by endoscopic probes, and is more conducive to clinical use.

[0198] Specifically, see Figure 5 The first light source 101 of the Raman spectroscopy analysis module 100 uses a Raman excitation source with a wavelength of 785 nm. Along the excitation light emission path of the Raman spectrum, a first light source switch SW1, an intermediate reflector 110, a first beam splitter 102, a co-positioning module 300, a second dichroic mirror 106', a first coupling objective lens 103, a probe fiber 501, and a probe lens 502 are arranged. The probe lens 502 shown is a probe focusing lens. The Raman spectroscopy analysis module 100 also includes a grating 107 for splitting the reflected light from the first beam splitter 102, a receiving lens 108 for receiving the emitted light from the grating 107, and a spectrometer 109 for receiving the emitted light from the receiving lens 108. The probe fiber 501 is a multi-core fiber.

[0199] The detection focusing lens is shown in the accompanying drawings of this invention. Figure 2The first collecting lens 115 and the second focusing lens 217 in the embodiments described above have the same or similar functions. The selection of the detection focusing lens can also be based on performance and parameters such as spot size, high dispersion, and / or high numerical aperture. By using a second dichroic mirror 106' to couple the excitation light of the Raman spectroscopy analysis module 100 and the imaging probe light of the optical coherence tomography module 200 before entering the probe fiber, the coupled light can be focused and controlled using a single optical fiber and a single detection focusing lens. This configuration allows for further reduction in probe size, for example, to even 2-5 mm, making it more suitable for use in existing endoscopy systems. Due to the small inner diameter of the probe, it can be used in flexible endoscopy scenarios such as gastric, digestive tract, and bladder tumor detection. Preferably, in order to obtain spatially co-located spectral and image information more accurately, the detection focusing lens makes the spot size of the excitation light of the Raman spectroscopy analysis module 100 and the imaging probe light of the optical coherence tomography module 200 smaller than the size of the region of interest. This allows for control over scanning at both the first and second sampling locations within the area of ​​interest, resulting in valuable spatial co-location diagnostic / screening information.

[0200] Figure 5A schematic, enlarged view of the cross-section of the probe fiber 501 near the target object is provided. A central core group, consisting of at least one core from the central portion of the multi-core fiber, is used to transmit imaging probe light from the second light source 201 and light from the target object for obtaining an image of the target object's tissue structure. Multiple peripheral core groups, each consisting of at least one core from the central portion of the multi-core fiber, are used to transmit excitation light and scattered signal light of the Raman spectrum from the first light source 101, respectively. In this embodiment, the peripheral core groups for transmitting excitation light and scattered signal light of the Raman spectrum are arranged alternately, with one peripheral core group for transmitting excitation light of the Raman spectrum provided for every two peripheral core groups for transmitting scattered signal light of the Raman spectrum. The central fiber core assembly has a bandpass filter (BP-oct) at its end near the target object. The peripheral fiber core assembly, used to transmit the excitation light of the Raman spectrum, also has a bandpass (BP) filter (BP-rs) at its end near the target object. The peripheral fiber core assembly, used to transmit the scattered signal light of the Raman spectrum, has a notch filter (NF) at its end near the target object. In other words, the excitation light of the Raman spectrum excites the Raman spectrum of the target object through the bandpass (BP) filter, and the Raman scattered light signal is filtered by the notch filter (NF) and the long-pass filter (not shown) to remove background noise. This arrangement effectively improves the signal-to-noise ratio.

[0201] Regarding the drawings in this invention specification Figure 5 The arrangement of the central fiber core assembly for transmitting OCT imaging probe light in another embodiment of the multimodal imaging apparatus described herein is not limited to this. (See attached drawings of the invention specification.) Figure 5 In another embodiment of the multimodal imaging apparatus, the fiber core arrangement used for the OCT imaging probe light can also be as shown in the accompanying drawings. Figure 6 As shown, the OCT imaging probe light uses a non-centralized core assembly. However, it has been found that using a central core assembly to transmit the OCT imaging probe light results in a return of scattered light (…). Figure 5 The arrangement shown is relative to using other locations (e.g.) Figure 6 The arrangement of the fiber cores (in the center) results in better resolution. It has been found that if the fiber core group transmitting the OCT imaging probe light and the returned scattered light uses the central fiber core group for transmission with other fiber core groups, the Rayleigh range of the focused spot is reduced, thus resulting in better imaging resolution compared to using an "off-center" fiber core group for transmission.

[0202] Regarding the drawings in this invention specification Figure 5The arrangement of the peripheral fiber core assembly for transmitting Raman spectral excitation light and scattered signal light in another embodiment of the multimodal imaging device involved in the invention is not limited to this. Figure 5 The arrangement shown in the accompanying drawings of this invention. Figure 5 In another embodiment of the multimodal imaging apparatus, the arrangement of the fiber core assembly for transmitting the excitation light and the scattered signal light for transmitting the Raman spectrum can also be as shown in the accompanying drawings. Figure 6 As shown in the illustrations. However, the accompanying drawings of this invention... Figure 5 The arrangement in the core group is preferred; the alternating arrangement of cores for transmitting excitation light and scattered signal light for transmitting Raman spectra within the outer core group maintains relative symmetry, meaning that each core for transmitting excitation light for Raman spectra is surrounded by a core for transmitting scattered signal light for Raman spectra. Therefore, in conjunction with the accompanying drawings of this invention... Figure 2 The fiber core arrangement described in the embodiments involved in this invention is similar. Figure 5 This arrangement method can achieve better results than asymmetrical or less symmetrical arrangements (e.g.) Figure 6 Higher signal-to-noise ratio.

[0203] As an alternative implementation, in addition to multi-core optical fibers, the core of a single-mode optical fiber can be used to transmit OCT light sources and signals, and the cladding can be used to transmit Raman light sources and signals (not shown).

[0204] However, the preferred embodiment is shown in the accompanying drawings of this invention. Figure 5 The implementation methods involved, Figure 5 The arrangement method can achieve better signal-to-noise ratio and resolution.

[0205] The optical coherence tomography (OCT) module 200 includes a second light source 201, a second light source switch SW2, a beam splitter 202, an interferometer 203, a coupling fiber 204, a remote scanning submodule 205', and a detector 206. The second light source 201 uses a swept-frequency light source with a wavelength of 1325 nm. The detector 206 is a balanced photodetector equipped with a digital-to-analog converter and an FPGA acquisition card. The remote scanning submodule 205' is a MEMS-driven mirror. However, the invention is not limited to this; the remote scanning submodule 205' may also employ multiple Galvo mirrors or other components capable of scanning.

[0206] The scanning mode of the remote scanning mirror used in the remote scanning submodule 205' is consistent with the accompanying drawings of this invention. Figure 2 The multimodal imaging devices described in this invention differ from those in the embodiments described herein. (See attached drawings of this invention.) Figure 5The scanning of the multimodal imaging apparatus in another embodiment described herein is performed 'line by line' rather than in a spiral manner.

[0207] According to the accompanying drawings of the present invention Figure 5 In another embodiment of the multimodal imaging device described herein, the first light source 101 and the second light source 201 used are consistent with the drawings in this specification. Figure 2 The first light source 101 and the second light source 201 in the multimodal imaging apparatus of the embodiments described herein are the same.

[0208] Drawings of this invention Figure 5 The co-localization module 300 in another embodiment of the multimodal imaging apparatus described herein is consistent with the accompanying drawings. Figure 2 The same applies to the multimodal imaging apparatus described in the embodiments. The co-positioning module 300 includes a first flip mirror 301 and a second flip mirror 302 disposed between the first beam splitter 102 and the second dichroic mirror 106', and a first scanning galvanometer 303 and a second scanning galvanometer 304 paired with the two flip mirrors.

[0209] The probe 400 includes a housing 401 and a detection window 402.

[0210] A portion of the detection fiber 501 and the detection lens 502 are disposed inside the probe 400. This detection focusing lens is consistent with the accompanying drawings of this invention. Figure 2 The first and second focusing lenses in the embodiments described herein have the same or similar functions. However, by using a second dichroic mirror 106' to couple the excitation light of the Raman spectroscopy analysis module 100 and the imaging probe light of the optical coherence tomography module 200 before they enter the probe fiber, the coupled light can be focused and controlled using a single optical fiber (probe fiber 501) and a single probe focusing lens 502. This configuration allows for a further reduction in probe size, making it more suitable for use in existing endoscopy systems. Due to its small inner diameter, the probe can be used in flexible endoscopy procedures such as gastric, digestive tract, and bladder tumor detection.

[0211] The following describes the accompanying drawings of this invention. Figure 5 The operation mode of the multimodal imaging device in another embodiment involved in the invention.

[0212] Excitation light from the first light source 101, with the first light source switch SW1 open, sequentially passes through the intermediate reflecting mirror 110, the first beam splitter 102, the co-positioning module 300, and the second dichroic mirror 106', and is collimated by the first coupling objective lens 103 before entering the detection fiber 501. The outgoing light from the detection fiber 501 is focused by the detection lens 502; the detection fiber 501 is a multi-core fiber. The parameters of the detection lens 502 can be selected to control the spot size of the Raman spectroscopy detection light. In one embodiment of the invention, the spot diameter can be adjusted within the range of 5 micrometers to 1 millimeter. The outgoing light from the detection lens 502 detects the target object through the detection window 402. The Raman spectral scattered light from the target object returns along a substantially the same optical path as the excitation light, is reflected by the first beam splitter 102, and is incident on the grating 107 for dispersion. The outgoing light from the grating 107 passes through the receiving lens 108 and is detected by the spectrometer 109.

[0213] When the second light source switch SW2 is turned on, the imaging probe light from the second light source 201, after passing through the beam splitter 202, is reflected by the remote scanning submodule 205' and then incident on the second dichroic mirror 106'. Thus, the Raman spectral excitation light from the first light source 101 and the imaging probe light from the second light source 201 are coupled at the second dichroic mirror 106'. The scanning mode of the MEMS-driven mirror used by the remote scanning submodule 205' is consistent with the accompanying drawings of this invention. Figure 2 The multimodal imaging apparatus described in this embodiment differs from that in the MEMS-driven mirror (or Galvo mirror). Scanning with the MEMS-driven mirror (or Galvo mirror) involves rotating it about at least one axis to change the angle and / or position of the imaging probe light incident on the first coupling objective 103. This scanning method allows the scan to be performed 'line-by-line' rather than in a spiral manner. The image formation method of this scanning method can be found in [reference needed]. Figure 7b The images are obtained "line by line" (i.e., row by row).

[0214] In this embodiment of the invention, the detection of the two modes can be switched in the time domain by externally triggering the light source switch of the two modes, thereby reducing the mutual influence between the modes and improving the signal-to-noise ratio. This is highly advantageous in maintaining acceptable signal-to-noise ratios for both modes while ensuring a reduction in probe size. Therefore, it is preferable to alternately detect the two modes by externally triggering the light source switch of the two modes to obtain the improved signal-to-noise ratio.

[0215] The light coupled by the first coupling lens 103 is collimated by the first coupling lens 103 and then incident into the probe fiber 501. The outgoing light from the probe fiber 501 is incident into the probe lens 502 and passes through the probe window 402 to image / detect the target object. The scattered light from the object sample returns along a light path that is basically the same as the incident light, and is detected by the detector 206 after passing through the beam splitter 202 and the interferometer 203.

[0216] In this embodiment, the co-positioning module 300 can switch between a first mode and a second mode by adjusting the angle of the optical path between the first flip mirror 301 and the second flip mirror 302 relative to the first beam splitter 102 and the second dichroic mirror 106'. Figure 5 The specific internal structure of the co-positioning module 300 is not shown because it is related to... Figure 2 The same as in [the previous sentence].

[0217] In the first mode, when the mirror surfaces of the first flip mirror 301 and the second flip mirror 302 are parallel to the optical path between the first beam splitter 102 and the second dichroic mirror 106', the presence of the co-positioning module 300 does not affect the incident direction of the optical path between the first beam splitter 102 and the second dichroic mirror 106', thus not affecting the sampling position of the Raman spectral excitation light on the target object.

[0218] In the second mode, when the mirror surfaces of the first flip mirror 301 and the second flip mirror 302 are not parallel to the optical path between the first beam splitter 102 and the second dichroic mirror 106', for example... Figure 2 When the angles shown are set, the first flip mirror 301 is used to receive and reflect the light transmitted through the first beam splitter 102, the first scanning mirror 303 is used to receive and reflect the reflected light from the first flip mirror 301, the second scanning mirror 304 is used to receive and reflect the reflected light from the first scanning mirror 303, the second flip mirror 302 is used to receive and reflect the reflected light from the second scanning mirror 304, the second dichroic mirror 106' is used to receive the reflected light from the second flip mirror, and the first coupling objective 103 is used to receive the emitted light from the second dichroic mirror 106'. The co-positioning module 300 of the second mode will affect the excitation light of the Raman spectrum. For example, rotating the first scanning mirror 303 and / or the second scanning mirror 304 around a preset axis will cause the emitted light from the first beam splitter 102 to deviate from the direction of the original optical path (i.e., the direction of the optical path in the first mode) at a certain angle, which will cause the position of the incident light of the second dichroic mirror 106' to also change. This results in a change in the sampling position of the excitation light in the Raman spectrum on the target object. In this embodiment, the first scanning mirror 303 and the second scanning mirror 304 can rotate along mutually orthogonal axes.

[0219] However, the present invention is not limited thereto. Those skilled in the art can, based on the above disclosure, set the spatial orientation of the axes of the first scanning mirror 303 and the second scanning mirror 304 according to actual conditions or needs. An axis with a certain orientation in a given coordinate system can be selected, thereby influencing the position / angle / shape of the excitation light in the Raman spectrum in different ways by rotating the first scanning mirror 303 and the second scanning mirror 304. In this embodiment, the first scanning mirror 303 and the second scanning mirror 304 use MEMS-driven mirrors. However, the present invention is not limited thereto. Those skilled in the art can, based on the above disclosure, use other optical elements with equivalent functions, such as Galvo mirrors, according to actual conditions or needs.

[0220] In the accompanying drawings of this invention specification Figure 5 The multimodal imaging apparatus of another embodiment involved has a variety of operating modes, one of which is only listed exemplarily below.

[0221] As an operational mode, its co-location detection process can be found in [reference needed]. Figure 7aIn this approach, the probe lens 502 ensures that the spot size of the optical coherence tomography module 200 and the Raman spectroscopy analysis module 100 on the target object is smaller than the size of the region of interest. In step S1, the target object is imaged using NBI or WLR to identify the region of interest A1 (the shaded area). This identification process can be performed by an operator or physician, or by an algorithm. In a preferred embodiment, the region of interest (tumor or suspected tumor site) is automatically identified using a deep learning model (CNN) image segmentation algorithm. In step S2, the remote scanning submodule 205' controls the second sampling position of the optical coherence tomography (OCT) module 100 to scan and detect within the region of interest A1 to obtain tissue structure / morphological images. Simultaneously and / or concurrently, the colocalization module 300 controls the first sampling position of the Raman spectroscopy analysis module 200 to scan and detect within the region of interest A1 to obtain Raman spectral information. This yields spatially colocalized Raman spectral information and tissue structure image information. Raman spectral information can reflect the structural information of specific molecules, such as lipids and proteins, and the signal intensity is concentration-dependent. Therefore, selecting specific molecules associated with cancerous lesions is highly helpful for cancer screening and detection. In step S3, the obtained spatially colocalized multimodal information is analyzed. Preferably, the analysis is performed using a Long Short-Term Memory (LSTM) algorithm fusion method. Therefore, the multimodal imaging device of this invention utilizes the co-localization module 300 to control the first sampling position of the Raman spectroscopy analysis module 100, enabling it to perform scanning analysis / imaging within the region of interest along with the second sampling position of the optical coherence tomography (OCT) module 100. This allows for the acquisition of spatially co-located Raman spectral information and tissue structure image information. Due to the high spatial consistency, this invention obtains truly simultaneous multimodal information. This helps improve the correlation between Raman spectral information and tissue structure image information, thereby improving the accuracy and efficiency of diagnosis / screening.

[0222] As for another operating mode, its co-location detection process can be found in [reference needed]. Figure 7bIn step S1, a 3D-OCT image is obtained by scanning line by line using a MEMS-driven mirror in the remote scanning submodule 205'. In step S2, the region of interest A1 (shaded area) is identified. This identification process can be performed by an operator or doctor, or by an algorithm. In step S3, the sampling position A1' (shaded area) of the Raman spectroscopy analysis module is controlled by the co-localization module 300 to obtain Raman spectral information in the region of interest A1. Raman spectral information can reflect the structural information of specific molecules, such as lipids and proteins, and the signal intensity is related to concentration. Therefore, selecting specific molecules associated with cancerous lesions is very helpful for cancer screening and detection. Optionally, the Long Short-Term Memory (LSTM) algorithm can be used to fuse multimodal information (OCT tissue structure image and co-localized Raman spectral data) to further improve the accuracy of cancer diagnosis. Therefore, the co-localization module 300 of the present invention achieves highly efficient collaborative operation of Raman spectroscopy analysis and optical coherence tomography (OCT) imaging by controlling the sampling position of the Raman spectroscopy excitation light in a specific region of interest, enabling high-accuracy and high-specificity cancer / tumor screening and diagnosis. As mentioned above, the present invention is not limited to this; the Raman spectroscopy analysis module 100 can also be used to obtain the region of interest A1, and the second sampling position of the OCT imaging module 200 can be controlled to detect the region of interest A1, obtaining spatially co-localized Raman spectral information and tissue structure image information. The advantage of this approach is that the region of interest obtained based on two-dimensional Raman spectral information can be further used to obtain corresponding axial tissue structure image information using 3D-OCT imaging. Those skilled in the art can select specific implementation methods based on the above disclosure and different areas of interest.

[0223] In the multimodal imaging device of the present invention, exemplary technical parameters of frequency-sweep OCT and Raman are as follows:

[0224] OCT technical parameters include: imaging speed of 2-5 volume images / second or 200-500 B-scan frames / second; imaging field of view of 0.5mm-2mm; and resolution and imaging depth depending on the center wavelength of the light source. With a center wavelength of 1325nm, the resolution is 15-20μm and the depth is 1-2mm; with a center wavelength of 800nm, the resolution is 4-10μm and the depth is 0.5-1mm.

[0225] The technical parameters of the Raman system include: spectral acquisition speed of 2-5 Hz, spectral resolution of 5-10 wavenumbers, spectrometer receiving wavelength range of 800-1100 nm, and Raman spectral detection range of 800-1800 wavenumbers and 2800-3600 wavenumbers.

[0226] Based on the above combined with appendix Figure 1As described in section 7 and the detailed description of specific embodiments of the present invention, the co-localization module of the multimodal imaging device of the present invention is used to control the first sampling position of the excitation light in the Raman spectroscopy analysis module and / or the second sampling position in the optical coherence tomography (OCT) module according to the determined region of interest of the target object, so that the first sampling position and the second sampling position achieve spatial co-localization within the region of interest. That is, the multimodal imaging device of the present invention utilizes the co-localization module to control the first sampling position of the Raman spectroscopy analysis module so that it is coordinated or matched with the scanning imaging of the OCT module in the spatial detection region (e.g., the first sampling position substantially coincides with the region of interest, and then the scanning imaging of the OCT module is performed within the region of interest, or the first sampling position is controlled to be synchronized with and / or simultaneously detected within the region of interest). This allows for the acquisition of spatially co-localized Raman spectral information and tissue structure image information. Due to the high spatial consistency, the present invention obtains multimodal information at the same precise location. This helps to improve the correlation between Raman spectral information and tissue structure image information, thereby improving the accuracy and efficiency of diagnosis / screening.

[0227] Furthermore, in one operating mode, the optical coherence tomography module in the multimodal imaging device of the present invention is used to acquire tissue structure images of the target object and determine the region of interest of the target object, while the co-localization module is used to control the (first) sampling position of the excitation light in the Raman spectroscopy analysis module according to the determined region of interest, thereby obtaining Raman spectral information at different positions in the region of interest. Clearly, this implementation also obtains spatially co-localized Raman spectral information and tissue structure image information. Based on obtaining beneficial spatial co-localization information, this implementation achieves the coordinated operation of the two modules, resulting in allowing the operator to first preliminarily identify a region of interest suspected of having cancer risk based on the tissue structure image, and then obtain the Raman spectral information of the region of interest by guiding and controlling the sampling position of the Raman spectrum through the co-localization module. This utilizes the high accuracy and high specificity of Raman spectroscopy to obtain more accurate information as a diagnostic basis. The region of interest can be determined according to an algorithm or based on the operator's experience. In a preferred embodiment, the algorithm quickly and automatically identifies the region of interest, and then the excitation light of the Raman spectrum is guided to the sampling position through a co-localization module, thereby obtaining relatively accurate spectral information of cancer / tumor-related specific molecules (lipids, proteins, etc.) as a diagnostic basis. In this embodiment, the process from identifying the region of interest to guiding the excitation light of the Raman spectrum through the co-localization module can be automated, which allows for real-time and accurate in vivo cancer detection. In the above process, the co-localization module can reduce the area that needs to be detected by Raman spectroscopy analysis. That is, it is not necessary to perform Raman spectroscopy analysis on the entire region, but only on the region of interest. This largely avoids the disadvantage of slow detection speed of Raman spectroscopy analysis, while still utilizing the advantages of high accuracy and high specificity of Raman spectroscopy analysis, thereby improving the overall detection efficiency.

[0228] In addition, the co-location module of the present invention can switch between two modes, so that the co-location function can be turned off and on as needed.

[0229] Furthermore, the probe configuration of the multimodal imaging device of the present invention allows for a reduction in probe size, for example, to 2-10 mm, or even 2-5 mm. This facilitates integration into the working channel of existing endoscopy systems and is beneficial for clinical use.

[0230] Finally, taking cervical cancer, mentioned in the background section, as an example, when using the multimodal imaging device of the present invention for diagnosis as an endoscope, it is anticipated that highly efficient, non-invasive, and real-time results can be obtained: the sensitivity and specificity for diagnosing cervical intraepithelial neoplasia (CIN) exceeds 98%, and the diagnostic accuracy for early CIN1 precancerous lesions exceeds 90%.

[0231] Furthermore, the control method of the operating device according to embodiments of the present invention can be recorded in a computer-readable recording medium. Specifically, according to the present invention, a computer-readable recording medium storing computer-executable instructions can be provided, which, when executed by a processor, cause the processor to perform the control method as described above. Examples of computer-readable recording media may include magnetic media (e.g., hard disks, floppy disks, and magnetic tapes); optical media (e.g., CD-ROMs and DVDs); magneto-optical media (e.g., optical discs); and hardware devices specially configured for storing and executing program instructions (e.g., read-only memory (ROM), random access memory (RAM), flash memory, etc.). Furthermore, according to the present invention, a device including a processor and a memory storing computer-executable instructions can also be provided, wherein, when executed by a processor, the computer-executable instructions cause the processor to perform the control method as described above. Examples of computer-executable instructions include, for example, machine code generated by a compiler and files containing high-level code that can be executed by a computer using an interpreter.

[0232] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code containing at least one executable instruction for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0233] Generally, the various exemplary embodiments or implementations disclosed may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of the invention are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein may be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0234] The exemplary embodiments of the present invention described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of the invention, and such modifications should fall within the scope of the invention.

Claims

1. A multimodal imaging device, comprising: The Raman spectroscopy analysis module is used to obtain the Raman spectral information of the target object at the first sampling position using excitation light, and includes, in sequence along the light transmission direction, a first light source, a first beam splitter, a first coupling objective, a first optical fiber, a spectrometer, a first lens group, and a first dichroic mirror; An optical coherence tomography module is used to acquire an image of the tissue structure of a target object at a second sampling location using an imaging probe beam; and The co-localization module is used to control the first sampling position of the excitation light in the Raman spectroscopy analysis module or the first sampling position of the excitation light in the Raman spectroscopy analysis module and the second sampling position in the optical coherence tomography module, based on the determined region of interest of the target object, so that the first sampling position and the second sampling position achieve spatial co-localization within the region of interest. The co-positioning module includes a first flip mirror, a second flip mirror, a first scanning galvanometer, and a second scanning galvanometer; The first and second flip mirrors are disposed between the first beam splitter and the first coupling objective. The first and second flip mirrors are used to control whether the mirror surfaces of the first and second flip mirrors are parallel or non-parallel to the optical path between the first beam splitter and the first coupling objective by rotating about an axis orthogonal to the optical path between the first beam splitter and the first coupling objective. The first and second scanning mirrors are used to control the first sampling position by rotating about different axes.

2. The multimodal imaging device as claimed in claim 1, wherein the multimodal imaging device includes a probe having a housing and a detection window, and is used to detect a target object, wherein excitation light from a Raman spectroscopy analysis module and imaging detection light from an optical coherence tomography module are coupled in the probe.

3. The multimodal imaging device as described in claim 2, wherein the first beam splitter is used to transmit excitation light from the first light source and reflect scattered signal light from the Raman spectrum of the target object, the spectrometer is used to receive the scattered signal light from the Raman spectrum of the target object reflected by the first beam splitter, the first coupling objective is used to receive outgoing light from the first beam splitter or from the co-positioning module, the first optical fiber is used to receive outgoing light from the first coupling objective, the first lens group is used to receive outgoing light from the first optical fiber, and the first dichroic mirror is used to receive and transmit outgoing light from the first lens group.

4. The multimodal imaging device as claimed in claim 3, wherein a grating and a receiving lens are disposed between the spectrometer and the first beam splitter, the grating being used to split the reflected light from the first beam splitter, the receiving lens being used to receive the emitted light from the grating, and the spectrometer being used to receive the emitted light from the receiving lens.

5. The multimodal imaging apparatus of claim 3, wherein the first lens group includes a first collecting lens.

6. The multimodal imaging apparatus of claim 3, wherein the first optical fiber comprises a multi-core optical fiber, wherein a central core group consisting of at least one core of the central portion of the multi-core optical fiber is used to transmit excitation light of the Raman spectrum from the first light source, and a plurality of peripheral core groups consisting of groups of at least one core of the multi-core optical fiber surrounding the central portion are used to transmit scattered signal light of the Raman spectrum.

7. The multimodal imaging device of claim 6, wherein the peripheral fiber core groups are symmetrically distributed with respect to the central fiber core group.

8. The multimodal imaging device as claimed in claim 7, wherein the central fiber core assembly is provided with a bandpass filter at its end near the target object, and the peripheral fiber core assembly is provided with a notch filter and / or a long-pass filter at its end near the target object.

9. The multimodal imaging device of claim 8, wherein the cross-sections of the central fiber core assembly and the peripheral fiber core assembly are substantially circular.

10. The multimodal imaging device as described in claim 3, wherein the optical coherence tomography module includes a second light source, a beam splitter, an interferometer, a second optical fiber, a scanning submodule, a detector, a second lens group, and a reflector; The second light source, interferometer, detector, and scanning submodule are optically coupled to the beam splitter; The scanning submodule is optically coupled to the beam splitter via a second fiber optic cable; A portion of the second optical fiber passes through the scanning submodule; The second lens group is used to receive the outgoing light from the second optical fiber, the reflector is used to reflect the outgoing light from the second lens group, and the first dichroic mirror is used to reflect the reflected light from the reflector, thereby coupling the light from the first lens group with the light from the second lens group.

11. The multimodal imaging apparatus of claim 10, wherein the second light source and / or interferometer and / or detector is optically coupled to the beam splitter via a coupling optical fiber.

12. The multimodal imaging device as claimed in claim 10, wherein the first lens group and the second lens group are arranged in parallel inside the probe, and the first dichroic mirror, the scanning submodule, the reflector and at least a portion of the second optical fiber are located inside the probe.

13. The multimodal imaging apparatus of claim 11, wherein the scanning submodule is used to control the position of the imaging probe light of the second light source to obtain the position of the tissue structure image of the target object by controlling the position of the second optical fiber.

14. The multimodal imaging apparatus of claim 13, wherein the scanning submodule comprises a piezoelectric ceramic tube.

15. The multimodal imaging apparatus of claim 10, wherein the second lens group comprises a second focusing lens and a diffraction lens, the diffraction lens being located between the second focusing lens and the reflecting mirror.

16. The multimodal imaging apparatus of claim 10, wherein a circulator is disposed between the detector and the beam splitter.

17. The multimodal imaging apparatus of claim 11, wherein the second optical fiber and / or the coupling optical fiber comprises a single-mode optical fiber.

18. The multimodal imaging device of claim 10, wherein the colocalization module is located in the excitation light incident optical path of the Raman spectroscopy analysis module.

19. The multimodal imaging apparatus of claim 18, wherein the co-localization module is located between the first beam splitter and the first coupling objective.

20. The multimodal imaging apparatus of claim 19, wherein the co-localization module has a switchable first mode and a second mode; In the first mode, the co-localization module does not change the first sampling position; In the second mode, the co-location module is used to control the first sampling position.

21. The multimodal imaging apparatus of claim 20, wherein in the first mode, the mirror surfaces of the first and second flip mirrors are parallel to the optical path between the first beam splitter and the first coupling objective; and In the second mode, the mirror surfaces of the first and second flip mirrors are not parallel to the optical path between the first beam splitter and the first coupling objective.

22. The multimodal imaging apparatus of claim 21, wherein in the second mode, the first flip mirror is used to receive and reflect light transmitted via the first beam splitter, the first scanning mirror is used to receive and reflect reflected light from the first flip mirror, the second scanning mirror is used to receive and reflect reflected light from the first scanning mirror, the second flip mirror is used to receive and reflect reflected light from the second scanning mirror, and the first coupling objective is used to receive reflected light from the second flip mirror.

23. The multimodal imaging device of claim 21, wherein the first scanning mirror and the second scanning mirror include a Galvo mirror, a MEMS-driven reflector, or a resonant mirror.

24. The multimodal imaging apparatus of claim 1, wherein the co-localization module is used to move the first sampling position to substantially coincide with the position of the region of interest.

25. The multimodal imaging apparatus of claim 3, wherein the first lens group is configured to ensure that the spot size of the excitation light from the Raman spectroscopy analysis module at the first sampling position is substantially the same as the size of the region of interest.

26. The multimodal imaging apparatus according to any one of claims 1-25, further comprising: The image processing module is used to fuse Raman spectral information from the first sampling location and tissue structure images from the second sampling location to generate fused multimodal information of the region of interest.

27. The multimodal imaging apparatus according to any one of claims 1-25, wherein the multimodal imaging apparatus is an endoscope.

28. The multimodal imaging device according to any one of claims 2-25, wherein the diameter of the probe is 2-10 mm.

29. The multimodal imaging device according to any one of claims 2-25, wherein the diameter of the probe is 2-5 mm.

30. A multimodal imaging device, comprising: The Raman spectroscopy analysis module is used to obtain Raman spectral information of the target object at the first sampling position using excitation light; An optical coherence tomography module is used to obtain an image of the tissue structure of a target object at a second sampling location using an imaging probe light; The co-location module is used to control the first sampling position of the excitation light in the Raman spectroscopy analysis module or the first sampling position of the excitation light in the Raman spectroscopy analysis module and the second sampling position in the optical coherence tomography module, based on the determined region of interest of the target object, so that the first sampling position and the second sampling position achieve spatial co-location within the region of interest. as well as Detection lens and detection fiber; The Raman spectroscopy analysis module includes: a first light source, a first beam splitter, a first coupling objective, a spectrometer, and a second dichroic mirror; The optical coherence tomography module includes: a second light source, a beam splitter, an interferometer, a remote scanning submodule, and a detector; The co-positioning module includes a first flip mirror, a second flip mirror, a first scanning galvanometer, and a second scanning galvanometer; The first and second flip mirrors are used to control whether the mirror surfaces of the first and second flip mirrors are parallel or non-parallel to the optical path between the first beam splitter and the second dichroic mirror by rotating about an axis orthogonal to the optical path between the first beam splitter and the second dichroic mirror; the first and second scanning galvanometers are used to control the first sampling position by rotating about different axes. The first beam splitter, the co-positioning module, the second dichroic mirror, and the first coupling objective are arranged sequentially along the transmission direction of the emitted light from the first light source. The first beam splitter is used to transmit excitation light from the first light source and reflect scattered signal light from the Raman spectrum of the target object. The spectrometer is used to receive the scattered signal light from the Raman spectrum of the target object reflected by the first beam splitter. The second dichroic mirror is used to transmit the outgoing light from the co-positioning module or the outgoing light from the first beam splitter and reflect the imaging probe light from the second light source, thereby coupling the two. The first coupling objective is used to receive the coupled light from the second dichroic mirror; The probe fiber is used to receive the outgoing light from the first coupling objective; The probe lens is used to receive the outgoing light from the probe fiber; The remote scanning submodule is positioned between the second dichroic mirror and the beam splitter and is used to receive and reflect imaging probe light transmitted from the second light source through the beam splitter; The second light source, interferometer, and detector are optically coupled to the beam splitter.

31. The multimodal imaging apparatus of claim 30, wherein the second light source and / or interferometer and / or detector is optically coupled to the beam splitter via a coupled fiber optic cable.

32. The multimodal imaging device of claim 30, wherein the multimodal imaging device includes a probe, and a detection lens and at least a portion of the detection optical fiber are disposed inside the probe.

33. The multimodal imaging apparatus of claim 32, wherein the co-localization module has a switchable first mode and a second mode; In the first mode, the co-localization module does not change the first sampling position; In the second mode, the co-location module is used to control the first sampling position.

34. The multimodal imaging apparatus of claim 33, wherein... In the first mode, the mirror surfaces of the first and second flip mirrors are parallel to the optical path between the first beam splitter and the second dichroic mirror; and In the second mode, the mirror surfaces of the first and second flip mirrors are not parallel to the light path between the first beam splitter and the second dichroic mirror.

35. The multimodal imaging apparatus of claim 34, wherein in the second mode, the first flip mirror is used to receive and reflect light transmitted via the first beam splitter, the first scanning mirror is used to receive and reflect reflected light from the first flip mirror, the second scanning mirror is used to receive and reflect reflected light from the first scanning mirror, the second flip mirror is used to receive and reflect reflected light from the second scanning mirror, and the second dichroic mirror is used to receive and transmit reflected light from the second flip mirror.

36. The multimodal imaging device of claim 34, wherein the first scanning mirror and the second scanning mirror include a Galvo mirror, a MEMS-driven reflector, or a resonant mirror.

37. The multimodal imaging apparatus of claim 30, wherein the remote scanning submodule is used to control the imaging probe light of the second light source to obtain the position of the tissue structure image of the target object by rotating about at least two axes.

38. The multimodal imaging apparatus of claim 30, wherein the detection optical fiber comprises a multi-core optical fiber. A central core assembly, consisting of at least one core from the central portion of a multi-core optical fiber, is used to transmit imaging probe light from a second light source and light from the target object for obtaining an image of the target object's tissue structure. Multiple peripheral core groups, each consisting of at least one core of a multi-core optical fiber surrounding the central portion, are used to transmit excitation light from the Raman spectrum of a first light source and scattered signal light from the Raman spectrum, respectively.

39. The multimodal imaging apparatus of claim 38, wherein the peripheral fiber core group for transmitting the excitation light of the Raman spectrum and the peripheral fiber core group for transmitting the scattered signal light of the Raman spectrum are arranged alternately.

40. The multimodal imaging apparatus of claim 39, wherein the central fiber core assembly and the peripheral fiber core assembly for transmitting the excitation light of the Raman spectrum are provided with bandpass filters at their ends near the target object, and the peripheral fiber core assembly for transmitting the scattered signal light of the Raman spectrum is provided with notch filters and / or long-pass filters at its ends near the target object.

41. The multimodal imaging apparatus of claim 40, wherein the cross-sections of the central fiber core assembly and the peripheral fiber core assembly are substantially circular.

42. The multimodal imaging apparatus of claim 41, wherein the cross-sectional area of ​​the central core group is greater than the cross-sectional area of ​​the single peripheral core group for transmitting the excitation light of the Raman spectrum and the cross-sectional area of ​​the single peripheral core group for transmitting the scattered signal light of the Raman spectrum.

43. The multimodal imaging apparatus of claim 37, wherein the remote scanning submodule includes a remote scanning galvanometer.

44. The multimodal imaging device of claim 43, wherein the remote scanning mirror comprises a MEMS-driven mirror, a Galvo mirror, or a resonant mirror.

45. The multimodal imaging apparatus of claim 30, wherein the detection lens includes a detection focusing lens.

46. ​​The multimodal imaging apparatus of claim 30, wherein a grating and a receiving lens are disposed between the spectrometer and the first beam splitter, the grating being used to split the reflected light from the first beam splitter, the receiving lens being used to receive the emitted light from the grating, and the spectrometer being used to receive the emitted light from the receiving lens.

47. The multimodal imaging apparatus as described in claim 30, wherein the first light source and the second light source are respectively provided with a first light source switch and a second light source switch, and an optional intermediate reflector is provided between the first light source and the first beam splitter for reflecting the excitation light from the first light source to the first beam splitter.

48. The multimodal imaging apparatus of claim 47, wherein the co-localization module is used to move the first sampling position to a position substantially overlapping with the region of interest.

49. The multimodal imaging apparatus of claim 30, wherein the co-localization module is used to synchronously control the first sampling position and the second sampling position with the remote scanning submodule so that the first sampling position and the second sampling position substantially overlap.

50. The multimodal imaging apparatus according to any one of claims 30-49, further comprising: The image processing module is used to fuse Raman spectral information from the first sampling location and tissue structure images from the second sampling location to generate fused multimodal information of the region of interest.

51. The multimodal imaging apparatus according to any one of claims 30-49, wherein the multimodal imaging apparatus is an endoscope.

52. The multimodal imaging device according to any one of claims 30-49, wherein the diameter of the probe is 2-10 mm.

53. The multimodal imaging device according to any one of claims 30-49, wherein the diameter of the probe is 2-5 mm.

54. The multimodal imaging apparatus of any one of claims 30-49, wherein the region of interest of the target object is determined from an image of the target object acquired by an imaging device different from the multimodal imaging apparatus, from spectral information acquired by a Raman spectroscopy analysis module, or from an image acquired by an optical coherence tomography module.

55. The multimodal imaging apparatus of claim 54, wherein the imaging apparatus, different from the multimodal imaging apparatus, includes a white light endoscope module and / or a narrowband imaging module.

56. The multimodal imaging apparatus of claim 54, wherein the region of interest is a medical region of interest.

Images