Endoscope device integrating optical, photoacoustic, OCT, and fluorescence multimodal forward imaging and imaging method thereof

Abstract

The present invention discloses an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multimodal forward imaging and an imaging method thereof. The device includes a control system, a light source system, an optical path system, an endoscope probe, and a data acquisition and reconstruction system; the control system synchronously triggers the light source system and the data acquisition system, the excitation light enters the endoscope probe from the optical path system, the dichroic mirror inside the probe realizes the photoacoustic / fluorescence and OCT excitation light beam combination, the beam combination light irradiates the surface of the tissue to be imaged via a rod lens, the echo signal is processed by the data acquisition system and the photoacoustic image, OCT image, and fluorescence image reconstruction are completed, and the optical signal is collected by the CCD at the top of the endoscope for optical image reconstruction. The photoacoustic, fluorescence, and OCT excitation lights of the present invention share an optical path, chromatic aberration is eliminated by an adjustable focus collimator, and a rod lens structure is used to penetrate into the cavity organ for in vivo detection, thereby realizing photoacoustic imaging, fluorescence imaging, and OCT imaging in real time with high resolution confocal endoscope imaging.

Description

Technical Field

[0001] The present invention belongs to the technical field of endoscopic nondestructive testing, and in particular relates to an endoscopic device integrating optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging and an imaging method thereof. Background Art

[0002] Most cases are already in the middle and late stages when diagnosed, and the survival rate is low. Therefore, early diagnosis and treatment of cancer has become an urgent problem to be solved. Taking nasopharyngeal carcinoma as an example, nasopharyngeal carcinoma is a common malignant tumor of the head and neck. It is prone to occur on the top and side walls of the nasopharynx. It is hidden in the location and grows invasively in the early stage. The clinical manifestations are not obvious, and the diagnosis mainly relies on imaging performance. Radiotherapy is its first choice of treatment, but it is prone to recurrence. Therefore, early diagnosis, early treatment and follow-up after radiotherapy of nasopharyngeal carcinoma are particularly important. CT and MRI are conventional means for the diagnosis, staging and post-radiotherapy review of nasopharyngeal carcinoma. CT cannot distinguish between local inflammation and tumors, resulting in a CT detection rate of only 57.1% for nasopharyngeal carcinoma. The disadvantages of MRI imaging are long imaging time, large artifacts, and insensitivity to calcification foci and cortical bone imaging.

[0003] White light endoscope has always been the standard tool for diagnosing nasopharyngeal carcinoma. By irradiating white light to the tissue area to be imaged and receiving the white light image through CCD or CMOS, the morphological information of the imaging area can be obtained. However, since the basic principle is to obtain the detection image using the reflected signal, it is greatly affected by the scattering of the medium and can only image the shallow surface of the imaging area.

[0004] As an emerging non-invasive biomedical imaging method, photoacoustic imaging technology has broken through the limitations of traditional optical imaging and ultrasound imaging in their respective biological tissue imaging fields, and has great potential in biomedical imaging and clinical applications. Photoacoustic imaging technology is based on the photoacoustic effect. Under the irradiation of a short-pulse laser beam, the biological tissue to be detected absorbs photon energy and expands due to heat, generating an ultrasonic signal. The transducer receives the ultrasonic signal, and after algorithm reconstruction, the ultrasonic signal is used to obtain a distribution image reflecting the difference in light absorption inside the tissue. Therefore, photoacoustic imaging technology has the characteristics of high optical imaging resolution and deep ultrasound imaging depth, and can perform functional and structural imaging of tissues. Photoacoustic imaging can obtain images of blood vessel distribution near tumor tissues, which are used to determine the depth of tumor tissue infiltration.

[0005] OCT imaging technology is a non-invasive high-resolution medical imaging technology that can achieve micron-level resolution at millimeter-level depths compared to traditional imaging technology. OCT imaging technology is based on the interferometer principle. When near-infrared weak coherent light irradiates the tissue to be imaged, the backscattered light of the reference arm interferes with the reflected light of the sample arm. The interference signal is received by the spectrometer to obtain the superficial structure information of the tissue.

[0006] Fluorescence imaging technology is a technique that injects exogenous fluorescent substances into blood vessels, excites the fluorescent substances with lasers, and simultaneously receives the fluorescence with a receiving device. The tissue is imaged through the received fluorescence information to obtain the distribution information of different tissues in the region. Since fluorescent molecular targeted imaging is highly sensitive and specific, it can provide an important reference for delineating biological target areas.

[0007] Currently, the commonly used endoscope is a white light endoscope, which can only image the surface of tissues, lacks tissue depth information, cannot determine the depth of tumor infiltration, and cannot demarcate tumor boundaries, which has obvious drawbacks in preoperative diagnosis and postoperative review. An endoscope device that integrates optical, photoacoustic, OCT, and fluorescent multimodal forward imaging integrates four imaging modes: optical, photoacoustic, OCT, and fluorescence inside the integrated endoscope probe. Among them, photoacoustic imaging technology combines the characteristics of high resolution of optical imaging and deep depth of acoustic imaging, which can image tumor microvascular networks and determine the depth of tumor infiltration; OCT endoscopic imaging technology combined with endoscopic technology can perform in vivo high-resolution imaging of biological tissue cavities to obtain tissue structure information; fluorescent molecular targeted imaging has high sensitivity and specificity, can mark tumor boundaries, and provide important reference for delineating biological target areas during radiotherapy.

[0008] The application number is 201410412368.9, and the patent name is "A Multimodal Microscopic Imaging System". It discloses a multimodal microscopic imaging system that uses optical detection methods, combining ultrasound, OCT, and photoacoustic imaging modes to image the internal structure and function of tissues. Since there is no fluorescence imaging in the imaging mode, the morphological information of the tissue surface cannot be obtained; at the same time, since there is no achromatic device in the device, confocal imaging cannot be performed, and the scanning structure is a lateral imaging method, and the tissue to be tested cannot be scanned forward.

[0009] The application number is 201710112585.X, and the patent name is "An Intravascular Fluorescence-Photoacoustic-Ultrasound Multi-Mode Imaging Device", which discloses an intravascular fluorescence-photoacoustic-ultrasound multi-mode imaging device and method, integrating three modal endovascular imaging systems: fluorescence imaging, photoacoustic imaging, and ultrasound imaging. However, there is no OCT in the imaging mode, and high-resolution structural imaging of the tissue to be detected cannot be performed. At the same time, the scanning structure is a lateral imaging method, and forward scanning imaging of the tissue to be detected cannot be performed.

[0010] The application number is 202010263211.X, and the patent name is "Multimodal Microendoscopic Imaging Device and Method", which discloses a multimodal microendoscopic imaging device that combines three imaging methods: confocal, photoacoustic, and ultrasonic, and has two modes: forward scanning and side scanning. However, there is no OCT in the imaging mode, and high-resolution structural imaging of the tissue to be detected cannot be performed. At the same time, since there is no achromatic device in the device, confocal imaging cannot be performed when the wavelengths of the excitation light are far apart. Summary of the invention

[0011] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide an endoscope device and an imaging method thereof that integrates optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging. The device can realize imaging in four modes: optical, photoacoustic, OCT, and fluorescence, and obtain tissue surface morphology and deep anatomical information, providing a multi-dimensional data detection device and method for the field of medical imaging.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions:

[0013] On one hand, the present invention provides an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multimodal forward imaging, including a control system, a light source system, an optical path system, an endoscope probe, and a data acquisition and reconstruction system, wherein the control system is connected to the light source system and the data acquisition system respectively, and the light source system, the optical path system, and the endoscope probe are connected in sequence, and the data acquisition system is also connected to the endoscope probe and the data reconstruction system;

[0014] The control system uses an FPGA development board as a control module to generate timing pulses to control each imaging submodule;

[0015] The light source system includes a pulse laser, a continuous laser, and a white light source; the continuous laser and the white light source control end are connected to the FPGA development board to control the light source output, and the light source output end is connected to the optical path system through an optical fiber;

[0016] The optical path system includes an optical fiber coupler, a single-mode optical fiber, a second dichroic mirror, a second focusing lens, a coupler, a third collimator, a fourth collimator, a second reflector, a spectrometer, and a photomultiplier tube; the pulse laser output light is reflected by the second dichroic mirror and enters the optical fiber coupler, and the optical fiber coupler is connected to the second collimator in the endoscope probe through a single-mode optical fiber, and the fluorescence excitation light is emitted through the optical fiber coupler and incident on the second dichroic mirror, and the light emitted from the second dichroic mirror is focused by the second focusing lens and incident on the photocathode of the photomultiplier tube; the continuous laser output light is connected to the coupler through a single-mode optical fiber, and the coupler divides the input light into two, one way enters the endoscope probe through the first collimator, and the other way is collimated by the third collimator and incident on the second reflector, and the sample arm return light emitted by the first collimator interferes with the reflected light from the third collimator in the coupler, and is emitted through the fourth collimator and incident on the spectrometer. In the light source system, the white light source is connected to the optical fiber on the side wall of the endoscope probe through a single-mode optical fiber for tissue illumination;

[0017] The endoscope probe is composed of an integrated endoscope, including a charge coupled device (CCD), an eyepiece, a beam splitter, a first collimator, a second collimator, a first dichroic mirror, a first focusing lens, a first reflector, a MEMS mirror, a rod lens, an optical fiber, an ultrasonic transducer, a light-transmitting reflective mirror, and an endoscope housing; the pulsed laser output light enters the integrated endoscope via the second collimator, is horizontally incident through the first dichroic mirror, the continuous laser output light enters the integrated endoscope via the first collimator, is reflected by the first dichroic mirror and then combined with the pulsed light, the combined light is focused by the first focusing lens and then incident on the first reflector, and then incident on the surface of the MEMS mirror, the scanning light is reflected by the beam splitter and then incident on the rod lens, and is transmitted inside the rod lens and passes through the light-transmitting reflective mirror to be incident on the tissue surface;

[0018] The data acquisition and reconstruction system includes a first signal amplifier, a first acquisition card, a second signal amplifier, a second acquisition card, a third acquisition card, and a computer. The multimodal imaging system simultaneously images white light, photoacoustic, OCT, and fluorescence images. In photoacoustic imaging, a pulsed laser is incident on the tissue surface, and the surface is stimulated to generate ultrasonic waves that are received by the transducer on the side wall of the endoscope probe. The signal is amplified by the first signal amplifier and then enters the first acquisition card, which sends the original data to the computer; in OCT imaging, the spectrometer receives the interference signal, which is sent to the computer by the third acquisition card; in fluorescence imaging, the photomultiplier tube converts the fluorescence signal into a current signal, which is converted into a voltage signal via the second signal amplifier, and the second acquisition card sends the voltage signal to the computer.

[0019] Preferably, the first dichroic mirror is used to make the fluorescence imaging, photoacoustic imaging, and OCT imaging share a common optical path. The wavelength of the fluorescence and photoacoustic signal excitation light is quite different from that of the OCT excitation light. By selecting the first dichroic mirror with a cutoff frequency of 850nm to 1180nm, the OCT signal excitation light is reflected and the fluorescence and photoacoustic signal excitation light are transmitted, thereby realizing spatial light beam combining.

[0020] Preferably, the first dichroic mirror is a short-wave pass dichroic mirror with a cut-off wavelength of 850 nm to 1180 nm.

[0021] Preferably, the second dichroic mirror is used to make the fluorescence imaging and the photoacoustic imaging share the same optical path. The fluorescence signal and the photoacoustic signal excitation light are reflected by the second dichroic mirror and enter the fiber coupler. The fluorescence signal wavelength is higher than the fluorescence excitation light. The fluorescence signal is emitted from the fiber coupler through the second dichroic mirror, and the light below the cut-off wavelength of the second dichroic mirror will be reflected. The fluorescence signal is focused by the focusing lens and enters the photomultiplier tube, thereby realizing simultaneous imaging of photoacoustic imaging and fluorescence imaging.

[0022] Preferably, the second dichroic mirror is a long-wave pass dichroic mirror with a cut-off wavelength of 550 nm to 780 nm.

[0023] Preferably, the pulsed laser and continuous laser inside the endoscope probe are combined through a first dichroic mirror, focused through a laser focusing optical path, and then reflected by a reflector to the surface of the MEMS mirror. The control system controls the deflection of the MEMS mirror to adjust the scanning range. Compared with galvanometer scanning, the MEMS mirror reduces the size of the endoscope, reduces the weight of the endoscope, and has a high reflectivity to the incident light.

[0024] Preferably, the first collimator is a focus-adjustable collimator. Since the wavelengths of the excitation light of the photoacoustic and fluorescence signals are greatly different from those of the OCT signal, dispersion will occur when passing through the same lens. By adjusting the focus-adjustable collimator, the multi-mode imaging confocal point can be achieved, and clear images can be obtained at the same position under different imaging modes.

[0025] Preferably, the pulse laser is one of a semiconductor laser, a solid laser, a dye laser or a gas laser, and the output pulse laser has a wavelength range of 400nm to 2500nm and a pulse width of 5ns to 50ns.

[0026] Preferably, the amplification gain of the first signal amplifier is 50dB-60dB, and the gain of the second signal amplifier is 46dB.

[0027] Another aspect of the present invention provides an imaging method of an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging, comprising the following steps:

[0028] (1) System triggering: The control system generates a trigger signal to control the pulse laser to generate pulse light and the continuous laser to generate continuous light. At the same time, it controls the illumination light source. The pulse laser enters the fiber coupler through the second dichroic mirror and enters the endoscope through the second collimator. The continuous laser passes through the coupler and the first collimator and enters the endoscope. After being reflected by the first dichroic mirror, it is combined with the pulse laser. The combined light is reflected by the reflector and focused by the focusing lens and finally reaches the surface of the tissue to be imaged. The illumination light irradiates the surface of the tissue to be imaged through the optical fiber.

[0029] (2) Data acquisition: Photoacoustic data acquisition: The photoacoustic signal excited by the pulsed laser is reflected by the light-transmitting reflective mirror and received by the ultrasonic transducer on the side wall of the endoscope, and then converted into an electrical signal. The signal enters the first amplifier through the coaxial cable, and the amplified electrical signal is collected by the first acquisition card and sent to the computer; OCT data acquisition: The OCT signal excited by the continuous laser is the backscattered light from the surface of the tissue to be imaged. It is received by the endoscope, emitted along the incident light path, enters the coupler through the first collimator, interferes with the reflected light from the reference arm, and the interference signal intensity is detected by the spectrometer and converted into an electrical signal at the same time. The third acquisition card collects and sends it to the computer; Fluorescence data acquisition: The fluorescence signal excited by the pulsed laser is received by the endoscope, emitted along the incident light path, enters the fiber coupler through the second collimator, is focused by the focusing lens through the second dichroic mirror, and enters the photomultiplier tube. The photomultiplier tube converts the fluorescence signal into an electrical signal, which is collected by the second acquisition card after passing through the second amplifier and sent to the computer; Optical data acquisition: The optical signal is received by the endoscope, passes through the rod lens group and the spectroscope to enter the CCD. The CCD can convert the incident light signal into a charge output, and the electrical signal enters the computer through the data line;

[0030] (3) Image processing: The computer uses the collected data to reconstruct optical images, photoacoustic images, OCT images, and fluorescence images.

[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0032] 1. The present invention realizes the integration of four imaging methods: optical imaging, photoacoustic imaging, OCT imaging and fluorescence imaging, and can perform real-time high-resolution confocal imaging. Among them, OCT technology has the characteristics of non-invasiveness, high resolution, and in vivo detection of biological tissue structure compared with other technologies. After combining with endoscopic technology, OCT endoscopic imaging technology can perform in vivo high-resolution imaging of biological tissue cavities to obtain tissue structure information. OCT technology can be used for structural detection of malignant tumor tissue lesions. When the tissue structure changes, the tissue scattering coefficient will change. At the same time, it can monitor the changes in tissue structure during treatment to achieve the purpose of precise treatment. Fluorescence imaging is to reconstruct fluorescence images by detecting the fluorescence signal after the fluorescent substance is excited. Fluorescent molecular targeted imaging has high sensitivity and specificity, can mark the tumor boundary, and provide important reference for delineating biological target areas during treatment. It can be used for real-time positioning of lesions and real-time inspection of tumor resection margin residues. Photoacoustic imaging technology combines the characteristics of high resolution of optical imaging and deep depth of acoustic imaging. It can image the tumor microvascular network, judge the depth of tumor infiltration, and provide important reference for judging tumor staging. At the same time, it can be used for postoperative screening, imaging the microstructure morphology of the edge area, and judging whether there is residual resection margin. Compared with traditional white light endoscopes, the multimodal endoscope device of the present invention has the characteristics of information integration, tomographic imaging, high resolution, and deep imaging depth. It is suitable for in vivo diagnosis of various malignant tumors. It can combine the advantages of various imaging devices to obtain medical images of tissues and organs, especially early detection of nasopharyngeal carcinoma and preoperative and postoperative imaging evaluation, providing a multi-dimensional data detection device and method for the field of medical imaging.

[0033] 2. The combination of the light-transmitting reflective mirror and the rod lens inside the multimodal endoscope probe of the present invention can realize optical beam combining and optical path multiplexing, realize real-time high-resolution confocal imaging of the target tissue area, and simultaneously obtain photoacoustic images, OCT images, fluorescence images, and white light images. The photoacoustic, OCT, and fluorescence excitation light are focused and incident on the tissue to be imaged via the rod lens. The presence of the rod lens can extend the working distance of the endoscope. The photoacoustic excitation light is incident on the biological tissue, and the echo signal is reflected by the light-transmitting reflective mirror and received by the lateral transducer. The OCT backscattered light and the fluorescence signal are returned from the incident light path.

[0034] 3. The multi-modal endoscope probe of the present invention realizes multi-mode confocal by using an adjustable focus collimator. The different wavelengths of the photoacoustic, OCT, and fluorescence excitation light cause dispersion when passing through the same lens, reducing the image resolution. By using an adjustable focus collimator, the position of the incident light spot on the tissue is changed, and the three imaging modes of photoacoustic, OCT, and fluorescence are confocal, thereby improving the image resolution.

[0035] 4. The internal scanning structure of the multi-modal endoscope probe of the present invention adopts a MEMS mirror. Compared with the galvanometer scanning, the MEMS mirror can reduce the volume and weight of the endoscope, and at the same time has a high reflectivity to the incident light.

[0036] 5. The internal lens structure of the multimodal endoscopic probe of the present invention can change the focal position of the photoacoustic system and the OCT system, extend the working distance of the multimodal imaging system, and the endoscopic probe can penetrate into narrow cavity organs for in vivo detection. The system working distance can be changed by adjusting the number of lens groups, thereby achieving high-resolution confocal endoscopic imaging of cavity organ tissues at different depths. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 It is a structural schematic diagram of an endoscopic device that integrates optical, photoacoustic, OCT, and fluorescent multimodal forward imaging.

[0039] Figure 2 The present invention is a schematic diagram of an imaging device of an endoscopic device that integrates optical, photoacoustic, OCT, and fluorescence multimodal forward imaging.

[0040] Figure 3 The schematic diagram of the endoscope structure is an endoscopic device that integrates optical, photoacoustic, OCT, and fluorescence multimodal forward imaging.

[0041] Explanation of the accompanying drawings: 1. Computer; 2. Second acquisition card; 3. Second signal amplifier; 4. First acquisition card; 5. FPGA development board; 6. Third acquisition card; 7. Photomultiplier tube; 8. First signal amplifier; 9. White light source; 10. Superluminescent diode; 11. Fourth collimator; 12. Spectrometer; 13. Second focusing lens; 14. Coupler; 15. Integrated endoscope; 16. Third collimator; 17. Second reflector; 18. Pulse laser; 19. Second dichroic mirror; 20. Fiber coupler; 21. Charge coupled device; 22. Eyepiece; 23. Spectroscope; 24. MEMS mirror; 25. First collimator; 26. Second collimator; 27. Optical fiber; 28. First reflector; 29. ​​First focusing lens; 30. First dichroic mirror; 31. Rod lens; 32. Ultrasonic transducer; 33. Transparent acoustic mirror. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0043] like Figure 2 As shown, this embodiment discloses an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging of the present invention, including a control system, a light source system, an optical path system, an endoscope probe, and a data acquisition and reconstruction system.

[0044] Among them, the control system uses the FPGA development board 5 as the control module to generate timing pulses to control each imaging sub-module; the light source system includes a pulse laser 18, a continuous laser, and a white light source 9; the optical path system includes an optical fiber coupler 20, a single-mode optical fiber, a second dichroic mirror 19, a second focusing lens 13, a coupler 14, a third collimator 16, a fourth collimator 11, a second reflector 17, a spectrometer 12, and a photomultiplier tube 7; the endoscope probe is composed of an integrated endoscope 15, including a charge coupled device 21 (CCD), an eyepiece 22, a spectroscope 23, a first collimator 25, a second collimator 26, a first dichroic mirror 30, a first focusing lens 29, a first reflector 28, a MEMS mirror 24, a rod lens 31, an optical fiber 27, an ultrasonic transducer 32, a light-transmitting reflective mirror 33, and an endoscope housing; the data acquisition and reconstruction system includes a first signal amplifier 8, a first acquisition card 4, a second signal amplifier 3, a second acquisition card 2, a third acquisition card 6, and a computer 1.

[0045] Furthermore, the control system uses an FPGA5 development board as a control module to generate timing pulses to control each imaging submodule; each imaging submodule includes an optical imaging module, a photoacoustic imaging module, an OCT imaging module, and a fluorescence imaging module.

[0046] Furthermore, in the light source system, the continuous laser and the white light source control terminal are connected to the FPGA development board 5 for controlling the light source output, and the light source output terminal is connected to the optical path system through an optical fiber.

[0047] Furthermore, in the optical path system, the output light of the pulse laser 18 is reflected by the second dichroic mirror 19 and enters the fiber coupler 20. The fiber coupler 20 is connected to the second collimator 26 in the endoscope probe through a single-mode optical fiber. The fluorescence excitation light is emitted through the fiber coupler 20 and enters the second dichroic mirror 19. The light emitted by the second dichroic mirror 19 is focused by the second focusing lens 13 and enters the photocathode of the photomultiplier tube 7. The output light of the continuous laser is connected to the coupler 14 through a single-mode optical fiber. The coupler 14 divides the input light into two parts. One part enters the endoscope probe through the first collimator 25, and the other part is collimated by the third collimator 16 and enters the second reflector 17. The sample arm return light emitted by the first collimator 25 interferes with the reflected light from the third collimator 16 in the coupler 14, and is emitted through the fourth collimator 11 and enters the spectrometer. In the light source system, the white light source is connected to the optical fiber on the side wall of the endoscope probe through a single-mode optical fiber for tissue illumination.

[0048] Furthermore, in the endoscopic probe, the output light of the pulsed laser 18 enters the integrated endoscope 15 via the second collimator 26, and is horizontally incident through the first dichroic mirror 30; the output light of the continuous laser enters the integrated endoscope via the first collimator 25, and is combined with the pulsed light after being reflected by the first dichroic mirror 30; the combined light is focused by the first focusing lens 29 and then incident on the first reflecting mirror 28, and then reflected and incident on the surface of the MEMS mirror 24; the scanning light is reflected by the beam splitter 23 and then incident on the rod lens 31, and is transmitted inside the rod lens 31 through the light-transmitting reflective mirror to enter the tissue surface.

[0049] Furthermore, the data acquisition and reconstruction system includes a first signal amplifier 8, a first acquisition card 4, a second signal amplifier 3, a second acquisition card 2, a third acquisition card 6, and a computer 1. The multimodal imaging system simultaneously images white light, photoacoustic, OCT, and fluorescence images. In photoacoustic imaging, a pulsed laser is incident on the tissue surface, and the surface is stimulated to generate ultrasonic waves that are received by the transducer on the side wall of the endoscope probe. The signal is amplified by the first signal amplifier and then enters the first acquisition card 4, which sends the original data to the computer; in OCT imaging, the spectrometer receives the interference signal, which is sent to the computer 1 by the third acquisition card 6; in fluorescence imaging, the photomultiplier tube 7 converts the fluorescence signal into a current signal, which is converted into a voltage signal via the second signal amplifier 3, and the second acquisition card 2 sends the voltage signal to the computer 1.

[0050] In this embodiment, the multimodal imaging system can simultaneously image white light, photoacoustic, OCT, and fluorescence images, and the imaging process is completed by the optical imaging module, the photoacoustic imaging module, the OCT imaging module, and the fluorescence imaging module.

[0051] The optical imaging module includes the charge coupled device 21 (CCD) in the above-mentioned endoscope probe, the eyepiece 22, the beam splitter 23, the rod lens 31, the light-transmitting reflective mirror 33, the optical fiber 27 and the FPGA development board 5 in the control system, the white light source 9 in the light source system, and the computer 1 in the data acquisition and reconstruction system.

[0052] The photoacoustic imaging module includes the second collimator 26, the first dichroic mirror 30, the first focusing lens 29, the first reflecting mirror 28, the MEMS mirror 24, the beam splitter 23, the rod lens 31, the light-transmitting reflective mirror 33, the ultrasonic transducer 32 in the above-mentioned endoscopic probe and the FPGA development board 5 in the control system, the pulse laser 18 in the light source system, the second dichroic mirror 19, the fiber coupler 20, the first collimator 26 in the optical path system, the first signal amplifier 8, the first acquisition card 4, and the computer 1 in the data acquisition and reconstruction system.

[0053] The OCT imaging module includes the first collimator 25, the first dichroic mirror 30, the first focusing lens 29, the first reflector 28, the MEMS mirror 24, the beam splitter 23, the rod lens 31, the light-transmitting reflective mirror 33 in the above-mentioned endoscopic probe and the FPGA development board 5 in the control system, the continuous laser 10 in the light source system, the coupler 14, the fourth collimator 11, the spectrometer 12, the third collimator 16, the second reflector 17, the second collimator 25 in the optical path system, the third acquisition card 6 and the computer 1 in the data acquisition and reconstruction system.

[0054] The fluorescence imaging module includes the second collimator 26, the first dichroic mirror 30, the first focusing lens 29, the first reflecting mirror 28, the MEMS mirror 24, the beam splitter 23, the rod lens 31, the light-transmitting reflective mirror 33 in the above-mentioned endoscope probe and the FPGA development board 5 in the control system, the pulse laser 18 in the light source system, the second dichroic mirror 19, the fiber coupler 20, the first collimator 26, the second focusing lens 13, the photomultiplier tube 7 in the optical path system, and the second amplifier 3, the second acquisition card 2, and the computer 1 in the data acquisition and reconstruction system.

[0055] In this embodiment, the second dichroic mirror is a long-wave pass dichroic mirror with a cut-off wavelength of 550 nm. The addition of the second dichroic mirror allows fluorescence imaging and photoacoustic imaging to share the same optical path. The excitation light wavelength of the fluorescence signal and the photoacoustic signal is 527 nm. The excitation light is reflected by the dichroic mirror and enters the fiber coupler. The wavelength of the fluorescence signal is higher than the fluorescence excitation light. The fluorescence signal is emitted from the fiber coupler through the dichroic mirror, and the light below the cut-off wavelength of the dichroic mirror will be reflected. The fluorescence signal is focused by the focusing lens and enters the photomultiplier tube, so that photoacoustic imaging and fluorescence imaging can be achieved simultaneously.

[0056] In this embodiment, the fluorescent dye is selected as rhodamine B, the laser wavelength emitted by the pulse laser is 527nm, and the laser wavelength emitted by the superluminescent diode is 1310nm. The pulse laser is a semiconductor laser, a solid laser, a dye laser or a gas laser, and the output pulse laser wavelength ranges from 400nm to 2500nm, and the pulse width is 5ns to 50ns.

[0057] In this embodiment, the optical fiber on the side wall of the endoscope probe is used to transmit the illuminating white light, the light reflected from the tissue surface passes through the beam splitter via the rod lens, the double plano-convex lens of the eyepiece group is used to focus the white light, and the CCD collects the white light image.

[0058] In this embodiment, the first dichroic mirror is a short-wave dichroic mirror with a cutoff wavelength of 950nm. The addition of the first dichroic mirror allows fluorescence imaging, photoacoustic imaging, and OCT imaging to share the same optical path. The wavelengths of fluorescence and photoacoustic signal excitation light and OCT excitation light are quite different. By selecting a dichroic mirror with an appropriate cutoff frequency, it is possible to reflect fluorescence and photoacoustic signal excitation light and transmit OCT signal excitation light, thereby achieving spatial light beam combination.

[0059] In this embodiment, the pulsed laser and continuous laser inside the endoscope probe are combined by a dichroic mirror, focused by the laser focusing optical path, and then reflected by the reflector to the surface of the MEMS mirror. The control system controls the deflection of the MEMS mirror to adjust the scanning range. Compared with galvanometer scanning, the MEMS mirror can reduce the size of the endoscope, reduce the quality of the endoscope, and at the same time has a high reflectivity to the incident light.

[0060] In this embodiment, white light guides the probe to the tissue area to be imaged for scanning, and the endoscope probe adopts a forward scanning mode. The use of a rod lens can extend the working distance, and the addition of a light-transmitting reflective mirror at the front end of the probe can reflect the photoacoustic echo signal, which is received by the side wall high-frequency ultrasonic transducer, and the main frequency of the high-frequency ultrasonic transducer is 40M and the bandwidth is 80%.

[0061] In this embodiment, the addition of an adjustable focus collimator can achieve the confocal point of multi-mode imaging. Among them, the first collimator is an adjustable focus collimator. Since the wavelengths of the photoacoustic and fluorescence signal excitation light and the OCT signal excitation light are quite different, dispersion will occur when passing through the same lens. By adjusting the adjustable focus collimator, the confocal point of multi-mode imaging can be achieved, and clear images can be obtained in different imaging modes at the same position.

[0062] In this embodiment, the amplification gain of the first signal amplifier is 50 dB, and the gain of the second signal amplifier is 46 dB.

[0063] In this embodiment, the endoscope housing is a metal housing with biocompatibility, the working distance is 20 cm, and the endoscope diameter is 3 mm.

[0064] In order to achieve the above second purpose, the present invention adopts the following technical solutions:

[0065] The method for performing imaging using a quad-modal imaging device of the present invention comprises the following steps:

[0066] System triggering: The control system generates a trigger signal to control the pulse laser and continuous laser to generate laser light, and controls the illumination light source at the same time. The pulse laser enters the fiber coupler through the dichroic mirror and enters the endoscope through the collimator. The continuous laser passes through the coupler and enters the endoscope through the collimator. After being reflected by the dichroic mirror, it is combined with the pulse laser. The combined light is reflected by the reflector and focused by the focusing lens and finally reaches the surface of the tissue to be imaged. The illumination light irradiates the surface of the tissue to be imaged through the optical fiber.

[0067] Data acquisition: Photoacoustic data acquisition: The photoacoustic signal excited by the pulsed laser is reflected by the light-transmitting reflective mirror and received by the ultrasonic transducer on the side wall of the endoscope, and then converted into an electrical signal. It enters the first amplifier through the coaxial cable, and the amplified electrical signal is collected by the first acquisition card and sent to the computer. OCT data acquisition: The OCT signal excited by the continuous laser is the backscattered light from the surface of the tissue to be imaged. It is received by the endoscope, emitted along the incident light path, enters the coupler through the first collimator, interferes with the reflected light from the reference arm, and the intensity of the interference signal is detected by the spectrometer and converted into an electrical signal at the same time. It is collected by the third acquisition card and sent to the computer. Fluorescence data acquisition: The fluorescence signal excited by the pulsed laser is received by the endoscope, emitted along the incident light path, enters the fiber coupler through the second collimator, and enters the photomultiplier tube after being focused by the focusing lens through the dichroic mirror. The photomultiplier tube converts the fluorescence signal into an electrical signal, which is collected by the second acquisition card after passing through the second amplifier and sent to the computer. Optical data acquisition: The optical signal is received by the endoscope, passes through the rod lens group and enters the CCD through the spectroscope. The CCD can convert the incident light signal into an electric charge output and enter the electrical signal into the computer through the data line.

[0068] Image processing: The computer uses the collected data to reconstruct optical images, photoacoustic images, OCT images, and fluorescence images.

[0069] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0070] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An endoscope device integrating optical, photoacoustic, OCT, and fluorescence multimodal forward imaging, characterized in that: It includes a control system, a light source system, an optical path system, an endoscope probe, and a data acquisition and reconstruction system. The control system is connected to the light source system and the data acquisition system respectively. The light source system, the optical path system, and the endoscope probe are connected in sequence. The data acquisition system is also connected to the endoscope probe and the data reconstruction system. The control system uses an FPGA development board as a control module to generate timing pulses to control each imaging submodule; The light source system includes a pulse laser, a continuous laser, and a white light source; the continuous laser and the white light source control end are connected to the FPGA development board to control the light source output, and the light source output end is connected to the optical path system through an optical fiber; The optical path system includes an optical fiber coupler, a single-mode optical fiber, a second dichroic mirror, a second focusing lens, a coupler, a third collimator, a fourth collimator, a second reflector, a spectrometer, and a photomultiplier tube; the pulse laser output light is reflected by the second dichroic mirror and enters the optical fiber coupler, the optical fiber coupler is connected to the second collimator in the endoscope probe through the single-mode optical fiber, the fluorescent excitation light is emitted through the optical fiber coupler and enters the second dichroic mirror, and the output light of the second dichroic mirror is focused by the second focusing lens and enters the photocathode of the photomultiplier tube; The output light of the continuous laser is connected to the coupler through a single-mode optical fiber. The coupler splits the input light into two. One path passes through the first collimator and enters the endoscope probe. The other path passes through the third collimator and enters the second reflector. The return light of the sample arm emitted from the first collimator interferes with the reflected light from the third collimator in the coupler and is emitted through the fourth collimator and enters the spectrometer. The white light source in the light source system is connected to the optical fiber on the side wall of the endoscope probe through a single-mode optical fiber for tissue illumination; the first dichroic mirror is used to make the fluorescence imaging, photoacoustic imaging, and OCT imaging share a common optical path. The wavelengths of the fluorescence and photoacoustic signal excitation light and the OCT excitation light differ greatly. By selecting the first dichroic mirror with a cutoff frequency of 850nm~1180nm, the OCT signal excitation light is reflected and the fluorescence and photoacoustic signal excitation light are transmitted, thereby realizing the combination of spatial light; the second dichroic mirror is used to make the fluorescence imaging and the photoacoustic imaging share a common optical path. The fluorescence signal and the photoacoustic signal excitation light are reflected by the second dichroic mirror and enter the optical fiber coupler. The wavelength of the fluorescence signal is higher than that of the fluorescence excitation light. The fluorescence signal is emitted from the optical fiber coupler through the second dichroic mirror, and the light below the cutoff wavelength of the second dichroic mirror will be reflected. The fluorescence signal is focused by the focusing lens and enters the photomultiplier tube, thereby realizing the simultaneous imaging of photoacoustic imaging and fluorescence imaging; The endoscope probe is composed of an integrated endoscope, including a charge coupled device (CCD), an eyepiece, a beam splitter, a first collimator, a second collimator, a first dichroic mirror, a first focusing lens, a first reflector, a MEMS mirror, a rod lens, an optical fiber, an ultrasonic transducer, a light-transmitting reflective mirror, and an endoscope housing; the pulsed laser output light enters the integrated endoscope via the second collimator, is horizontally incident through the first dichroic mirror, the continuous laser output light enters the integrated endoscope via the first collimator, is reflected by the first dichroic mirror and then combined with the pulsed light, the combined light is focused by the first focusing lens and then incident on the first reflector, and then incident on the surface of the MEMS mirror, the scanning light is reflected by the beam splitter and then incident on the rod lens, and is transmitted inside the rod lens and passes through the light-transmitting reflective mirror to be incident on the tissue surface; The data acquisition and reconstruction system includes a first signal amplifier, a first acquisition card, a second signal amplifier, a second acquisition card, a third acquisition card, and a computer. The multimodal imaging system simultaneously images white light, photoacoustic, OCT, and fluorescence images. In photoacoustic imaging, a pulsed laser is incident on the tissue surface, and the surface is stimulated to generate ultrasonic waves that are received by the transducer on the side wall of the endoscope probe. The signal is amplified by the first signal amplifier and then enters the first acquisition card, and the first acquisition card sends the original data to the computer. In OCT imaging, the spectrometer receives the interference signal, which is sent to the computer by the third acquisition card. In fluorescence imaging, the photomultiplier tube converts the fluorescence signal into a current signal, which is converted into a voltage signal via the second signal amplifier, and the second acquisition card sends the voltage signal to the computer.

2. According to claim 1, an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multimodal forward imaging, characterized in that: The first dichroic mirror is a short-wave pass dichroic mirror with a cut-off wavelength of 850nm~1180nm.

3. According to claim 1, an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging, characterized in that: The second dichroic mirror is a long-wave pass dichroic mirror with a cut-off wavelength of 550nm~780nm.

4. According to claim 1, an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging, characterized in that: The pulsed laser and continuous laser inside the endoscope probe are combined through a first dichroic mirror, focused through a laser focusing optical path, and then reflected by a reflector to the surface of the MEMS mirror. The control system controls the deflection of the MEMS mirror to adjust the scanning range. Compared with galvanometer scanning, the MEMS mirror reduces the size of the endoscope, reduces the weight of the endoscope, and has a high reflectivity to the incident light.

5. According to claim 1, an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging, characterized in that: The first collimator is a focus-adjustable collimator. Since the wavelengths of the excitation light of the photoacoustic and fluorescence signals are greatly different from those of the OCT signal, dispersion will occur when passing through the same lens. By adjusting the focus-adjustable collimator, the multi-mode imaging confocal point can be achieved, and clear images can be obtained at the same position under different imaging modes.

6. According to claim 1, an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multimodal forward imaging, characterized in that: The pulse laser is one of a semiconductor laser, a solid laser, a dye laser or a gas laser, and the output pulse laser has a wavelength range of 400nm to 2500nm and a pulse width of 5ns to 50ns.

7. According to claim 1, an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging, characterized in that: The amplification gain of the first signal amplifier is 50dB-60dB, and the gain of the second signal amplifier is 46dB.

8. An imaging method of an endoscope device integrating optical, photoacoustic, OCT, and fluorescence multi-modal forward imaging according to any one of claims 1 to 7, characterized in that: The steps include: (1) System triggering: The control system generates a trigger signal to control the pulse laser to generate pulse light and the continuous laser to generate continuous light. At the same time, the illumination light source is controlled. The pulse laser enters the fiber coupler through the second dichroic mirror and enters the endoscope through the second collimator. The continuous laser passes through the coupler and the first collimator and enters the endoscope. After being reflected by the first dichroic mirror, it is combined with the pulse laser. The combined light is reflected by the reflector and focused by the focusing lens and finally reaches the surface of the tissue to be imaged. The illumination light irradiates the surface of the tissue to be imaged through the optical fiber. (2) Data acquisition: Photoacoustic data acquisition: The photoacoustic signal excited by the pulsed laser is reflected by the light-transmitting reflective mirror and received by the ultrasonic transducer on the side wall of the endoscope, and then converted into an electrical signal. The signal enters the first amplifier through the coaxial cable, and the amplified electrical signal is collected by the first acquisition card and sent to the computer. OCT data acquisition: The OCT signal excited by the continuous laser is the backscattered light from the surface of the tissue to be imaged. It is received by the endoscope, emitted along the incident light path, enters the coupler through the first collimator, interferes with the reflected light from the reference arm, and the interference signal intensity is detected by the spectrometer and converted into an electrical signal at the same time. The third acquisition card collects and sends it to the computer; Fluorescence data acquisition: The fluorescence signal excited by the pulsed laser is received by the endoscope, emitted along the incident light path, enters the fiber coupler through the second collimator, is focused by the focusing lens through the second dichroic mirror, and enters the photomultiplier tube. The photomultiplier tube converts the fluorescence signal into an electrical signal, which is collected by the second acquisition card after passing through the second amplifier and sent to the computer; Optical data acquisition: The optical signal is received by the endoscope, passes through the rod lens group and the spectroscope to enter the CCD. The CCD can convert the incident light signal into a charge output, and the electrical signal enters the computer through the data line; (3) Image processing: The computer uses the collected data to reconstruct optical images, photoacoustic images, OCT images, and fluorescence images.

Images