Image forming apparatus

Abstract

To provide a technique that enables distinguishable detection of individual fluorescence images of two types of fluorescent substances.SOLUTION: Provided is an image forming apparatus (1), wherein excitation light for each of a first fluorescent substance and a second fluorescent substance is emitted from an excitation light source (10) onto a subject, wherein fluorescence in individual wavelength regions characteristic of each fluorescent substance is transmitted through a notch filter (20) from return light from the subject, wherein fluorescence images in the individual wavelength regions are separately captured by an imaging unit (30), and wherein these images are superimposed by an image processing unit (40).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image forming apparatus. [Background technology]

[0002] Fluorescence imaging is known as a medical observation system for identifying the presence or absence of a tumor in a living tissue or the location of the tumor. Fluorescence imaging is a technique in which a fluorescent reagent is administered into a living body to specifically accumulate in a tumor or the like within the living body, the fluorescent reagent is excited with light of a specific wavelength, and the fluorescence emitted by the fluorescent reagent is captured and displayed as an image. In this way, by detecting the fluorescence within the living body, it is possible to ascertain the presence or absence and location of a tumor.

[0003] As a medical observation system, a technique is known in which a fluorescent substance, indocyanine green (hereinafter also referred to as "ICG"), is administered into a living body, and excitation light is irradiated onto the ICG, so that the ICG fluorescence in the near infrared region generated in response to the excitation light is visualized and used as a marker. As a technique for imaging the fluorescence of a fluorescent reagent, a technique related to an endoscope is known in which a single image sensor detects the near infrared (also referred to as "NIR") fluorescence (800 to 850 nm) of ICG and the fluorescence (680 to 740 nm) of methylene blue (also referred to as "MB") (see, for example, Patent Document 1). As a technique for imaging the fluorescence of a fluorescent reagent, a technique related to a fluorescence observation system is known in which only the short wave infrared (also referred to as "SWIR") fluorescence (900 to 2000 nm) of ICG is detected (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 018126 [Patent Document 2] Special Publication No. 2019-510220 Summary of the Invention [Problem to be solved by the invention]

[0005] Labeling with ICG alone produces a monochromatic image, making it difficult to distinguish between multiple locations within a living body. In the technology described in Patent Document 1, in addition to ICG, multicolor fluorescence imaging is attempted using MB, which is an approved pharmaceutical. However, the fluorescence spectrum of MB partially overlaps with that of ICG (see FIG. 2 below). Therefore, in fluorescence imaging using the vicinity of the peak wavelength of ICG fluorescence (800 to 850 nm), MB fluorescence of 800 nm or more is also detected in the image sensor for ICG, and it may be difficult to distinguish between the ICG fluorescence image and the MB fluorescence image.

[0006] An object of one aspect of the present invention is to provide a technique for detecting the respective fluorescent images of two types of fluorescent substances in a distinguishable manner. [Means for solving the problem]

[0007] In order to solve the above problems, an image forming apparatus according to one aspect of the present invention has an excitation light source that irradiates an object with excitation light of a first fluorescent substance and excitation light of a second fluorescent substance; an optical filter unit that transmits fluorescence of the first fluorescent substance in a first wavelength range including near infrared or short wave infrared and fluorescence of the second fluorescent substance in a second wavelength range including short wave infrared from the light from the object; an imaging unit that captures images of the fluorescence in the first wavelength range and the fluorescence in the second wavelength range that have transmitted through the optical filter unit; and an image processing unit that superimposes the image of the fluorescence in the first wavelength range and the image of the fluorescence in the second wavelength range in the imaging unit. Effect of the Invention

[0008] According to one aspect of the present invention, the fluorescence images of two types of fluorescent substances can be detected in a distinguishable manner. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram illustrating a schematic configuration of an image forming apparatus according to a first embodiment of the present invention. [Diagram 2] FIG. 2 shows the excitation spectrum and the fluorescence spectrum of methylene blue (MB) which is a first fluorescent substance and indocyanine green (ICG) which is a second fluorescent substance in the first embodiment of the present invention. [Diagram 3] 2 is a diagram showing the transmission wavelength range and the attenuation wavelength range of the notch filter according to the first embodiment of the present invention. FIG. [Figure 4] FIG. 2 is a diagram illustrating a schematic functional configuration of an image processing unit according to the first embodiment of the present invention. [Diagram 5] FIG. 2 is a photograph showing an example of an image of fluorescence in a first wavelength range obtained by the image forming apparatus according to the first embodiment of the present invention. [Figure 6] FIG. 3 is a photograph showing an example of an image of fluorescence in the second wavelength range obtained by the image forming apparatus according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of a composite image obtained by superimposing an image of fluorescence in a first wavelength range and an image of fluorescence in a second wavelength range on an image of visible light obtained by the image forming apparatus according to the first embodiment of the present invention. [Figure 8] FIG. 4 shows the excitation spectrum and fluorescence spectrum of a cyanine dye (FD-1080) which is a second fluorescent substance in embodiment 2 of the present invention. [Figure 9] FIG. 11 is a diagram showing the transmission wavelength range and the attenuation wavelength range of a notch filter according to a second embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an excitation spectrum of a rare earth ion-containing ceramic nanoparticle probe (OTN ceramic probe Y) which is a second fluorescent substance in the third embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing a fluorescence spectrum of a rare earth ion-containing ceramic nanoparticle probe (OTN ceramic probe Y) which is a second fluorescent substance in the third embodiment of the present invention. [Figure 12] FIG. 11 is a diagram showing the transmission wavelength range and the attenuation wavelength range of a notch filter according to a third embodiment of the present invention. [Figure 13]FIG. 13 shows an excitation spectrum and a fluorescence spectrum of a Cy dye-labeled carboxylic acid (Cy5.5) which is a first fluorescent substance in embodiment 4 of the present invention. [Figure 14] FIG. 11 is a diagram showing the transmission wavelength range and the attenuation wavelength range of a notch filter according to a fourth embodiment of the present invention. [Figure 15] FIG. 13 shows the excitation spectrum and the fluorescence spectrum of a DAD-type dye (CH1055) which is a second fluorescent substance in embodiment 5 of the present invention. [Figure 16] FIG. 11 is a diagram showing the transmission wavelength range and the attenuation wavelength range of a notch filter according to a fifth embodiment of the present invention. [Figure 17] FIG. 13 is a diagram illustrating a schematic configuration of an image forming apparatus according to a sixth embodiment of the present invention. [Figure 18] FIG. 13 is a diagram showing the transmission wavelength range and the attenuation wavelength range of a notch filter according to a sixth embodiment of the present invention. [Figure 19] FIG. 13 is a diagram illustrating an overall configuration of an image forming apparatus according to a seventh embodiment of the present invention. [Figure 20] FIG. 13 is a diagram illustrating a schematic functional configuration of an image forming apparatus according to a seventh embodiment of the present invention. [Figure 21] FIG. 13 is a plan view illustrating an imaging section according to a seventh embodiment of the present invention. [Figure 22] FIG. 13 is a cross-sectional view illustrating an imaging section according to a seventh embodiment of the present invention. [Diagram 23] FIG. 13 is a cross-sectional view illustrating a schematic exploded state of an imaging unit according to a seventh embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention detects the fluorescence of a first fluorescent substance and the fluorescence of a second fluorescent substance in wavelength ranges that do not substantially overlap with each other, thereby detecting the images of each fluorescent substance, thereby enabling further imaging using the images of the two fluorescent substances.

[0011] In the present invention, the first fluorescent substance and the second fluorescent substance may be fluorescent substances in which only one of the fluorescence of the first fluorescent substance and the fluorescence of the second fluorescent substance is substantially detected in at least one of the wavelength ranges of the fluorescence of the first fluorescent substance and the fluorescence of the second fluorescent substance. "Substantially detectable" means, for example, when one of the first fluorescent substance and the second fluorescent substance does not emit fluorescence and the other emits fluorescence, or when both the first fluorescent substance and the second fluorescent substance emit fluorescence but there is a sufficient difference in fluorescence intensity so that only one of the fluorescences can be detected.

[0012] In the present invention, unless otherwise specified, "near infrared" refers to a wavelength range between visible light and shortwave infrared, for example, a wavelength range of 700 to 900 nm. In the present invention, "visible light" may be a wavelength range of 380 to 780 nm or any wavelength range included therein. In addition, "shortwave infrared" refers to a wavelength range of 900 to 2500 nm. Furthermore, in the present invention, "a first wavelength range including near infrared or shortwave infrared" means that the first wavelength range at least partially overlaps with near infrared or shortwave infrared, and "a second wavelength range including shortwave infrared" means that the second wavelength range at least partially overlaps with shortwave infrared.

[0013] Here, in biological imaging as described above in the Background Art, fluorescent substances that emit near-infrared and short-wave infrared fluorescence are preferably used. When the present invention is applied to biological imaging, two types of fluorescent substances that emit either or both of near-infrared and short-wave infrared fluorescence and have at least one fluorescent wavelength range in which only one of them is substantially detectable can be determined as the first fluorescent substance and the second fluorescent substance in the present invention.

[0014] In the present invention, an optical filter suitable for two types of fluorescent substances having at least one fluorescent wavelength range in which only one of the two types of fluorescent substances is substantially detectable can be appropriately selected from fluorescent substances that emit fluorescence in either or both of the near infrared and short wave infrared wavelengths. In addition, the optical filter can be easily attached to and detached from the optical system of the image forming apparatus, and the fluorescence of a plurality of fluorescent substances can be detected.

[0015] Hereinafter, the embodiments of the present invention will be described in detail. The following embodiments are applied to biological imaging using a first fluorescent substance that produces near-infrared (NIR) fluorescence and a second fluorescent substance that produces shortwave infrared (SWIR) fluorescence for fluorescent labeling. These fluorescent substances are respectively applied to detection of different sites (e.g., liver segments and liver tumors) that may overlap in the same biological tissue. The combinations of the first fluorescent substance and the second fluorescent substance in the following embodiments are shown in Table 1 below.

[0016] [Table 1]

[0017] Table 2 shows the excitation wavelengths, maximum wavelengths, and excitation light wavelength ranges for the first fluorescent substance or the second fluorescent substance used in the following embodiments.

[0018] [Table 2]

[0019] [Embodiment 1] The configuration of an image forming apparatus according to the first embodiment of the present invention is shown in Fig. 1. The image forming apparatus according to this embodiment has a function of photographing an observation target using visible light, and a function of photographing fluorescence emitted by MB and ICG administered to the observation target upon excitation by irradiation with excitation light such as near-infrared light. As shown in Fig. 1, the image forming apparatus 1 has an excitation light source 10, a notch filter 20, an imaging unit 30, an image processing unit 40, and a monitor 50.

[0020] [Excitation light source] The excitation light source 10 is disposed at the tip of the irradiation probe 11 so as to irradiate the object of observation with excitation light of a first fluorescent substance (MB in this embodiment) and excitation light of a second fluorescent substance (ICG in this embodiment).

[0021] The excitation light source 10 includes, for example, a first laser that generates light with a wavelength of 660 nm and a second laser that generates light with a wavelength of 808 nm. The first laser is a light source that generates excitation light for MB, and the second laser is a light source that generates excitation light for ICG. These lasers are configured so that their outputs can be adjusted independently. The first laser and the second laser may be configured so that they each output light with a specific intensity, or each output can be appropriately changed.

[0022] The excitation light source 10 independently controls the output of the first laser and the output of the second laser to irradiate the object of observation with each excitation light. For example, in consideration of the difference in fluorescence intensity between MB, whose fluorescence is detected in a wavelength range near the peak wavelength, and ICG, whose fluorescence is detected near the end of the fluorescence wavelength, the output of the second laser is set to be larger than the output of the first laser so that the SWIR fluorescence of ICG is stronger.

[0023] The irradiation probe 11 further has a visible light source (not shown) at its tip. The visible light source is a light source for irradiating an observation target with visible light.

[0024] The irradiation probe 11 is a part that is inserted into the body of a test subject who has been administered MB and ICG in advance. The irradiation probe 11 is, for example, a cylindrical object having a diameter of about 5 to 10 mm. The irradiation probe 11 has an excitation light source 10 and a visible light source at its tip, and further includes an optical system. The optical system is, for example, an objective lens.

[0025] [Focus shift compensation] The objective lens is corrected for focus shift of light in the wavelength range from visible light to short-wave infrared light (for example, 400 to 1700 nm). For example, the objective lens is optically designed to reduce deviation in the back focus (BF) position in the wavelength ranges of visible light, near infrared, and short-wave infrared. "Back focus" (BF) is the distance from the surface closest to the image side of the optical system to the focal position, and in this embodiment, it is the focus position of paraxial light rays (light rays passing at a height very close to the optical axis). The value of the focus position does not change even if the F-number of the lens changes.

[0026] From the viewpoint of effectively correcting the aberration of the image detected by each sensor, it is preferable that the objective lens satisfies the following formula (1): In the formula, "BF_550nm" represents the back focus of the entire optical system at 550nm, and "BF_850nm" represents the back focus of the entire optical system at 850nm. |BF_550nm-BF_850nm|<0.03mm (1)

[0027] The difference between the presence and absence of focus shift correction is more noticeable on the long wavelength side, and the focus position shift on the long wavelength side is corrected to a greater extent (more effectively). By correcting the focus shift of the objective lens, it becomes possible to use a common light projection system for three types of image sensors in the light receiving system described below, and further, adjustment of the positions of the image sensors according to the focus shift is reduced or eliminated. Therefore, it is more preferable to correct the focus shift of the objective lens from the viewpoint of simplifying the optical design of the imaging unit 30.

[0028] The irradiation probe 11 is detachably connected to the imaging unit 30 via a notch filter 20. The image forming apparatus 1 is configured so that light received by the irradiation probe 11 is guided to the imaging unit 30 via the notch filter 20. Such a light-guiding configuration is, for example, a configuration that realizes a method called a relay lens or pupil relay, which transmits an image formed by light in a relay format. Alternatively, the configuration is, for example, an optical fiber capable of transmitting image information, such as an image guide fiber.

[0029] [Notch filter] The notch filter 20 is an optical filter section that transmits the NIR fluorescence of MB and the SWIR fluorescence of ICG from the light from the observation target. Thus, in this embodiment, the wavelength range (first wavelength range) of the fluorescence of MB, which is the first fluorescent material, is included in the NIR, and the wavelength range (second wavelength range) of the fluorescence of ICG, which is the second fluorescent material, is included in the SWIR.

[0030] As shown in Figure 2 (source: "Figure 3" in "Image-Guided Surgery using Invisible Near-Infrared Light: Fundamentals of Clinical Translation," Mol Imaging. Author manuscript; available in PMC 2011 October 1), MB has a peak of fluorescence intensity in the NIR. ICG has fluorescence in the NIR, but in the wavelength range near the peak wavelength of MB fluorescence, the intensity of ICG's NIR fluorescence is sufficiently smaller than that of MB. On the other hand, ICG has fluorescence in the SWIR, but the intensity of MB's SWIR fluorescence is sufficiently lower than that of ICG, so it can be said that MB has virtually no SWIR fluorescence compared to ICG. In addition, since MB is not excited by the wavelength of the second laser, 808 nm, the difference between the intensity of ICG's SWIR fluorescence and that of MB's SWIR fluorescence can be increased by increasing the output of the second laser, and the output of the second laser does not affect the SWIR intensity of MB. The notch filter 20 is designed to transmit light in the wavelength range in which the intensity difference of these fluorescent lights is observed, and to attenuate light in the other wavelength ranges except for visible light.

[0031] The transmission wavelength range and attenuation wavelength range of the notch filter 20 are shown in FIG. 3. As shown in FIG. 3, the notch filter 20 transmits light in a specific visible light wavelength range WR VIS , NIR wavelength range WR NIR , and SWIR wavelength range WR SWIR WR VIS and W.R.NIR The first attenuation wavelength region WR A1 , and W.R. NIR and W.R. SWIR The first attenuation wavelength region WR A2 The IR-250 is designed to attenuate the light emitted by the

[0032] More specifically, in the wavelength range in which light is transmitted, WR VIS is 400-650 nm, WR NIR is 700-750 nm, and WR SWIR is 925~1300nm. VIS is the wavelength range of visible light, and WR VIS is the wavelength range of NIR fluorescence of MB, and WR SWIR is the wavelength range of SWIR fluorescence of ICG.

[0033] In the wavelength range where light is attenuated, WR A1 is 650-700 nm (OD (optical density) > 6), and WR A2 750-925nm (OD>6 at 790-812nm). A1 includes the wavelength range of the MB excitation light, and WR A2 includes the wavelength range of the excitation light of ICG and the wavelength range of the NIR fluorescence of ICG.

[0034] In this manner, the notch filter 20 is configured to attenuate light in a first attenuation wavelength range that is shorter than the first wavelength range, and light in a second attenuation wavelength range that is longer than the first wavelength range and shorter than the second wavelength range. The notch filter 20 is also configured to transmit visible light from the light from the object of observation on the shorter wavelength side than the first attenuation wavelength range.

[0035] Furthermore, the notch filter 20 attenuates light in a wavelength range longer than the wavelength of the fluorescence in the first wavelength range, where the intensity of the fluorescence in the first wavelength range is greater than the intensity of the fluorescence in the second wavelength range. This is advantageous from the viewpoint of improving the detection accuracy of the fluorescence in the second wavelength range, since the influence of the fluorescence in the first wavelength range on the detection of the fluorescence in the second wavelength range can be reduced.

[0036] Furthermore, the notch filter 20 is detachably disposed in the optical path of the fluorescence in the image forming apparatus 1. Therefore, it is easily detachable and replaceable in the image forming apparatus 1. Therefore, optical filters corresponding to various fluorescent substances or different fluorescence wavelength ranges can be appropriately adopted, which is advantageous from the viewpoint of enhancing the versatility of the image forming apparatus 1. Note that "detachable" includes not only cases where the user can easily attach and detach it, but also cases where the user cannot easily disassemble it. In the latter case, for example, a mechanism for attaching and detaching the notch filter 20 is provided inside a waterproof mechanism or a protective cover.

[0037] [Imaging section] The imaging unit 30 includes an imaging lens 31 , a beam splitter 32 , a near-infrared sensor 33 , a short-wave infrared sensor 34 , and a visible light sensor 35 .

[0038] The imaging lens 31 is an optical element that focuses the light (NIR fluorescence, SWIR fluorescence, and visible light) from the notch filter 20 so as to form an image on each sensor. The imaging lens 31 is a lens that can handle light ranging from visible light to short wave infrared, and its transmission band is, for example, 400 to 1700 nm.

[0039] Like the objective lens described above, the imaging lens 31 is also subjected to focus shift correction. From the viewpoint of effectively correcting the aberration of the image detected by each sensor, it is preferable that the imaging lens 31 satisfies the following formula (2). In the following formula, "BF_1600nm" represents the back focus of the entire optical system at 1600nm. |BF_550nm-BF_1600nm|<0.05mm (2)

[0040] Since the imaging lens 31 is focus-shift corrected, it is not necessary to adjust the positions of the three types of image sensors in the imaging unit 30. Therefore, it is more preferable that the imaging lens 31 is focus-shift corrected from the viewpoint of simplifying the optical design of the imaging unit 30. For example, the imaging lens 31 can be a focus-shift corrected lens with a focal length of 24 mm, an F-number of 5.5, BF_550 nm of -0.015 mm, BF_850 nm of 0.008 mm, and BF_1600 nm of 0.028 mm.

[0041] Beam splitter 32 is a beam splitter that splits the light in the wavelength range of the transmitted light set by notch filter 20 into different directions, and is, for example, a cubic beam splitter having two types of optical thin films that are perpendicular to each other. Of the light that has passed through notch filter 20 and imaging lens 31, beam splitter 32 guides visible light (400 to 650 nm) in one direction perpendicular to the incident direction, guides shortwave infrared (925 to 1700 nm) in the other direction perpendicular to the incident direction, and transmits (straight-travels) near infrared (700 to 880 nm).

[0042] The positions on the optical path of the near-infrared sensor 33, the shortwave infrared sensor 34 and the visible light sensor 35 are adjusted so that the focus position is the sensor position (image plane) according to the wavelength of each component of the light (VIS light, NIR light and SWIR light) received by each sensor.

[0043] The near-infrared sensor 33 is an imaging element that exposes incident light and outputs an image signal obtained by photoelectrically converting the exposed light, and is a monochrome imaging element that has sensitivity to near-infrared light. The near-infrared sensor 33 is an image sensor that has sensitivity in a wavelength range of, for example, 400 to 1000 nm. The near-infrared sensor 33 outputs a signal of an image of the received NIR light (NIR image).

[0044] The shortwave infrared sensor 34 is an image sensor that is sensitive to shortwave infrared light, and outputs an image signal obtained by photoelectrically converting the exposed light. The shortwave infrared sensor 34 is a monochrome image sensor that is sensitive to shortwave infrared light. The shortwave infrared sensor 34 is an image sensor that is sensitive to a wavelength range of, for example, 400 to 1700 nm. The shortwave infrared sensor 34 outputs a signal of the received SWIR light image (SWIR image).

[0045] The visible light sensor 35 is an imaging element that exposes incident light and outputs an image signal obtained by photoelectrically converting the exposed light, and is an imaging element having sensitivity in the wavelength range of visible light. The visible light sensor 35 is an image sensor having sensitivity in the wavelength range of, for example, 400 to 1000 nm. The visible light sensor 35 has color filters of the three primary colors red (R), green (G) and blue (B), or cyan (C), magenta (M) and yellow (Y) arranged in a Bayer array or honeycomb array on the imaging surface. The visible light sensor 35 outputs a signal of the image (VIS image) of the received visible light.

[0046] In this manner, the imaging section 30 is configured to capture each of the NIR fluorescence image and the SWIR fluorescence image transmitted through the notch filter 20. In addition, the imaging section 30 is configured to further capture a visible light image transmitted through the notch filter 20.

[0047] [Image processing section] The image processing unit 40 superimposes the NIR image and the SWIR image captured by the imaging unit 30. In addition, the image processing unit 40 further superimposes the VIS image captured by the imaging unit 30 on the NIR image and the SWIR image. An example of the functional configuration of the image processing unit 40 is shown in FIG.

[0048] The image processing section 40 includes a fluorescent light image processing section 41 , a visible light image processing section 42 , and an image synthesis section 43 .

[0049] The fluorescence image processing unit 41 receives the signal of the NIR image (image of the NIR fluorescence of MB) from the near-infrared sensor 33 and the signal of the SWIR image (image of the SWIR fluorescence of ICG) from the short-wave infrared sensor 34. The fluorescence image processing unit 41 performs predetermined image processing suitable for the fluorescence image on the input fluorescence image signal and outputs it. For example, the fluorescence image processing unit 41 processes the NIR image and the SWIR image into images of different colors that can be distinguished.

[0050] The visible light image processing unit 42 receives the signal of the VIS image (image of the visible light of the observation target) from the visible light sensor 35. The visible light image processing unit 42 performs predetermined image processing suitable for the visible light image on the input visible light image signal and outputs it.

[0051] The image composition unit 43 receives the processed fluorescence image signal from the fluorescence image processing unit 41 and the processed visible light image signal from the visible light image processing unit 42. The image composition unit 43 performs a process of, for example, composing the processed fluorescence image with the processed visible light image.

[0052] The image processing unit 40 transmits the signal of the composite image composed by the image composition unit 43 to the monitor 50. The monitor 50 displays the image input from the image processing unit 40. The monitor 50 is, for example, a display device such as a liquid crystal display (LCD).

[0053] [Image Formation] FIG. 5 is a diagram showing a photograph of an example of the NIR image obtained by the image forming apparatus 1, and FIG. 6 is a diagram showing a photograph of an example of the SWIR image obtained by the image forming apparatus 1. The photographs in FIGS. 5 and 6 are both photographs of images when observing an observation target example in which a glass tube (MB tube) filled with an MB solution and a glass tube (ICG tube) filled with an ICG solution are arranged side by side and covered with 1.5 mm thick roast ham on top. The observation target example simulates a biological tissue in which a fluorescent substance has settled in blood vessels that are close to each other but different. Also, the arrows in the figure indicate the positions of the images of the test specimens.

[0054] The excitation light source 10 irradiates the MB excitation light, the ICG excitation light, and visible light from the first laser and the second laser, respectively, to the observation object substantially simultaneously. Note that "irradiation substantially simultaneously" means not only the case where the light irradiation periods completely coincide, but also the case where the light irradiation periods partially overlap.

[0055] When the first laser (light with a wavelength of 660 nm) is irradiated onto the example of the observation subject, the excitation efficiency of MB is maximized, and the MB tubules emit near-infrared fluorescence with a maximum fluorescence wavelength of about 690 nm. In addition, ICG, which has a maximum excitation wavelength of about 808 nm, is also excited by irradiation with the excitation light with a wavelength of 660 nm, so that the ICG tubules also emit near-infrared fluorescence with a maximum fluorescence wavelength of about 835 nm. In this way, by irradiating the example of the observation subject with the first laser, the near-infrared sensor 33 detects a signal of an NIR image including an image of the MB tubules and an image of the ICG tubules, as shown in FIG. 5. As the NIR image, an NIR image as shown in FIG. 5 is obtained.

[0056] When the observation target example is irradiated with the second laser (light with a wavelength of 808 nm), the excitation light rate of ICG is maximized, and the ICG tubules emit SWIR fluorescence as well as NIR fluorescence. On the other hand, MB is not substantially excited in the wavelength range of the second laser. Therefore, the shortwave infrared sensor 34 detects the signal of the SWIR image of the ICG tubules as shown in FIG. 6.

[0057] By performing a process of superimposing the NIR image signal and the SWIR image signal in the image synthesis unit 43, a fluorescent synthesis image signal in which both the MB tubule image and the ICG tubule image are clearly displayed can be obtained.

[0058] On the other hand, when an observation target example is irradiated with visible light (e.g., light with a wavelength of 400 to 650 nm) from a visible light source, a color VIS image signal of the observation target example is detected by the visible light sensor 35. By performing a process in which the image synthesis unit 43 further superimposes the signal of the aforementioned fluorescent synthetic image on the signal of the VIS image, a synthetic image signal in which the NIR fluorescent image of the MB tubules and the SWIR fluorescent image of the ICG tubules are superimposed on the color image of the observation target example is obtained.

[0059] Therefore, when MB is used as a fluorescent labeling reagent for liver segments and ICG is used as a fluorescent labeling reagent for liver tumors, and excitation light is irradiated to the entire liver by the image forming device 1, an image as shown in FIG. 7 is formed. FIG. 7 is a diagram showing an example of a composite image formed by the image forming device 1. For example, when excitation light is irradiated to the entire liver during surgery, a composite image in which an NIR fluorescent image of MB spreading in the liver segments and a SWIR fluorescent image of ICG are superimposed on a color VIS image of the entire liver is displayed on the monitor 50 in real time. The NIR fluorescent image of MB is, for example, a light green image in which the VIS image is visible and easily distinguishable from the VIS image. The SWIR fluorescence image of ICG is, for example, a blue to purple image that is visible as a VIS image and easily distinguishable from the NIR fluorescence image of MB.

[0060] The color display of the VIS image can be adjusted by a color filter arranged in the visible light sensor 35. The colors of the NIR fluorescence image of MB and the SWIR fluorescence image of ICG can be arbitrarily set by the fluorescence image processor 41. Thus, according to the image forming device 1, even if two types of fluorescent substances having overlapping fluorescence wavelength ranges are used, each of the sites marked with each fluorescent substance can be displayed clearly and identifiably in biological imaging in which the sites detected by fluorescence may overlap. The image forming device 1 can be used in open surgery, but can also be applied to laparoscopy in laparoscopic surgery.

[0061] In the embodiment of the present invention, an example of a rigid endoscope system such as a laparoscope is shown, but it is naturally possible to apply the present invention to other than endoscopes. By removing the irradiation probe 11 in FIG. 1, it can also be used as an endoscope. Considering that the notch filter can be easily attached and detached to the optical system of the image forming device and can be used with a plurality of fluorescent reagents, as in the present invention, it is more preferable to use it as an endoscope than an endoscope, which requires sterilization of the entire device and places importance on airtightness. When applied to an endoscope, the attachment and detachment part of the notch filter can be sealed by sealing or the like to improve airtightness and prevent foreign matter from entering the inside, so that it can be suitably used as an endoscope.

[0062] [Embodiment 2] Other embodiments of the present invention will be described below. For the sake of convenience, in the following embodiments, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0063] This embodiment is an embodiment for using MB as the first fluorescent substance and FD-1080 as the second fluorescent substance. The image forming apparatus of this embodiment is substantially the same as that of the above-mentioned embodiment 1, except that the wavelength of the light of the second laser in the excitation light source and the transmitted light wavelength range and the attenuated light wavelength range in the notch filter are different.

[0064] FIG. 8 (Source: Figure 1b of "J-aggregates of Cyanine Dye for NIR-II In-vivo Dynamic Vascular Imaging Beyond 1500 nm", Journal of the American Chemical Society, November 2019) shows the excitation spectrum (solid line) and fluorescence spectrum (dotted line) of FD-1080, which is the second fluorescent substance in this embodiment. FD-1080 is a type of cyanine dye. The maximum excitation wavelength of FD-1080 is 1064 nm, and the wavelength range of the fluorescence wavelength is 900 to 1400 nm.

[0065] The second laser in this embodiment is a laser that generates light with a wavelength of 1064 nm.

[0066] The transmission wavelength range and attenuation wavelength range of the notch filter in this embodiment are shown in FIG. VIS is 400-650 nm, WR NIR is 700-750 nm, and WR SWIR In addition, in the wavelength range in which the notch filter attenuates light, WR A1 is 650-700 nm, WR A2 is 750~1100nm.

[0067] In this embodiment, a composite image can be formed in which an image of NIR fluorescence from MB and an image of SWIR fluorescence from FD-1080 are superimposed on a color VIS image. Thus, according to this embodiment, similar to the above-mentioned embodiment 1, the images of the fluorescence of each of the two fluorescent substances can be detected in a distinguishable manner. Therefore, similar to the above-mentioned embodiment 1, even if two fluorescent substances having overlapping fluorescence wavelength ranges are used, each of the sites labeled with each fluorescent substance can be clearly and distinguishably displayed in biological imaging in which the sites detected by fluorescence may overlap.

[0068] [Embodiment 3] This embodiment is a form for using ICG as the first fluorescent substance and an ONT ceramic probe Y as the second fluorescent substance. The image forming apparatus of this embodiment is substantially the same as that of the above-described embodiment 1, except that the wavelengths of the first and second laser lights in the excitation light source and the transmitted light wavelength range and the attenuated light wavelength range in the notch filter are different.

[0069] FIG. 10 (Source: Figure 2 of "Evaluation of spectroscopic properties of Er(3+) / Yb(3+) / Pr(3+): SrGdGa3O7 crystal for use in mid-infrared lasers", Scientific Reports, September 2015) shows the excitation spectrum and fluorescence spectrum of the OTN ceramic probe Y, which is the second fluorescent material in this embodiment. The OTN ceramic probe Y is a type of rare earth ion-containing ceramic nanoparticle probe. The maximum excitation wavelength of the OTN ceramic probe Y is 980 nm. FIG. 11 (Source: "Figure 4" of the same source) shows the fluorescence spectrum of the OTN ceramic probe Y. The wavelength range of the fluorescence wavelength is 1400 to 1650 nm.

[0070] The first laser in this embodiment is a laser that generates light with a wavelength of 808 nm, and the second laser in this embodiment is a laser that generates light with a wavelength of 980 nm.

[0071] The transmission wavelength range and attenuation wavelength range of the notch filter in this embodiment are shown in FIG. VIS is 400-650 nm, WR NIR is 820-880 nm, WR SWIR In addition, in the wavelength range in which the notch filter attenuates light, WR A1 is 650-820 nm, WR A2 is 880~1400nm.

[0072] In this embodiment, a composite image can be formed in which an image of NIR fluorescence from ICG and an image of SWIR fluorescence from the OTN ceramic probe Y are superimposed on a color VIS image. Thus, according to this embodiment, similar to the above-mentioned embodiment 1, the images of the fluorescence of each of the two fluorescent substances can be detected in a distinguishable manner. Therefore, similar to the above-mentioned embodiment 1, even if two fluorescent substances having overlapping fluorescence wavelength ranges are used, each of the sites labeled with each fluorescent substance can be clearly and distinguishably displayed in biological imaging in which the sites detected by fluorescence may overlap.

[0073] In this embodiment, the notch filter has a WR filter that is longer than the peak wavelength (835 nm) of the NIR fluorescence of ICG and shorter than the peak wavelength (1530 nm) of the SWIR fluorescence of the OTN ceramic probe Y. A2 This attenuates the light from the NIR region of ICG and the SWIR region of the OTN ceramic probe Y, which is advantageous in terms of using these fluorescent reagents effectively.

[0074] [Embodiment 4] This embodiment is a form for using Cy5.5 as the first fluorescent substance and ICG as the second fluorescent substance. The image forming apparatus of this embodiment is substantially the same as that of the above-mentioned embodiment 1, except that the wavelength of the first laser light in the excitation light source and the transmitted light wavelength range and the attenuated light wavelength range in the notch filter are different.

[0075] FIG. 13 (source: https: / / www.lumiprobe.com / p / cy55-nhs-ester) shows the excitation spectrum and fluorescence spectrum of Cy5.5, which is the first fluorescent substance in this embodiment. Cy5.5 is a type of Cy dye-labeled carboxylic acid. The maximum excitation wavelength of Cy5.5 is 685 nm, and the wavelength range of the fluorescence wavelength is 660 to 850 nm.

[0076] The first laser in this embodiment is a laser that generates light with a wavelength of 665 nm.

[0077] The transmission wavelength range and attenuation wavelength range of the notch filter in this embodiment are shown in FIG. VIS is 400-650 nm, WR NIR is 710-750 nm, and WR SWIR In addition, in the wavelength range in which the notch filter attenuates light, WR A1 is 650-710 nm, WR A2 is 750~925nm.

[0078] In this embodiment, a composite image can be formed in which an image of NIR fluorescence of Cy5.5 and an image of SWIR fluorescence of ICG are superimposed on a color VIS image. Thus, according to this embodiment, similar to the above-mentioned embodiment 1, the images of the fluorescence of each of the two fluorescent substances can be detected in a distinguishable manner. Therefore, similar to the above-mentioned embodiment 1, even if two fluorescent substances having overlapping fluorescence wavelength ranges are used, each of the sites labeled with each fluorescent substance can be clearly and distinguishably displayed in biological imaging in which the sites detected by fluorescence may overlap.

[0079] [Embodiment 5] This embodiment is an embodiment for using Cy5.5 as the first fluorescent substance and CH1055 as the second fluorescent substance. The image forming apparatus of this embodiment is substantially the same as that of the above-described embodiment 1, except that the wavelengths of the first and second laser beams in the excitation light source and the transmitted light wavelength range and the attenuated light wavelength range in the notch filter are different.

[0080] FIG. 15 (Source: Fig. 16(B) of "Near-infrared dyes, nanomaterials and proteins", Chinese Chemical Letters 30, (2019), 1856-1882) shows the excitation spectrum and fluorescence spectrum of CH1055, which is the second fluorescent substance in this embodiment. CH1055 is a type of donor-acceptor-donor (DAD) type dye. The maximum excitation wavelength of CH1055 is 750 nm, and the wavelength range of the fluorescence wavelength is 900 to 1350 nm.

[0081] The first laser in this embodiment is a laser that generates light with a wavelength of 665 nm, and the second laser in this embodiment is a laser that generates light with a wavelength of 780 nm.

[0082] The transmission wavelength range and attenuation wavelength range of the notch filter in this embodiment are shown in FIG. VIS is 400-650 nm, and WR NIR is 700-750 nm, and WR SWIR In addition, in the wavelength range in which the notch filter attenuates light, WR A1 is 650-700 nm, WR A2 is 750~1000nm.

[0083] In this embodiment, a composite image can be formed in which an image of NIR fluorescence of Cy5.5 and an image of SWIR fluorescence of CH1055 are superimposed on a color VIS image. Thus, according to this embodiment, similar to the above-mentioned embodiment 1, the images of the fluorescence of each of the two fluorescent substances can be detected in a distinguishable manner. Therefore, similar to the above-mentioned embodiment 1, even if two fluorescent substances having overlapping fluorescence wavelength ranges are used, each of the sites labeled with each fluorescent substance can be clearly and distinguishably displayed in biological imaging in which the sites detected by fluorescence may overlap.

[0084] [Embodiment 6] This embodiment is a form for using CH1055 as the first fluorescent material and OTN ceramic probe Y as the second fluorescent material. The image forming apparatus of this embodiment is different in the wavelengths of the first and second lasers in the excitation light source, the transmitted light wavelength range and the attenuated light wavelength range in the notch filter, and the beam splitter. Also, as shown in FIG. 17, the image forming apparatus 6 according to this embodiment has a shortwave infrared sensor 64 instead of the near-infrared sensor 33, and is otherwise substantially the same as the image forming apparatus 1 of the above-described embodiment 1. The shortwave infrared sensor 64 is a monochrome image sensor sensitive to shortwave infrared like the shortwave infrared sensor 34, and the shortwave infrared sensors 34 and 64 are both image sensors sensitive to a wavelength range of, for example, 400 to 1700 nm.

[0085] The first laser in this embodiment is a laser that generates light with a wavelength of 750 nm, and the second laser in this embodiment is a laser that generates light with a wavelength of 980 nm. With reference to the quantum efficiency of the OTN ceramic probe Y and the quantum yield of CH1055, the output of the first laser is set to be larger than the output of the second laser so as to reduce the intensity difference between them.

[0086] The transmission wavelength range and attenuation wavelength range of the notch filter in this embodiment are shown in Figure 18. The notch filter has wavelength ranges that transmit light in the visible light wavelength range and the short wave infrared wavelength range, and is designed to attenuate light in the wavelength range between them. In other words, the visible light wavelength range WR VIS The wavelength range of the shortwave infrared light that is transmitted is 400 to 650 nm. SWIR The wavelength range WR for attenuating light in the notch filter is 1000 to 1650 nm. A is 650~1000nm. A The excitation wavelength range of the CH1055 (600 to 900 nm) and the excitation wavelength range of the OTN ceramic probe Y (920 to 1020 nm) are substantially included. SWIRThe fluorescence wavelength range of CH1055 (900 to 1350 nm) and the fluorescence wavelength range of OTN ceramic probe Y (1400 to 1650 nm) are substantially included.

[0087] Beam splitter 62 is a beam splitter that splits the light in the wavelength range of the transmitted light set by notch filter 20 into different directions, and is, for example, a cubic beam splitter having two types of optical thin films that are perpendicular to each other. Of the light that has passed through notch filter 20 and imaging lens 31, beam splitter 62 guides visible light (400 to 650 nm) in one direction perpendicular to the incident direction, guides shortwave infrared (900 to 1350 nm) in the other direction perpendicular to the incident direction, and transmits (travels straight) shortwave infrared (1400 to 1700 nm).

[0088] In this embodiment, an image of SWIR fluorescence from the OTN ceramic probe Y is captured by the shortwave infrared sensor 34, an image of SWIR fluorescence from CH1055 is captured by the shortwave infrared sensor 64, and a composite image can be formed in which these SWIR fluorescence images are superimposed on a color VIS image. Thus, according to this embodiment, similar to the above-mentioned embodiment 1, it is possible to detect the respective fluorescence images of two types of fluorescent substances in a distinguishable manner. Therefore, similar to the above-mentioned embodiment 1, even if two types of fluorescent substances having overlapping fluorescence wavelength ranges are used, it is possible to clearly and distinguishably display each of the sites labeled with each fluorescent substance in biological imaging in which the sites detected by fluorescence may overlap.

[0089] [Embodiment 7] The overall configuration of an image forming apparatus according to a seventh embodiment of the present invention is shown in Fig. 19. This embodiment is an embodiment in which the present invention is applied to an endoscope. As shown in Fig. 19, the image forming apparatus 7 does not have an irradiation probe 11, and is configured substantially the same as that of the first embodiment described above, except that the positions of the excitation light source, imaging lens, and notch filter of the imaging unit are different.

[0090] The image forming device 7 has an excitation light source, an imaging unit 70, and an image processing unit 40. The excitation light source has excitation light diffusion lenses 81 and 82, a connection cable 85 connected to the excitation light diffusion lenses 81 and 82, and an excitation light adjustment unit 86 connected to the connection cable 85. The excitation light adjustment unit 86 includes a first laser and a second laser. The first laser outputs light having a wavelength of 660 nm, which is the excitation light for MB, and the second laser outputs light having a wavelength of 808 nm, which is the excitation light for ICG. The light output by the first laser is transmitted to the excitation light diffusion lens 81 via the connection cable 85, and the light output by the second laser is transmitted to the excitation light diffusion lens 82 via the connection cable 85. The excitation light diffusion lenses 81 and 82 are each supported by a frame 84 facing the target, and are configured to irradiate the target with light having a wavelength of 660 nm and light having a wavelength of 808 nm.

[0091] The imaging unit 70 is supported by a frame 84 facing the object. A functional configuration of the image forming device 7 according to this embodiment is shown in FIG. 20. As shown in FIG. 20, the imaging unit 70 has an imaging lens 31, a notch filter 20, and a beam splitter 32 from the object (target) side, and has a near-infrared sensor 33, a short-wave infrared sensor 34, and a visible light sensor 35 on the image plane side of the beam splitter 32. Each sensor is connected to the image processing unit 40. In this manner, the notch filter 20 is disposed in the imaging unit 70.

[0092] A plan view of the imaging unit 70 is shown in FIG. 21. As shown in FIG. 21, the imaging unit 70 has a first adapter 71, a second adapter 72, and screws 73 for fixing them. The first adapter 71 has a lens barrel 711, which has a plurality of lenses on an optical axis OA. In this embodiment, the imaging lens 31 is configured by these plurality of lenses. The second adapter 72 has a notch filter 20, a beam splitter 32, a near-infrared sensor 33, a short-wave infrared sensor 34, and a visible light sensor 35. The notch filter 20 is fitted into a recess 721 formed in the second adapter 72 and is disposed on the optical axis OA.

[0093] The beam splitter 32 is disposed on one optical axis (optical axis OA) and is composed of three prisms 321, 322, and 323 made of the same glass material. All of the three prisms 321, 322, and 323 are trapezoidal prisms with one end of a right-angle prism cut off.

[0094] The prism 321 includes a total reflection surface 3211 and a first dielectric multilayer film 3212. The total reflection surface 3211 is a surface that corresponds to the hypotenuse of the trapezoid of the prism 321, and the prism 321 is arranged so that the total reflection surface 3211 is perpendicular to the optical axis OA. The first dielectric multilayer film 3212 is formed on a surface that corresponds to the base of the trapezoid of the prism 321, and is a film that reflects visible light in the incident light. A visible light sensor 35 is arranged on the image surface side of the prism 321, and a short pass filter 3213 is arranged between the prism 321 and the visible light sensor 35. The short pass filter 3213 is an optical element that transmits light with a wavelength equal to or less than the boundary of 650 nm, which is the boundary between visible light, near infrared light, and short wave infrared light, (i.e., visible light components) and blocks the transmission of light with a wavelength exceeding the boundary (i.e., near infrared light and short wave infrared light components).

[0095] The prism 322 includes a total reflection surface 3221. The total reflection surface 3221 is a surface that corresponds to the hypotenuse of the trapezoid of the prism 322, and the prism 322 is disposed so that the total reflection surface 3221 faces the first dielectric multilayer film 3212 of the prism 321 and intersects with the optical axis OA. Between the prisms 321 and 322, a gap (air gap) is provided to fully realize total reflection in the prism 322. The distance of the air gap is, for example, 5 to 15 μm. A shortwave infrared sensor 34 is disposed on the image surface side of the prism 322, and a longpass filter 3222 is disposed between the prism 322 and the shortwave infrared sensor 34. The longpass filter 3222 has the opposite characteristics to the shortpass filter 3213. In other words, the long-pass filter 3222 is an optical element that transmits light with wavelengths above the boundary of 925 nm, which is the boundary between visible light, near-infrared light, and short-wave infrared light (i.e., short-wave infrared light components), and blocks the transmission of light with wavelengths below the boundary (i.e., visible light and near-infrared light components).

[0096] The prism 323 includes a second dielectric multilayer film 3231. The second dielectric multilayer film 3231 is formed on a surface of the prism 323 that corresponds to the hypotenuse of the trapezoid, and the prism 323 is disposed such that the surface having the second dielectric multilayer film 3231 abuts against the surface of the prism 322 that corresponds to the base of the trapezoid and intersects with the optical axis OA. The second dielectric multilayer film 3231 is a film that reflects short-wave infrared light in the incident light. A near-infrared sensor 33 is disposed on the image surface side of the prism 323, and a band-pass filter 3232 is disposed between the prism 323 and the near-infrared sensor 33. The band-pass filter 3232 is an optical element that selectively transmits near-infrared light components of 700 to 880 nm and blocks the transmission of light of other wavelengths (i.e., visible light and short-wave infrared light components).

[0097] The light that reaches prism 321 passes through total reflection surface 3211 and reaches first dielectric multilayer film 3212. The visible light components of the reaching light are reflected by first dielectric multilayer film 3212, then reflected by total reflection surface 3211, and reach and detect at visible light sensor 35 via short pass filter 3213. With this configuration, near-infrared light and shortwave infrared light components are prevented from being mixed into the visible light components that reach visible light sensor 35, and the color reproducibility of the visible light image is improved.

[0098] The near-infrared light and shortwave infrared light components transmitted through prism 321 are incident on prism 322. The shortwave infrared light component of the incident light components is reflected by second dielectric multilayer film 3231 forming the interface between prisms 322 and 323, then reflected by total reflection surface 3221, and reaches and is detected by shortwave infrared sensor 34 via longpass filter 3222. With this configuration, visible light and near-infrared light components are prevented from being mixed into the shortwave infrared light components reaching shortwave infrared sensor 34.

[0099] The near-infrared light components transmitted through the second dielectric multilayer film 3231 and incident on the prism 323 reach and are detected by the near-infrared sensor 33 via the bandpass filter 3232. With this configuration, the near-infrared light components reaching the near-infrared sensor 33 are prevented from being mixed with visible light and shortwave infrared light components.

[0100] 23 is a schematic cross-sectional view showing the disassembled state of the imaging unit 70. By loosening and removing the screws 73, the imaging unit 70 can be disassembled into a first adapter 71 and a second adapter 72. The notch filter 20 is simply fitted into a recess 721, and can therefore be easily attached, detached, and replaced.

[0101] When the image forming device 7 is in use, light having a wavelength of 660 nm is irradiated toward the target from the excitation light diffusing lens 81, light having a wavelength of 808 nm is irradiated toward the target from the excitation light diffusing lens 82, and visible light is illuminated from the room light. The excitation light and visible light substantially or completely overlap and illuminate the shooting range of the imaging lens 31 on the target.

[0102] The return light from the object passes through the imaging lens 31, reaches the notch filter 20, and passes through the beam splitter 32. The notch filter 20 transmits the visible light, near infrared light, and short wave infrared light components of the return light. As described above, the beam splitter 32 allows the visible light component of the return light to reach the visible light sensor 35, the short wave infrared light component of the return light to reach the short wave infrared sensor 34, and the near infrared light component of the return light to reach the near infrared sensor 33. The image data acquired by the imaging unit 70 is processed by the image processing unit 40 and provided to the user of the image forming device 7 by being displayed on the monitor 50, for example.

[0103] In this embodiment, the image forming device 7 does not include a light source of visible light, because in the case of an endoscope, lighting such as a room light or a surgical shadowless lamp that illuminates an object such as a person to be examined is usually present and can be used as a light source of visible light. Therefore, the image forming device 7 may further include a light source of visible light as an excitation light source. The light source of visible light may be fixed to the lens barrel 711 of the imaging unit 70, for example, so as to face in a direction along the optical axis OA.

[0104] Furthermore, in this embodiment, the position of the notch filter 20 in the optical system of this embodiment may be any position on the object side of the beam splitter 32. For example, the notch filter 20 may be disposed on the most object side of the optical system, or may be disposed at the tip of a lens barrel while being held by a lens adapter. This type of configuration is also preferable from the viewpoint of easily attaching, detaching, and replacing the notch filter 20.

[0105] In the embodiment of the present invention, the arrangement of the sensors 33, 34, and 35 in the imaging section can be appropriately changed within a range where the effects of the present invention can be obtained. For example, in the seventh embodiment, a dielectric multilayer film that reflects a shortwave infrared light component and transmits a visible light component and a near-infrared light component may be formed on the prism 321, and a dielectric multilayer film that reflects a near-infrared light component and transmits a visible light component may be formed on the interface between the prisms 322 and 323, respectively, a sensor that detects the component reflected by the dielectric multilayer film may be disposed on the image plane side of the dielectric multilayer film, and a visible light sensor 35 may be disposed on the image plane side of the beam splitter 32 on the optical axis OA. In this way, in the embodiment of the present invention, the arrangement of the sensors can be appropriately changed by appropriately designing the prism shape and the dielectric multilayer film in the beam splitter.

[0106] In addition, in the seventh embodiment, the beam splitter 32 is configured by arranging three prisms each having a dielectric multilayer film appropriately, but the form of the beam splitter is not limited in the embodiments of the present invention. That is, the beam splitter 32 in the seventh embodiment may be applied to other embodiments, and the cubic beam splitter in other embodiments may be applied to the seventh embodiment.

[0107] Other Embodiments The image forming apparatus according to the present invention can be applied to bioimaging using two specific fluorescent substances applicable to a living body by appropriately setting the optical characteristics of the excitation light source and the notch filter. The two specific fluorescent substances are two fluorescent substances that have fluorescent wavelength ranges in which only one of them is detected in the near infrared and shortwave infrared. Examples of fluorescent substances that are other than the fluorescent substances shown in the above-mentioned embodiments and that can be used as the first fluorescent substance or the second fluorescent substance when the present invention is applied to bioimaging are shown in Table 3 below.

[0108] [Table 3]

[0109] [Modifications] In the above-described embodiment, the excitation light source and the visible light source may be constituted by optical fibers optically connected to these light sources and guiding the light emitted from the light sources (excitation light and visible light) toward the observation target. This configuration is advantageous from the viewpoint of miniaturization of the light source.

[0110] Further, the excitation light source may be a device that irradiates light of a single wavelength included in both the excitation wavelength range of the first fluorescent substance and the excitation wavelength range of the second fluorescent substance as excitation light for both fluorescent substances. This configuration is advantageous from the viewpoint of simplification of the excitation light source.

[0111] Further, the excitation light source may irradiate excitation light having a wavelength other than the maximum wavelength of the absorption wavelength range at at least one of the excitation wavelength of the first fluorescent substance and the excitation wavelength of the second fluorescent substance. This configuration is advantageous from the viewpoint of creating a sufficient intensity difference between the fluorescence of the first fluorescent substance and the second fluorescent substance.

[0112] The optical filter unit may be constituted by a combination of two or more band-pass filters. This configuration is advantageous from the viewpoint of being able to sequentially construct an optical filter unit according to various combinations of the first fluorescent substance and the second fluorescent substance.

[0113] The imaging unit may have a separation transmission filter unit that transmits light in two or more different wavelength ranges at specific sites. Examples of the separation transmission filter unit include an NIR-SWIR filter in which an NIR filter unit that transmits near-infrared light and an SWIR filter unit that transmits short-wave infrared light are arranged in a checkerboard pattern, and a VIS-NIR-SWIR filter in which, in addition to the NIR filter unit and the SWIR filter unit, a VIS filter unit that transmits visible light is arranged in a checkerboard pattern. This configuration is advantageous from the viewpoint of simplifying the configuration of the imaging unit because images of light in two or more different wavelength ranges can be captured by a single imaging device.

[0114] The shortwave infrared sensor may include a sensor having sensitivity from near infrared to shortwave infrared. Such a shortwave infrared sensor may be used as shortwave infrared sensor 34 instead of near infrared sensor 33. This configuration is advantageous from the viewpoint of simplifying the type of sensor and configuring an imaging unit that can also handle a combination of fluorescent substances in which both the fluorescence of the first fluorescent substance and the fluorescence of the second fluorescent substance are shortwave infrared fluorescence.

[0115] The functions of the image processing unit 40 in the embodiment of the present invention can be realized by a program for causing a computer to function as the processing unit, and a program for causing a computer to function as each control block of the processing unit.

[0116] In this case, the processing unit includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0117] The program may be non-transiently recorded on one or more computer-readable recording media. The recording media may or may not be included in the processing unit. In the latter case, the program may be supplied to the processing unit via any wired or wireless transmission medium.

[0118] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0119] Furthermore, each process described in the above embodiment may be executed by AI (Artificial Intelligence). In this case, the AI ​​may be operated by the control device or may be operated by another device (such as an edge computer or a cloud server).

[0120] The image forming apparatus of the present invention may further include a diagnosis unit that diagnoses tissue from the created composite image. The diagnosis unit may include a judgment unit that judges the image using an image judgment model that has been trained using the composite image data as teacher data. Examples of the image judgment model include neural networks and support vector machines. Examples of the neural network include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected neural networks.

[0121] The image judgment model can be trained by referring to training data. The training data includes image data of a synthetic image and at least one piece of information (such as a disease in the part) related to a part in the image corresponding to the image data. The image judgment model can be trained by preparing a sufficient amount of the training data (image data and information on the part corresponding to the image data), training the neural network, and determining the weight of the path for each image data. Examples of algorithms for training the image judgment model include backpropagation and ID3.

[0122] The image determination model may be a model other than that based on machine learning. For example, the image determination model may be a regression model in which the image data is used as a response variable and information on the suitability of the image data is used as an explanatory variable.

[0123] The image processor 40 in the embodiment of the present invention may perform a process for creating a composite image by boundary restoration processing in processing the fluorescent image. Even in a bright image (shortwave NIR image), there should be a clear boundary between different tissues, but because the image is blurred, the boundary between these tissues becomes unclear. On the other hand, although the signal in a dark image (SWIR image) is small, the boundary between tissues is clear. By extracting the boundary in the dark image (SWIR image) and superimposing the obtained boundary on the blurred image, it is possible to reproduce the clear boundary between tissues that was originally present in the image.

[0124] For example, it is possible to use a dark image (SWIR image) with clear boundaries between tissues and apply top-hat transformation to this image. In this case, the image processor 40 selects a range of the SWIR image to create an original image. The entire image can be divided into two regions according to the magnitude of the signal value of the tissue. For example, it can be divided into a region of tissue emitting ICG fluorescence (bright region) and a region of parts outside the tissue (dark region).

[0125] Next, the image processor 40 creates an image by expanding the original image. For example, the image processor 40 copies the pixel values ​​of the pixels adjacent to the boundary of the bright region to the pixels outside the boundary that are adjacent to the boundary in the up, down, left, and right directions. As a result, a region (expanded region) expanded by one pixel from the bright region along the boundary is created.

[0126] The image processor 40 then creates an image of the boundary line by subtracting the original image before and after the dilation. The bright areas are subtracted from the dilated area, leaving only the area expanded by the dilation. In this way, a boundary area (border region) corresponding to the tissue boundary line is created.

[0127] Next, the image processing unit 40 superimposes an image of the boundary line on the shortwave NIR image. For example, the image processing unit 40 superimposes an image in which the pixel values ​​of the boundary area are multiplied by a constant and the boundary area is emphasized on the shortwave NIR image.

[0128] In this way, by applying a top-hat transform to a dark image (SWIR image), the boundary signal obtained is multiplied by a constant and then added to a bright image (shortwave NIR image) with unclear tissue boundaries, the clear boundaries between tissues in the dark image (SWIR image) are reflected in the bright image (shortwave NIR image). As a result, a bright image (synthetic image) with clear boundaries between tissues is synthesized.

[0129] In this case, a bright image with clear boundaries between tissues (a composite image) can be synthesized by subtracting an image with emphasized boundaries from a short-wave NIR image.

[0130] In addition, the image processing unit 40 in the embodiment of the present invention may perform processing to create a composite image by performing processing to enhance edges in processing the fluorescent image. For example, an image of tissue emitting ICG fluorescence is used as the original image.

[0131] First, the image processing unit 40 performs a two-dimensional wavelet transform on the SWIR image.

[0132] Next, the image processor 40 ranks the high-pass components of the SWIR image. Edge components appear in the high-pass components. Thus, when a dark image (SWIR image) is subjected to wavelet transform, the components of the high-pass filter corresponding to edges have large values.

[0133] The image processor 40 then selects the top ranked high-pass image components of the SWIR image and records the components with the largest values ​​in this high-pass filter.

[0134] Next, the image processing unit 40 performs two-dimensional wavelet transform on the shortwave side NIR image. For example, the wavelet transform is applied to a bright image (shortwave side NIR image). In the bright image (shortwave side NIR image) after the wavelet transform, the components corresponding to edges become small and are buried in components other than edges because the shortwave side NIR image is a high-brightness low-resolution image.

[0135] Next, the image processing unit 40 multiplies the selected high-pass components (components with large values ​​recorded in the previous processing) of the SWIR image by a constant in the wavelet transformed shortwave side NIR image. In this way, the corresponding components in the wavelet transformed shortwave side NIR image are restored using the wavelet transformed components of the SWIR image. As a result, a wavelet transformed shortwave side NIR image with clear edges is created.

[0136] Next, the image processor 40 performs an inverse wavelet transform on the wavelet transformed, edge-clarified short-wave NIR image, resulting in a composite image in which the clear edges of the dark image (SWIR image) are reflected in the bright image (shorter-wave NIR image).

[0137] 〔summary〕 A first aspect of the present invention is an image forming device having an excitation light source (10) that irradiates an object (object of observation) with excitation light of a first fluorescent substance (e.g., MB) and excitation light of a second fluorescent substance (e.g., ICG), an optical filter section (notch filter 20) that transmits the fluorescence of the first fluorescent substance in a first wavelength range including NIR or SWIR and the fluorescence of the second fluorescent substance in a second wavelength range including SWIR from the light from the object, an imaging section (30) that captures an image of the fluorescence of the first wavelength range and an image of the fluorescence of the second wavelength range that have passed through the optical filter section, and an image processing section (40) that superimposes the image of the fluorescence of the first wavelength range and the image of the fluorescence of the second wavelength range in the imaging section. According to the first aspect, the respective fluorescent images of the two fluorescent substances can be detected in a distinguishable manner.

[0138] A second aspect of the present invention is the first aspect, in which the optical filter unit attenuates light in a specific wavelength range that is longer than the peak wavelength of the fluorescence in the first wavelength range of the first fluorescent substance and shorter than the peak wavelength of the fluorescence in the second wavelength range of the second fluorescent substance. In the second aspect, the attenuated wavelength range is located between the fluorescence in the first wavelength range and the fluorescence in the second wavelength range, so that overlap of the two fluorescent substances is further suppressed. Thus, the second aspect is even more effective from the viewpoint of clarifying the images of the respective fluorescent substances and even more effective from the viewpoint of effectively using both fluorescent substances.

[0139] A third aspect of the present invention is the first or second aspect, in which the optical filter unit attenuates light in a wavelength range longer than the wavelength of the fluorescence at the fluorescence intensity in the first wavelength range where the intensity of the fluorescence in the first wavelength range is greater than the intensity of the fluorescence in the second wavelength range. In the third aspect, between the first and second wavelength ranges, light with a higher fluorescence intensity in the first wavelength range is attenuated by the optical filter unit. Therefore, the third aspect is even more effective from the viewpoint of further reducing the influence of the fluorescence in the first wavelength range on the fluorescence in the second wavelength range and from the viewpoint of increasing the detection accuracy of the fluorescence in the second wavelength range.

[0140] A fourth aspect of the present invention is any one of the first to third aspects, in which the optical filter unit is detachably provided in the image forming apparatus. In the fourth aspect, the optical filter unit can be appropriately replaced, thereby making it possible to deal with various combinations of fluorescent materials. Therefore, the fourth aspect is even more effective in terms of increasing the versatility and convenience of the image forming apparatus.

[0141] A fifth aspect of the present invention is a notch filter in which the optical filter unit in any one of the first to fourth aspects attenuates light in a first attenuation wavelength range shorter than the first wavelength range and light in a second attenuation wavelength range longer than the first wavelength range and shorter than the second wavelength range. In the fifth aspect, the optical filter unit can be configured with a single notch filter, and the notch filter can be easily attached to and detached from the optical system of the image forming apparatus. Thus, the fifth aspect is even more effective from the viewpoint of simplifying the optical filter and from the viewpoint of versatility of the image forming apparatus.

[0142] A sixth aspect of the present invention is any one of the first to fifth aspects, in which the excitation light source independently controls the output of the excitation light for the first fluorescent material and the output of the excitation light for the second fluorescent material to irradiate the target with the excitation light. In the sixth aspect, it is possible to further reduce the difference in the intensity of the fluorescence due to the difference in the optical properties between the fluorescent materials. Thus, the sixth aspect is even more effective in terms of aligning the distinguishability of the images of both fluorescent materials.

[0143] A seventh aspect of the present invention is any of the first to sixth aspects, wherein the image processing section includes a fluorescence image processing section that processes the image of the fluorescence in the first wavelength range and the image of the fluorescence in the second wavelength range into images of different colors. In the seventh aspect, it becomes possible to more easily distinguish between the images of both fluorescence. Thus, the seventh aspect is even more effective in terms of enhancing the distinguishability between the images of both fluorescence.

[0144] An eighth aspect of the present invention is any one of the first to seventh aspects, in which the optical filter unit further transmits visible light from the light from the object, the imaging unit further captures an image of the visible light transmitted through the optical filter unit, and the image processing unit further superimposes the visible light image in the imaging unit on the image of the fluorescence in the first wavelength range and the image of the fluorescence in the second wavelength range. In the eighth aspect, a composite image is obtained in which the images of both fluorescence are superimposed on the image of visible light, so that the positional relationship in the field of view of the sites identified by both fluorescent substances becomes clearer. Thus, the eighth aspect is even more effective in terms of clearly indicating the position of the site identified by the fluorescent substance in the field of view.

[0145] A ninth aspect of the present invention is any one of the first to eighth aspects, in which the image forming apparatus is an endoscope. The ninth aspect is even more effective in terms of being easily applicable to image formation of a living body, such as in vivo imaging.

[0146] According to the present invention, the results of tests that utilize fluorescence in vivo can be clearly displayed. The present invention is expected to contribute to the achievement of, for example, Goal 3 of the Sustainable Development Goals (SDGs) advocated by the United Nations, "Ensure healthy lives and promote well-being for all."

[0147] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in the different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0148] 1, 6, 7 Image forming device 10 Excitation Light Source 11 Irradiation probe 20 Notch Filter 30, 70 Imaging unit 31 Imaging Lens 32, 62 Beam splitter 33 Near-infrared sensor 34, 64 Short wave infrared sensor 35 Visible Light Sensor 40 Image processing section 41 Fluorescence image processing unit 42 Visible light image processing section 43 Image synthesis unit 50 Monitors 71 First Adapter 72 Second Adapter 73 Bis 81, 82 Excitation light diffusion lens 84 slots 85 Connection cable 86 Excitation light adjustment unit 321, 322, 333 Prism 711 Telescope tube 721 Recess 3211, 3221 Total reflection surface 3212 First Dielectric Multilayer Film 3213 Shortpass Filter 3222 Longpass Filter 3231 Second Dielectric Multilayer Film 3232 Bandpass Filter OA optical axis

Claims

1. an excitation light source that irradiates an object with excitation light of a first fluorescent substance and excitation light of a second fluorescent substance; an optical filter unit that transmits fluorescence of the first fluorescent material in a first wavelength range including near infrared or short wave infrared and fluorescence of the second fluorescent material in a second wavelength range including short wave infrared from the light from the object; an imaging unit configured to capture an image of the fluorescence in the first wavelength range and an image of the fluorescence in the second wavelength range transmitted through the optical filter unit; an image processing unit that superimposes an image of the fluorescence in the first wavelength range and an image of the fluorescence in the second wavelength range obtained by the imaging unit; An image forming apparatus comprising:

2. 2. The image forming apparatus according to claim 1, wherein the optical filter unit attenuates light in a specific wavelength range that is longer than a peak wavelength of fluorescence in the first wavelength range of the first fluorescent material and shorter than a peak wavelength of fluorescence in the second wavelength range of the second fluorescent material.

3. 2. The image forming apparatus according to claim 1, wherein the optical filter unit attenuates light in a wavelength range longer than a wavelength of the fluorescence in the first wavelength range, the intensity of the fluorescence in the first wavelength range being greater than an intensity of the fluorescence in the second wavelength range.

4. 2. The image forming apparatus according to claim 1, wherein the optical filter unit is detachably provided.

5. 2. The image forming apparatus according to claim 1, wherein the optical filter unit is a notch filter that attenuates light in a first attenuation wavelength range that is shorter than the first wavelength range, and light in a second attenuation wavelength range that is longer than the first wavelength range and shorter than the second wavelength range.

6. The image forming apparatus according to claim 1 , wherein the excitation light source irradiates the target with the excitation light by independently controlling an output of the excitation light for the first fluorescent material and an output of the excitation light for the second fluorescent material.

7. The image forming apparatus according to claim 1 , wherein the image processing section includes a fluorescent image processing section that processes the image of the fluorescent light in the first wavelength region and the image of the fluorescent light in the second wavelength region into images of different colors.

8. The optical filter unit further transmits visible light from the light from the object, The imaging unit further captures an image of visible light transmitted through the optical filter unit, the image processing unit further superimposes the visible light image from the imaging unit on the fluorescent light image in the first wavelength range and the fluorescent light image in the second wavelength range. The image forming apparatus according to claim 1 .

9. 10. The imaging device of claim 1, which is an exoscope.

Images