Image forming apparatus

Abstract

To provide a novel technology of acquiring an image having both features of a high brightness image and a high resolution image.SOLUTION: An image formation device (1) includes: a light source (10) that irradiates an observation target with excitation light; an imaging part (20) that separates short wave side NIR light and SWIR light from an observation target and receives the pieces of light; and an image processing part (30) that generates a composite image having an image density of the image of the short wave side NIR light and an image of the SWIR light from the image of the pieces of light.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image forming apparatus.

Background Art

[0002] Fluorescence imaging is known as a medical observation system for identifying the presence or absence of tumors and the location of tumors in living tissues. Fluorescence imaging is a technique in which a fluorescent reagent is administered into a living body and specifically accumulated in tumors or the like in the living body, and then the fluorescent reagent is excited by light of a specific wavelength, and the fluorescence emitted by the fluorescent reagent is imaged and displayed. Thus, by detecting the fluorescence in the living body, it becomes possible to grasp the presence or absence and location of tumors.

[0003] As a medical fluorescent reagent, indocyanine green (ICG) is generally used. ICG has the property of being excited by light having a wavelength included in the near-infrared light (NIR) region (750 to 850 nm) with excellent biopermeability and emitting fluorescence having a peak wavelength of about 835 nm included in the NIR region. It is known that the wavelength region of the fluorescence of ICG reaches the short-wavelength infrared (SWIR) region (900 to 1600 nm) on the longer wavelength side. The component in the SWIR region of the fluorescence of ICG is less affected by biological scattering than the component in the NIR region (835 nm). Therefore, observation of deep living bodies more than 1 cm under the skin or acquisition of high-resolution images is expected.

[0004] As a technique for detecting fluorescence in the short-wavelength infrared region, a technique for detecting fluorescence of ICG of 900 nm or more using a SWIR sensor is known (for example, see Patent Document 1). Further, as the above technique, a technique for detecting fluorescence of an inorganic fluorescent substance such as Yb, Nd, or Er using a SWIR sensor is known (for example, see Patent Document 2).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] While the wavelengths near the peak of ICG fluorescence have high brightness, when it typically occurs in deeper parts of the body, such as a few millimeters to 1 cm below the skin, it is easily scattered by biological tissue, resulting in lower resolution of the image of that light.

[0007] On the other hand, ICG fluorescence in the SWIR region is less affected by biological scattering compared to fluorescence at wavelengths near the peak, but its brightness is lower. In addition, water absorption is included in the SWIR region. Therefore, when it occurs in deeper parts of the body, the image of the fluorescence has high resolution, but the contrast is reduced.

[0008] Methods for increasing image contrast include increasing exposure time or excitation light intensity. However, the former also increases the background, and the latter can lead to reflections of excitation light or increased effects of excitation light on living organisms.

[0009] One aspect of the present invention aims to provide a novel technique for acquiring images that possess characteristics of both high-luminance and high-resolution images. [Means for solving the problem]

[0010] To solve the above problems, an image forming apparatus according to one aspect of the present invention includes: an excitation light source for irradiating an object to be observed with excitation light; an imaging unit that separates the light from the object to be observed irradiated with the excitation light into a first infrared light including wavelengths in the short-wave infrared region and a second infrared light including wavelengths in a wavelength region shorter than the short-wave infrared region, and receives each of the lights; and an image processing unit that generates a composite image synthesized from a first image showing the boundary of a specific region corresponding to the first infrared light received by the imaging unit and a second image including the specific region having an image density corresponding to the second infrared light received by the imaging unit. [Effects of the Invention]

[0011] According to one aspect of the present invention, a novel technique can be provided for acquiring images that possess the characteristics of both high-brightness images and high-resolution images. [Brief explanation of the drawing]

[0012] [Figure 1] This figure schematically shows the functional configuration of an image forming apparatus according to Embodiment 1 of the present invention. [Figure 2] This is a block diagram showing the functional configuration of the image processing unit of an image forming apparatus according to Embodiment 1 of the present invention. [Figure 3] This figure shows photographs of a short-wave NIR image of a test specimen directly captured when an excitation light is irradiated onto a test specimen included in Embodiment 1 of the present invention, a short-wave NIR image of the observation target example, and a SWIR image of the observation target example. [Figure 4] This flowchart shows an example of the image forming process in Embodiment 1 of the present invention. [Figure 5] This figure shows an example of a histogram of a SWIR image in Embodiment 1 of the present invention. [Figure 6] This figure shows an example of a binarized SWIR image in Embodiment 1 of the present invention. [Figure 7] This figure shows an example of the contour of a binarized SWIR image and a mask image formed from said contour in Embodiment 1 of the present invention. [Figure 8] It is a diagram showing an example of a composite image obtained by superimposing a mask image on a short-wavelength side NIR image in Embodiment 1 of the present invention. [Figure 9] It is a diagram schematically showing a functional configuration of an image forming apparatus according to Embodiment 2 of the present invention. [Figure 10] It is a diagram showing an example of the relationship of focus positions at each wavelength of a focus shift correction lens in an image forming apparatus according to Embodiment 2 of the present invention. [Figure 11] It is a diagram schematically showing a functional configuration of an image forming apparatus according to Embodiment 3 of the present invention. [Figure 12] It is a diagram schematically showing the configuration of a NIR-SWIR filter in Embodiment 3 of the present invention. [Figure 13] It is a diagram schematically showing a functional configuration of an image forming apparatus according to Embodiment 4 of the present invention. [Figure 14] It is a diagram schematically showing the configuration of a VIS-NIR filter in Embodiment 4 of the present invention. [Figure 15] It is a diagram schematically showing a functional configuration of an image forming apparatus according to Embodiment 5 of the present invention. [Figure 16] It is a diagram schematically showing a functional configuration of an image forming apparatus according to Embodiment 6 of the present invention. [Figure 17] It is a diagram schematically showing a functional configuration of an image forming apparatus according to Embodiment 7 of the present invention. [Figure 18] It is a diagram schematically showing the configuration of a SWIR-SWIR filter in Embodiment 7 of the present invention. [Figure 19] It is a diagram schematically showing a functional configuration of an image forming apparatus according to Embodiment 8 of the present invention. [Figure 20] It is a diagram schematically showing the configuration of a VIS-NIR-SWIR filter in Embodiment 8 of the present invention. [Figure 21] It is a block diagram showing a functional configuration of an image processing unit of an image forming apparatus according to Embodiment 8 of the present invention. [Figure 22]This figure shows a timing chart illustrating an example of operation of an image forming apparatus according to Embodiment 8 of the present invention. [Figure 23] This flowchart shows an example of image processing that forms a composite image using top-hat transform. [Figure 24] This diagram schematically shows the original images in an image processing process where a composite image is formed using the top-hat transform. [Figure 25] This diagram schematically shows an image obtained by expanding the original image using the top-hat transform. [Figure 26] This diagram schematically shows the extracted boundary image in an image processing process that forms a composite image using a top-hat transform. [Figure 27] This diagram schematically shows the superimposed image of the boundary image and the second image in image processing that forms a composite image using the top-hat transform. [Figure 28] This flowchart shows an example of image processing that uses wavelet transform to form a composite image. [Figure 29] This diagram schematically shows the original images in an image processing step where a composite image is formed using wavelet transform. [Figure 30] This diagram schematically shows the frequency components of the original image decomposed by the wavelet transform. [Modes for carrying out the invention]

[0013] [Embodiment 1] One embodiment of the present invention will be described in detail below. In this specification, "~" represents a range including both of the numerical values ​​and less than or equal to the range between them. The image forming apparatus according to the embodiment of the present invention will be described using a subject who has been administered ICG as a fluorescent reagent as the subject of observation.

[0014] [Configuration of the image forming apparatus] Figure 1 is a schematic diagram showing the functional configuration of an image forming apparatus according to Embodiment 1 of the present invention. The image forming apparatus 1 has a function for capturing images of an object to be observed using visible light, and a function for capturing fluorescence emitted when ICG administered to the object to be observed is excited by irradiation with excitation light such as near-infrared light. As shown in Figure 1, the image forming apparatus 1 has a light source 10, an imaging unit 20, an image processing unit 30, and a monitor 40.

[0015] The light source 10 comprises a visible light source and an excitation light source that emits near-infrared light as excitation light to excite the ICG. The excitation light source is, for example, a laser that generates light with a wavelength of 808 nm. The excitation light is irradiated onto the object of observation simultaneously with the visible light from the light source 10 located at the tip of the rigid insertion section 21. Note that simultaneous irradiation does not necessarily mean that the irradiation periods are completely identical; it is sufficient if at least a portion of the irradiation period overlaps.

[0016] Furthermore, the excitation light is not limited to light in the above wavelength range, but is appropriately determined depending on the type of fluorescent reagent.

[0017] The imaging unit 20 includes a rigid insertion part (probe) 21 and an imaging unit 22. The rigid insertion part 21 is the part that is inserted into the body of the subject who has been pre-administered ICG, and has, for example, a cylindrical shape with a diameter of approximately 5 to 10 mm. The rigid insertion part 21 is composed of a light source 10 and a first optical system 211. The first optical system 211 is, for example, an objective lens.

[0018] The light source 10 does not necessarily have to be located at the tip of the rigid insertion section 21. For example, instead of the light source 10, the rigid insertion section 21 may hold an optical fiber that guides the light emitted by the light source 10.

[0019] In the image forming apparatus 1, the rigid insertion section 21 and the imaging unit 22 are detachably connected, and the configuration is such that light received by the rigid insertion section 21 is guided to the imaging unit 22. The light-guiding configuration may be, for example, a configuration that realizes a method called a relay lens or pupil relay, which transmits an image by light in a relay format. Alternatively, it may be an optical fiber capable of transmitting image information, such as an image guide fiber.

[0020] The imaging unit 22 consists of a second optical system 221, an excitation light cut filter 222, a dichroic prism 223, a first imaging unit 224, a second imaging unit 225, and a third imaging unit 226.

[0021] The second optical system 221 is, for example, an imaging lens. The excitation light cut filter 222 is an optical filter that reflects or absorbs only the incident excitation light to attenuate it, for example, a notch filter.

[0022] The dichroic prism 223 is a beam splitter that splits the SWIR, shortwave-side NIR, and VIS components of incident light into different directions. For example, it is a cubic beam splitter having two mutually orthogonal optical thin films. The incident light is light from the person being inspected. The SWIR component of the incident light is the light component in the shortwave infrared region (e.g., 900 nm to 1600 nm) of the light from the person being inspected. The shortwave-side NIR component of the incident light is the infrared light component in the wavelength region shorter than the shortwave infrared region (e.g., 750 nm to less than 900 nm) of the light from the person being inspected. The VIS component of the incident light is the light component in the visible light region (e.g., 400 nm to less than 750 nm) of the light from the person being inspected. The dichroic prism 223 splits the SWIR component of the observation light into one direction perpendicular to the direction of incidence of the observation light, and splits the VIS component of the observation light into the other direction perpendicular to the direction of incidence of the observation light. The dichroic prism 223 transmits (propagates in a straight line) the shortwave NIR component of the observation light.

[0023] The first imaging unit 224 is an image sensor that exposes incident light and outputs an image signal obtained by photoelectric conversion of the exposed light. It is an image sensor that is sensitive to the visible region and outputs an image signal of the components of VIS light (VIS image). On the imaging surface of the first imaging unit 224, color filters of the three primary colors red (R), green (G), and blue (B), or cyan (C), magenta (M), and yellow (Y) are arranged in a Bayer array or a honeycomb array.

[0024] The second imaging unit 225 is an image sensor that exposes incident light and outputs an image signal obtained by photoelectric conversion of the exposed light. It is a monochrome image sensor that is sensitive to wavelengths shorter than the short-wave infrared region within the near-infrared region. The second imaging unit 225 outputs an image signal of the short-wave NIR light component (short-wave NIR image).

[0025] The third imaging unit 226 is an image sensor that exposes incident light and outputs an image signal obtained by photoelectric conversion of the exposed light, and is a monochrome image sensor that is sensitive to the short-wave infrared region of near-infrared light. The third imaging unit 226 outputs an image signal of the SWIR light component (SWIR image).

[0026] In this embodiment, the positions of the first imaging unit 224, the second imaging unit 225, and the third imaging unit 226 in the optical path are adjusted so that the focus position becomes the position of the sensor (image plane) according to the respective wavelengths of the components of the light (VIS light, shortwave NIR light, and SWIR light) received by each sensor.

[0027] The image processing unit 30 is an image processing device that generates an image of the object to be observed by performing image processing, described later, on the image signal input from the imaging unit 22. Figure 2 is a block diagram showing the functional configuration of the image processing unit 30. As shown in Figure 2, the image processing unit 30 includes a visible light image processing unit 31 that applies predetermined image processing suitable for a visible light image to an image signal corresponding to the input visible light and outputs it, a fluorescence image processing unit 32 that applies predetermined image processing suitable for a fluorescence image to an input fluorescence image signal and outputs it, an image correction unit 33 that extracts contour components from a SWIR image and generates a mask image, and corrects blurring of the shortwave NIR image using the mask image, and an image synthesis unit 34 that synthesizes the fluorescence image signal output from the image correction unit 33 with the visible light image signal output from the visible light image processing unit 31.

[0028] Furthermore, the image correction unit 33 consists of a binarization processing unit 331 that converts the SWIR image signal into two values, black or white; a mask image generation unit 332 that traces the outer edge of the binarized image, fills in the traced interior, and creates a mask image; a gradation correction processing unit 333 that sets the signal value to 0 for bright images in areas where there is no signal in the mask image; and a color processing unit 334 that converts the luminance signal into a color signal.

[0029] The monitor 40 is a display device, such as a liquid crystal display (LCD), that displays the image input from the image processing unit 30.

[0030] [Image formation] First, let's describe the images related to this embodiment. Figure 3 is a diagram showing photographs of a shortwave NIR image taken directly of a test specimen included in the observation example of Embodiment 1 of the present invention when excitation light is irradiated onto the specimen, a shortwave NIR image of the observation example, and a SWIR image of the observation example. From left to right in Figure 3, the images show a shortwave NIR image taken directly of the test specimen being irradiated with excitation light, a shortwave NIR image taken of the observation example being irradiated with excitation light via a Roth-Ham filter (described later), and a SWIR image taken of the observation example being irradiated with excitation light via a Roth-Ham filter (described later). The arrows in the figure indicate the position of the image of the test specimen.

[0031] The test specimen is a glass tubule containing an ICG solution. The test specimen corresponds to the tissue in which ICG accumulates in living organisms and falls under the aforementioned "specific region." The short-wave NIR image taken directly in Figure 3 is an image of the test specimen when excitation light is irradiated onto it. Irradiation with excitation light at a wavelength of 808 nm maximizes the excitation efficiency of ICG, causing it to emit near-infrared fluorescence with a maximum fluorescence wavelength of approximately 835 nm. The fluorescence of the test specimen is clear, as shown in Figure 3.

[0032] The observed sample consisted of two 1.5 mm thick slices of roast ham stacked on top of each other. The roast ham represented the biological tissue interposed between the tissue where ICG accumulates in the body and the probe. The observed sample mimicked biological tissue in which ICG had settled within blood vessels.

[0033] The shortwave NIR images of the observed example above were captured by irradiating the observed example with excitation light from the biological tissue example side and taking images from the biological tissue example side. The shortwave NIR images above were captured by placing a bandpass filter in front of the VIS-SWIR compatible lens described later, which is attached to the VIS-SWIR camera described later, and capturing light with wavelengths of 830 to 900 nm. As shown in Figure 3, the shortwave NIR images of the observed example are sufficiently bright images because some of the ICG fluorescence is scattered by the biological tissue example, but they are low-resolution images (high-brightness, low-resolution images).

[0034] The SWIR image shown above was captured by placing a 900nm long-pass filter in front of the VIS-SWIR compatible lens, described later, attached to the VIS-SWIR camera. As shown in Figure 3, SWIR images are images of ICG fluorescence at wavelengths that are less likely to scatter in biological tissue, so they have low brightness but high resolution (low-brightness, high-resolution images).

[0035] In this embodiment, the SWIR light may be light with wavelengths in a partial range of 900 to 1600 nm, light with wavelengths in the entire range of 900 to 1600 nm, or light with wavelengths in a wider range that includes the entire range of 900 to 1600 nm. Similarly, in this embodiment, the short-wave NIR light may be light with wavelengths in a partial range of 750 nm or more and less than 900 nm, light with wavelengths in the entire range of 750 nm or more and less than 900 nm, or light with wavelengths in a wider range that includes the entire range of 750 nm or more and less than 900 nm. Furthermore, in this embodiment, the VIS light may be light with wavelengths in a partial range of 400 nm or more and less than 750 nm, light with wavelengths in the entire range of 400 nm or more and less than 750 nm, or light with wavelengths in a wider range that includes the entire range of 400 nm or more and less than 750 nm.

[0036] Furthermore, in this embodiment, the wavelength range of the short-wave NIR light and the wavelength range of the SWIR light may overlap in part. When setting the short-wave NIR light and SWIR light in the overlapping wavelength range, the light with a shorter wavelength or a shorter wavelength range is set as the short-wave NIR light, and the light with a longer wavelength or a longer wavelength range is set as the SWIR light. The shorter wavelength range is the wavelength range in which, of the two overlapping wavelength ranges in part, its lower limit is lower than the lower limit of the other wavelength range, and its upper limit is lower than the upper limit of the other wavelength range. The longer wavelength range is the wavelength range in which, of the two overlapping wavelength ranges in part, its lower limit is higher than the lower limit of the other wavelength range, and its upper limit is higher than the upper limit of the other wavelength range.

[0037] [Imaging] Next, we will explain the image formation process in the image forming apparatus 1.

[0038] The light source 10 emits visible light and excitation light from the tip of the rigid insertion section 21, illuminating the object being examined. As a result, the first optical system 211 is incident on the visible light and excitation light reflected from the object being examined, as well as fluorescence emitted when the ICG is excited by the excitation light. The first optical system 211 guides the incident excitation light, visible light, and fluorescence to the second optical system 221 provided in the imaging unit 22.

[0039] The second optical system 221 emits the excitation light, visible light, and fluorescence incident from the first optical system 211 to the excitation light cut filter 222. The excitation light cut filter 222 emits the attenuated light (visible light and fluorescence) to the dichroic prism 223. The dichroic prism 223 branches the VIS light component emitted from the excitation light cut filter 222, the shortwave NIR component including the above-mentioned fluorescence, and the SWIR light component also including the above-mentioned fluorescence into optical paths to the first imaging unit 224, the second imaging unit 225, and the third imaging unit 226.

[0040] The first imaging unit 224 exposes the VIS light emitted from the dichroic prism 223 and outputs an image signal corresponding to the VIS light to the image processing unit 30. The second imaging unit 225 exposes the shortwave NIR light emitted from the dichroic prism 223 and outputs an image signal corresponding to the shortwave NIR light to the image processing unit 30. The third imaging unit 226 exposes the SWIR light emitted from the dichroic prism 223 and outputs an image signal corresponding to the SWIR light to the image processing unit 30. In this way, each of the first imaging unit 224, the second imaging unit 225, and the third imaging unit 226 provided in the imaging unit 22 outputs the image signal obtained by each unit to the image processing unit 30.

[0041] [Formation of composite images] Next, the image formation process in this embodiment, which utilizes the image described above, will be explained. Figure 4 is a flowchart showing an example of the image formation process in Embodiment 1 of the present invention.

[0042] In step S101, the fluorescence image processing unit 32 creates a histogram of the SWIR image. Figure 5 shows an example of a SWIR image histogram in Embodiment 1 of the present invention.

[0043] Specifically, the fluorescence image processing unit 32 generates an image based on an image signal corresponding to the short-wave NIR light input from the second imaging unit 225 provided in the imaging unit 22, that is, a short-wave NIR image that is high-luminance but low-resolution (with a lot of blurring due to biological scattering). The short-wave NIR image is an image with an image density corresponding to the short-wave NIR light and corresponds to the second image which includes a specific region. In addition, the fluorescence image processing unit 32 generates an image based on an image signal corresponding to the SWIR light input from the third imaging unit 226 provided in the imaging unit 22, that is, a SWIR image that is low-luminance but high-resolution (with little blurring due to biological scattering). The SWIR image is a high-resolution image despite its low luminance. Therefore, the histogram of the SWIR image generated by the third imaging unit 226 shows a normal distribution-like shape with a peak in the part of the pixel value corresponding to the test specimen.

[0044] In step S102, the binarization processing unit 331 specifies the signal values ​​to be retained in the SWIR image. The binarization processing unit 331, for example, refers to a histogram and determines the number of pixels that are greater than the number of pixels with large increases or decreases in the amount of change in the histogram as the signal values ​​to be retained. For example, the binarization processing unit 331 assumes that the lower limit of the signal values ​​is the top 0.5% of signals, obtains the signal values ​​corresponding to these signals from the histogram in Figure 5, and sets this value as the threshold.

[0045] In step S103, the binarization processing unit 331 binarizes the SWIR image. Figure 6 shows an example of a binarized SWIR image in Embodiment 1 of the present invention. The binarization processing unit 331 creates a binarized image of the SWIR image using a threshold set by the binarization processing unit 331. The binarized image corresponds to the first image that shows the boundary of the specific region mentioned above. Compared with the original image on the left of Figure 3, it can be seen that the SWIR image retains the main structural parts of the original image.

[0046] In step S104, the mask image generation unit 332 creates a mask image using the binarized SWIR image. Figure 7 shows an example of the contour of the binarized SWIR image and the mask image formed from that contour in Embodiment 1 of the present invention. In order to obtain the portion corresponding to the signal of the binarized image in Figure 6, the mask image generation unit 332 traces the contour of the image in Figure 6 (see left figure in Figure 7) and fills in the interior to create the mask image shown in right figure in Figure 7. The region to be retained outside the contour in the mask image can be determined by what percentage of the top of the aforementioned histogram to retain.

[0047] In step S105, the tone correction processing unit 333 superimposes the mask image and the shortwave NIR image to create a superimposed image as a composite image. Figure 8 shows an example of a composite image obtained by superimposing a mask image on a shortwave NIR image in Embodiment 1 of the present invention. The tone correction processing unit 333 superimposes the mask image generated by the mask image generation unit 332 onto the shortwave NIR image. The superimposed image (composite image) is composed of a portion of the second image with high brightness and low resolution, where the interior of the contour determined from the first image with low brightness and high resolution is composed of the interior of the second image with high brightness and low resolution. Therefore, it combines the high-resolution information of the first image and the high-brightness information of the second image.

[0048] The image processing unit 30 corrects the superimposed image as needed. That is, in step S106, the image processing unit 30 appropriately corrects the signal values ​​in the superimposed image. For example, the grayscale correction processing unit 333 sets the signal value to 0 for areas in the mask image where there is no signal. Through such correction, the effect of scattering in living organisms on the observation light is corrected compared to conventional devices, and a composite image with high contrast can be obtained.

[0049] In this way, the image correction unit 33 extracts contour components from the SWIR image generated by the third imaging unit 226, fills in the contours, and creates a mask image. Then, it superimposes this mask image onto the shortwave NIR image generated by the second imaging unit 225, and sets the signal value to 0 for areas in the mask image where there is no signal. In this way, the image correction unit 33 synthesizes a high-brightness fluorescence image in which the effects of biological scattering are corrected compared to conventional devices.

[0050] The fluorescence image can be directly combined with the VIS image, but since the fluorescence image is a monochrome signal, the visibility of the fluorescence image contour may decrease. Therefore, in step S106, the color processing unit 334 processes the color of the fluorescence image as appropriate. The color processing unit 334 sets the color of the image inside the contour based on various criteria. For example, to improve visibility, the color processing unit 334 may set the color of the image inside the contour to be similar to the color of actual biological tissue, or to a color that does not exist in biological tissue, or to a color corresponding to the intensity of ICG fluorescence and perform multi-level imaging.

[0051] The image synthesis unit 34 then combines the VIS image acquired by the visible light image processing unit 31 with the appropriately color-processed fluorescence image to generate a composite image. For example, the image processing unit 30 combines the VIS image generated by the visible light image processing unit 31 and the fluorescence image generated by the image correction unit 33 in a predetermined ratio at the image synthesis unit 34.

[0052] The image processing unit 30 outputs the data of the composite image synthesized by the image synthesis unit 34 to the monitor 40. The monitor 40 displays the composite image. The user views the composite image displayed on the monitor 40 and achieves the purpose of observing the object of observation.

[0053] With this configuration, the image forming apparatus 1 excites the ICG administered to the subject of examination with excitation light, and presents the image of the subject of observation, which is produced by the fluorescence emitted by the excited ICG, to the examiner.

[0054] [Summary of Embodiment 1] As is clear from the above description, the image forming apparatus 1 of this embodiment includes an excitation light source (light source 10) for irradiating an object to be observed with excitation light, an imaging unit (20) that separates the light from the object to be observed being irradiated with excitation light into a first infrared light (SWIR light) including wavelengths in the short-wave infrared region and a second infrared light (short-wave NIR light) including wavelengths in a wavelength region shorter than the short-wave infrared region, and receives each light, and an image processing unit (30) that generates a composite image synthesized from a first image (SWIR image) showing the boundary of a specific region corresponding to the first infrared light received by the imaging unit, and a second image (NIR image) including a specific region having an image density corresponding to the second infrared light received by the imaging unit.Therefore, the image forming apparatus 1 can acquire a composite image that combines the features of both a high-brightness image by SWIR and a high-resolution image by NIR in ICG inspection.

[0055] The image processing unit may include an infrared image synthesis unit (fluorescence image processing unit 32) that generates a first image showing the outline of a specific region by binarizing the light intensity of the first infrared light received by the imaging unit, and synthesizes the first image and the second image. This configuration is even more effective in clearly and easily reflecting the high-resolution characteristics of the SWIR image in the synthesized image.

[0056] The imaging unit may include a separation unit (dichroic prism 223) that separates the first infrared light component and the second infrared light component from the light emitted from the object being observed to which the excitation light is irradiated. This configuration is even more effective in easily creating a precise composite image with virtually no positional information shift, because the data of the shortwave NIR image and the SWIR image used for processing the composite image are each created based on the same original image.

[0057] The separation unit may further separate visible light wavelengths (VIS light) from the light emitted from the object being observed to which the excitation light is irradiated. This configuration makes it possible to reflect the information from the VIS image in the composite image, and is even more effective in terms of facilitating the confirmation of the image of the object being observed and the observation of the object itself.

[0058] The image processing unit further includes a visible light image processing unit (31) that generates a visible light image using visible light wavelengths received by the imaging unit. The image processing unit may also generate a composite image by adding the visible light image to the image synthesized from the first image and the second image at a specific ratio. This configuration makes it possible to incorporate visible light information into the composite image (for example, making the area outside the contour based on the second image a visible light image). Therefore, it is even more effective in making the inspection results based on the created composite image easier to understand.

[0059] The imaging unit may include a filter (excitation light cut filter 222) that cuts out the excitation light from the light emitted from the object being observed. This configuration is even more effective in reducing the influence of excitation light on image synthesis.

[0060] The image processing unit may generate a composite image in which the luminance signals of areas outside a specific region in the first image are corrected to zero. This configuration makes it possible to clearly display only the superimposed image obtained by superimposing the first and second images, and is therefore even more effective in terms of easily and clearly displaying the inspection results based on the created composite image.

[0061] Other embodiments of the present invention will be described below. For the sake of convenience, in the following descriptions of each embodiment, components having the same function as those described in the previously described embodiments will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0062] [Embodiment 2] Figure 9 is a schematic diagram showing the functional configuration of an image forming apparatus according to Embodiment 2 of the present invention. As shown in Figure 9, the image forming apparatus 2 has the same configuration as the image forming apparatus 1 of Embodiment 1 described above, except that the configuration of the rigid insertion section 21 and the imaging unit 22 are slightly different.

[0063] The rigid insertion section 21 has a first optical system 2112 instead of the first optical system 211, and the imaging unit 22 has a second optical system 2212 instead of the second optical system 221. The objective lens and imaging lens of the second optical system 2212 are both corrected for focus shift in the wavelength range of VIS light and NIR light (e.g., 400-1600 nm).

[0064] VIS light and NIR light pass through the imaging unit 20. Therefore, wavelength-dependent focus shifts may occur due to differences in refractive index in the lens depending on the wavelength. Thus, from the viewpoint of preventing such focus shifts, this embodiment requires an optical system with high imaging performance that is well aberration-corrected over a wide wavelength range from the visible light range to the short-wave infrared range. From the viewpoint of suppressing such wavelength-dependent focus shifts and improving coupling performance, in this embodiment, the first optical system 2112 and the second optical system 2212 are focus-shift corrected.

[0065] More specifically, both the first optical system 2112 and the second optical system 2212 are optically designed to minimize the shift in the back focus (BF) position in the VIS region, the shortwave NIR region, and the SWIR region. In this embodiment, the focus position is the focus position for paraxial rays (rays that pass at a height very close to the optical axis). The "back focus" (BF) is the distance from the image-side plane of the optical system to the focal position, and these focus position values ​​do not change even if the F-number of the lens changes.

[0066] In this embodiment, from the viewpoint of effectively correcting aberrations in the images detected by each sensor, it is preferable that the first optical system satisfies the following equation (1), and that the second optical system satisfies the following equation (2). In the following equations, "BF_550nm" represents the back focus of the entire optical system at 550nm, "BF_850nm" represents the back focus of the entire optical system at 850nm, and "BF_1600nm" represents the back focus of the entire optical system at 1600nm. |BF_550nm-BF_850nm|<0.03 (1) |BF_550nm-BF_1600nm|<0.05 (2)

[0067] Table 1 shows the back focus values ​​for the entire optical system when various products are used in this embodiment. Figure 10 shows the relationship between the focus position at each wavelength for the lens with and without focus shift correction for product C in Table 1. In Figure 10, the solid line shows the focus position of the lens with focus shift correction, and the dashed line shows the focus position of the lens without focus shift correction. From Figure 10, it can be seen that by using the first and second optical systems with focus shift correction, the shift in the focus position, especially on the longer wavelength side, is corrected.

[0068] [Table 1]

[0069] The imaging unit 22 has a second imaging unit 2252 in place of the second imaging unit 225, and a third imaging unit 2262 in place of the third imaging unit 226. The second imaging unit 2252 and the third imaging unit 2262 have VIS-SWIR sensors. The VIS-SWIR sensor is an element that enables imaging from VIS light to SWIR light. More specifically, the second imaging unit 2252 and the third imaging unit 2262 may have the following configuration. Camera (Image Sensor): VIS-SWIR sensor (sensitivity 400~1700nm) equipped camera BH-71IGA (manufactured by BITRAN) Lens: VIS-SWIR compatible lens (transmission range 400-1700nm, with focus shift correction)

[0070] In this embodiment, both the second imaging unit 2252 and the third imaging unit 2262 are capable of detecting images of light from VIS light to SWIR light. Furthermore, both the first optical system 2112 and the second optical system 2212 are corrected for focus shift. In this embodiment, the imaging devices can be standardized, and there is no need to adjust the position of each imaging device in accordance with the focus shift. Therefore, in addition to the features of Embodiment 1 described above, this embodiment is more preferable from the viewpoint of simplifying the optical design of the imaging unit 20.

[0071] [Embodiment 3] Figure 11 is a schematic diagram showing the functional configuration of an image forming apparatus according to Embodiment 3 of the present invention. As shown in Figure 11, the image forming apparatus 3 has the same configuration as the image forming apparatus 2 of Embodiment 2 described above, except that the configuration of the imaging unit 22 is slightly different.

[0072] The imaging unit 22 has a dichroic prism 2233 instead of the dichroic prism 223. Furthermore, the imaging unit 22 does not have a third imaging section 2262, but instead has an additional NIR-SWIR filter 301 corresponding to the second imaging section 2252.

[0073] The dichroic prism 2233 is a beam splitter that divides the VIS light component of incident light into different directions. The dichroic prism 2233 divides the VIS light component of the observation light into one direction perpendicular to the direction of incidence of the observation light, while allowing the shortwave NIR and SWIR light components of the observation light to pass through (travel in a straight line).

[0074] Figure 12 is a schematic diagram showing the configuration of an NIR-SWIR filter in Embodiment 3 of the present invention. As shown in Figure 12, the NIR-SWIR filter 301 is composed of an NIR filter that transmits substantially only shortwave NIR light and a SWIR filter that transmits substantially only SWIR light. In the NIR-SWIR filter 301, the NIR filter and the SWIR filter are arranged in a checkerboard pattern, for example, as shown in the figure.

[0075] In this embodiment, both shortwave NIR images and SWIR images are captured by a single second imaging unit 2252. Therefore, in addition to the features of Embodiment 2 described above, this embodiment is preferable from the viewpoint of further simplifying the configuration of the imaging unit 20.

[0076] [Embodiment 4] Figure 13 is a schematic diagram showing the functional configuration of an image forming apparatus according to Embodiment 4 of the present invention. As shown in Figure 13, the image forming apparatus 4 has the same configuration as the image forming apparatus 3 of Embodiment 3 described above, except that the configuration of the imaging unit 22 is slightly different.

[0077] The imaging unit 22 has a dichroic prism 2234 in place of the dichroic prism 223. Furthermore, the imaging unit 22 has a first imaging unit 2244 in place of the first imaging unit 224, and also has a VIS-NIR filter 401 corresponding to the first imaging unit 2244.

[0078] The dichroic prism 2234 is a beam splitter that separates the VIS light and short-wave NIR light components of incident light into directions different from the SWIR light. The dichroic prism 2234 separates the VIS light and short-wave NIR light components of the observation light into one direction perpendicular to the direction of incidence of the observation light, while transmitting (straightening) the SWIR light component of the observation light.

[0079] The first imaging unit 2244 has a VIS-NIR sensor. The VIS-NIR sensor is an element that enables imaging from VIS light to shortwave NIR light.

[0080] Figure 14 is a schematic diagram showing the configuration of a VIS-NIR filter in Embodiment 4 of the present invention. As shown in Figure 14, the VIS-NIR filter 401 is configured by arranging, for example as shown in the figure, filters for each of the RGB colors for transmitting VIS light and an NIR filter that transmits substantially only shortwave NIR light in a Bayer array.

[0081] In this embodiment, both the VIS image and the shortwave NIR image are captured by a single first imaging unit 2244. Therefore, this embodiment is preferable from the viewpoint of further simplifying the configuration of the imaging unit 20, similar to the features of Embodiment 2 described above.

[0082] [Embodiment 5] Figure 15 is a schematic diagram showing the functional configuration of an image forming apparatus according to Embodiment 5 of the present invention. As shown in Figure 15, the image forming apparatus 5 has the same configuration as the image forming apparatus 1 of Embodiment 1 described above, except that the configuration of the rigid insertion section 21 and the imaging unit 22 are slightly different.

[0083] The rigid insertion section 21 has a first optical system 2112 instead of the first optical system 211, and the imaging unit 22 has a second optical system 2212 instead of the second optical system 221. Furthermore, the imaging unit 22 has a dichroic prism 2235 and third imaging units 2263 and 2264 instead of the third imaging unit 226.

[0084] The 2235 dichroic prism is a beam splitter that divides the SWIR (Swiss Wave Infrared) component of incident light into different directions depending on its wavelength. The 2235 dichroic prism divides the longer wavelength component of the SWIR light (e.g., 1000-1600 nm) into one direction perpendicular to the direction of incidence, while allowing the shorter wavelength component of the SWIR light (e.g., 900 nm to less than 1000 nm) to pass through (travel in a straight line).

[0085] The third imaging units 2263 and 2264 each have a SWIR sensor. The third imaging unit 2263 exposes the light component of SWIR light emitted from the dichroic prism 2235 between 900 nm and 1000 nm, and outputs an image signal corresponding to the light component of that wavelength to the image processing unit 30. The third imaging unit 2264 exposes the light component of SWIR light emitted from the dichroic prism 2235 between 1000 nm and 1600 nm, and outputs an image signal corresponding to the light component of that wavelength to the image processing unit 30.

[0086] Then, the fluorescence image processing unit 32 generates SWIR images in each wavelength range based on image signals corresponding to the components of SWIR light input from the third imaging unit 2263 and the third imaging unit 2264 provided in the imaging unit 22.

[0087] In this embodiment, a SWIR image of 900 nm to less than 1000 nm is captured by the third imaging unit 2263, and a SWIR image of 1000 to 1600 nm is captured by the third imaging unit 2264. The fluorescence intensity of ICG has a bell-shaped peak characteristic that gradually decreases from a maximum of approximately 835 nm to 1600 nm. Therefore, a bright SWIR image with high resolution can be obtained from the SWIR light component of 900 nm to less than 1000 nm. Furthermore, a SWIR image with lower brightness but higher resolution can be obtained from the SWIR light component of 1000 to 1600 nm compared to the SWIR light component of 900 nm to less than 1000 nm. For this reason, for example, if the location of the ICG administered to the subject is deep within the body, brightness can be prioritized by using a SWIR image of 900 nm to less than 1000 nm, while resolution can be prioritized by using a SWIR image of 1000 to 1600 nm if the location is shallow.

[0088] Thus, this embodiment is preferable from the viewpoint that the optimal SWIR image can be selected based on the location of the ICG administered to the subject of observation.

[0089] [Embodiment 6] Figure 16 is a schematic diagram showing the functional configuration of an image forming apparatus according to Embodiment 6 of the present invention. As shown in Figure 16, the image forming apparatus 6 has the same configuration as the image forming apparatus 1 of Embodiment 1 described above, except that the configuration of the rigid insertion section 21 and the imaging unit 22 are partially different.

[0090] The rigid insertion section 21 has a first optical system 2112 instead of the first optical system 211, and the imaging unit 22 has a second optical system 2212 instead of the second optical system 221. Furthermore, the imaging unit 22 has a dichroic prism 2236 instead of the dichroic prism 223, and a second imaging unit 2252 and an NIR-SWIR filter 301 instead of the second imaging unit 225.

[0091] The dichroic prism 2236 is a beam splitter that separates the NIR (Near-Infrared) and visible light components of incident light into different directions. The dichroic prism 2236 transmits (propels in a straight line) the shorter wavelength component of the NIR light of the observed light (e.g., between 800 nm and 1000 nm). The dichroic prism 2236 also separates the 1000-1600 nm wavelength component of the NIR light of the observed light into one direction perpendicular to the direction of incidence of the observed light, and separates the VIS (Visible Light) component of the observed light into the other direction perpendicular to the direction of incidence of the observed light.

[0092] In this embodiment, from the light component of the incident NIR light with wavelengths between 800 nm and less than 1000 nm, an image signal of a short-wave NIR image with wavelengths between 800 nm and less than 900 nm and an image signal of a SWIR image with wavelengths between 900 nm and less than 1000 nm are obtained. Furthermore, in this embodiment, an image signal of a SWIR image can also be obtained from the light component with wavelengths between 1000 and 1600 nm. The detection intensity of ICG fluorescence varies depending on the type of detector; for example, InGaAs-based detectors have stronger sensitivity in the SWIR region compared to Si-based detectors. This embodiment is suitable from the viewpoint of further increasing the detection sensitivity of SWIR light when a sensor with stronger sensitivity in the SWIR region is used in the second imaging unit 2252.

[0093] [Embodiment 7] Figure 17 is a schematic diagram showing the functional configuration of an image forming apparatus according to Embodiment 7 of the present invention. As shown in Figure 17, the image forming apparatus 7 has the same configuration as the image forming apparatus 1 of Embodiment 1 described above, except that the configuration of the rigid insertion section 21 and the imaging unit 22 are slightly different.

[0094] The rigid insertion section 21 has a first optical system 2112 instead of the first optical system 211, and the imaging unit 22 has a second optical system 2212 instead of the second optical system 221. Furthermore, the imaging unit 22 has a SWIR-SWIR filter 701 corresponding to a third imaging section 226.

[0095] Figure 18 is a schematic diagram showing the configuration of a SWIR-SWIR filter in Embodiment 7 of the present invention. As shown in Figure 18, the SWIR-SWIR filter 701 has a first SWIR filter and a second SWIR filter arranged in a checkerboard pattern, for example as shown in the figure. The first SWIR filter is a filter that transmits only the light component of SWIR light with shorter wavelengths (e.g., 900 nm or more and less than 1000 nm). The second SWIR filter is a filter that transmits only the light component of SWIR light with longer wavelengths (e.g., 1000 to 1600 nm).

[0096] This embodiment, like Embodiment 6, is preferable from the viewpoint of further increasing the detection sensitivity of SWIR light when a sensor having stronger sensitivity in the SWIR region is used in the third imaging unit 226.

[0097] [Embodiment 8] Figure 19 is a schematic diagram showing the functional configuration of an image forming apparatus according to Embodiment 8 of the present invention. As shown in Figure 19, the image forming apparatus 8 includes a rigid insertion unit 21, an imaging unit 22, an image processing unit 80, and a monitor 40.

[0098] The rigid insertion section 21 includes a light source 1010 and a first optical system 2112. The light source 1010 is a light source that intermittently irradiates the object of observation with light in the red (R), green (G), blue (B), and near-infrared (NIR) wavelength bands. Specifically, the light source 1010 includes a red light source 1011 (e.g., wavelength 633 nm), a green light source 1012 (e.g., wavelength 470 nm), a blue light source 1013 (e.g., wavelength 525 nm), and a near-infrared light source 1014 (e.g., wavelength 808 nm) as an excitation light source. These light sources are connected to a control unit (not shown) and are configured to irradiate light at specific timings according to control signals from the control unit.

[0099] The imaging unit 22 includes a second optical system 2212, an excitation light cut filter 222, a VIS-NIR-SWIR filter 801, and a second imaging unit 2252. Figure 20 is a schematic diagram showing the configuration of the VIS-NIR-SWIR filter in Embodiment 8 of the present invention. As shown in Figure 20, the VIS-NIR-SWIR filter 801 is composed of a VIS filter that transmits substantially only VIS light, an NIR filter that transmits substantially only shortwave NIR light, and a SWIR filter that transmits substantially only SWIR light. In the VIS-NIR-SWIR filter 801, the VIS filter, NIR filter, and SWIR filter are arranged in a specific pattern, for example, a checkerboard pattern as shown in the figure.

[0100] Figure 21 is a block diagram showing the functional configuration of the image processing unit of an image forming apparatus according to Embodiment 8 of the present invention. As shown in Figure 21, the image processing unit 80 further includes an RGB synthesis processing unit 81 that receives image signals corresponding to R, G, and B input from the second imaging unit 2252, generates visible light image signals corresponding to each of the R, G, and B components, and outputs them to the visible light image processing unit 31. Otherwise, the image processing unit 80 has the same functional configuration as the image processing unit 30 described above.

[0101] In this embodiment, in synchronization with the imaging timing of the second imaging unit 2252, light in the red (R), green (G), blue (B), and near-infrared (NIR) wavelength bands is intermittently irradiated onto the object to be observed, and the red (R), green (G), and blue (B) reflected light from the object to be observed is exposed in a time-division manner, and a visible light image is generated by RGB synthesis processing. Figure 22 is a timing chart illustrating an example of the operation of an image forming apparatus according to Embodiment 8 of the present invention. Note that in Figure 22, "NIR" in "NIR+SWIR fluorescence" for the imaging unit 2252 means "shortwave NIR".

[0102] As shown in Figure 22, when the light source 1010 is irradiated with red light from the red light source 1011, the control unit causes the second imaging unit 2252 to output the image signal of the red light component of the VIS-NIR-SWIR filter 801 that has passed through the VIS filter, which has been received by the second imaging unit 2252. Similarly, when the light source 1010 is irradiated with green light from the green light source 1012, the control unit causes the second imaging unit 2252 to output the image signal of the green light component, and when the light source 1013 is irradiated with blue light, the control unit causes the second imaging unit 2252 to output the image signal of the blue light component. Furthermore, when the light source 1010 is irradiated with NIR light from the near-infrared light source 1014, the control unit causes the second imaging unit 2252 to output the image signal of the shortwave side NIR light component that has passed through the NIR filter, and also outputs the image signal of the SWIR light component that has passed through the SWIR filter, which has been received by the second imaging unit 2252. In this manner, the output and stop of the red light source 1011, green light source 1012, blue light source 1013, and near-infrared light source 1014 from the light source 1010 are repeated alternately, and the second imaging unit 2252 acquires a red image signal, a blue image signal, a green image signal, an image signal corresponding to shortwave NIR light, and an image signal corresponding to SWIR light for one frame.

[0103] Of these image signals, the red image signal, blue image signal, and green image signal are output to the RGB synthesis processing unit 81 of the image processing unit 80 to generate a visible light image, while the image signal corresponding to near-infrared fluorescence and the image signal corresponding to short-wave infrared fluorescence are output to the fluorescence image processing unit 32 of the image processing unit 80.

[0104] In this embodiment, a single second imaging unit 2252 captures VIS images, shortwave NIR images, and SWIR images. Therefore, this embodiment is preferable from the viewpoint of further simplifying the configuration of the imaging unit 20.

[0105] [Other embodiments] The following describes other embodiments for creating composite images according to the present invention.

[0106] [Creation of composite images using boundary restoration processing] Even in bright images (shortwave NIR images), there should be clear boundaries between different tissues, but because the image is blurred, these boundaries between tissues become unclear. On the other hand, although the signal in dark images (SWIR images) is small, the boundaries between tissues are sharp. By extracting the boundaries from the dark image (SWIR image) and overlaying the obtained boundaries onto the blurred image, it is possible to reproduce the sharp boundaries between tissues that the original image had.

[0107] A dark image with sharp boundaries between tissues (SWIR image) is used, and a top-hat transform is applied to this image. Figure 23 is a flowchart showing an example of image processing to form a composite image using the top-hat transform.

[0108] In step S201, the image processing unit 30 selects a range of the SWIR image to create a raw image. Figure 24 schematically shows the raw image in the image processing that forms a composite image using top-hat transform. The entire image can be divided into two regions according to the magnitude of the signal values ​​of the tissue. Specifically, it is an image with a boundary as shown in Figure 24. For example, let I21 be the region of tissue emitting ICG fluorescence. Region I20 is the region outside of that tissue.

[0109] In step S202, the image processing unit 30 creates an image by expanding the original image. Figure 25 schematically shows an image obtained by expanding the original image using a top-hat transform. For example, at the boundary of region I21, the image processing unit 30 copies the pixel values ​​of pixels adjacent to the boundary to pixels outside the boundary that are adjacent to the boundary in all directions (up, down, left, and right). As a result, region I22 is created, which is expanded by one pixel along the boundary of region I21. In Figure 25, the left side and top and bottom of region I22 are the edges of the image, so there is no change in the pixel values, and only the right side of the region is expanded.

[0110] In step S203, the image processing unit 30 creates a boundary image from the difference between the original image before and after expansion. Figure 26 schematically shows the extracted boundary image in the image processing that forms a composite image using top-hat transform. By subtracting region I21 from region I22, only the expanded portion remains. In this way, boundary region I23 corresponding to the tissue boundary is created.

[0111] In step S204, the image processing unit 30 superimposes the boundary image onto the shortwave NIR image. Figure 27 schematically shows the superimposed image of the boundary image and the second image (shortwave NIR image) in the image processing that forms a composite image using top-hat transform. Image I10 is the shortwave NIR image, and region I24 is an image with enhanced boundary areas, obtained by multiplying the pixel values ​​of region I23 by a constant.

[0112] In this way, by applying a top-hat transform to a dark image (SWIR image) and multiplying the boundary signal obtained by a constant factor, and adding it to a bright image (shortwave NIR image) where the tissue boundaries are unclear, the clear boundaries between tissues present in the dark image (SWIR image) are reflected in the bright image (shortwave NIR image), and a bright image (composite image) with clear boundaries between tissues is synthesized.

[0113] In addition, when combining images, the boundary image may be subtracted from the shortwave NIR image. In this case as well, a bright image (composite image) with clear boundaries between tissues will be combined.

[0114] [Creating composite images using edge enhancement] Figure 28 is a flowchart illustrating an example of image processing that forms a composite image using wavelet transform. Figure 29 is a schematic diagram showing the original image I0 in image processing that forms a composite image using wavelet transform. Original image I0 is, for example, an image of tissue emitting ICG fluorescence.

[0115] In step S301, the image processing unit 30 performs a two-dimensional wavelet transform on the SWIR image.

[0116] In step S302, the image processing unit 30 ranks the high-pass components of the SWIR image. Figure 30 schematically shows the frequency components of the original image I0 decomposed by the wavelet transform. Edge components appear in the high-pass components. Thus, when a wavelet transform is performed on a dark image (SWIR image), the high-pass filter components corresponding to the edges have large values.

[0117] In step S303, the image processing unit 30 selects the high-ranking high-pass image components of the SWIR image. Components with large values ​​are recorded using this high-pass filter.

[0118] In step S304, the image processing unit 30 performs a two-dimensional wavelet transform on the shortwave NIR image. For example, the wavelet transform is applied to a bright image (shortwave NIR image). In the bright image (shortwave NIR image) after the wavelet transform, because the shortwave NIR image is a high-brightness, low-resolution image, the components corresponding to edges become small and are buried in components other than edges.

[0119] In step S305, the image processing unit 30 multiplies the selected high-pass component of the SWIR image (the component with a large value recorded in step S303) by a constant in the wavelet-transformed shortwave NIR image. In this way, the component in the wavelet-transformed shortwave NIR image is restored using the wavelet-transformed component of the SWIR image. As a result, a wavelet-transformed shortwave NIR image with clear edges is created.

[0120] In step S306, the image processing unit 30 performs an inverse wavelet transform on the shortwave NIR image, which has been wavelet transformed and whose edges have been clarified. As a result, a composite image is obtained in which the clear edges of the dark image (SWIR image) are reflected in the bright image (shortwave NIR image).

[0121] [Variation] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0122] For example, in embodiments of the present invention, the excitation light cut filter may be a filter or a combination thereof that transmits light of a desired wavelength. For example, the excitation light cut filter may be a combination of a bandpass filter (transmission band 830-900 nm) that transmits the short-wave NIR component in ICG fluorescence and a long-pass filter (transmission band 900-1600 nm) that transmits the SWIR component.

[0123] Furthermore, the excitation light cut filter only needs to be placed between the object being observed and the image sensor, and the dichroic prism itself may also be equipped with an excitation light cut filter.

[0124] The function of the image processing unit 30 in the embodiment of the present invention is a program for causing a computer to function as the processing unit, and can be realized by a program for causing a computer to function as each control block of the processing unit.

[0125] In this case, the processing unit includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0126] The above program may be recorded on one or more computer-readable recording media, not temporary ones. This recording media may or may not be provided by the processing unit. In the latter case, the program may be supplied to the processing unit via any wired or wireless transmission medium.

[0127] Furthermore, some or all of the functions of each of the above control blocks can also be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as each of the above control blocks are formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above control blocks by, for example, a quantum computer.

[0128] Furthermore, each process described in the above embodiments may be performed by AI (Artificial Intelligence). In this case, the AI ​​may operate on the control device described above, or it may operate on other devices (for example, an edge computer or a cloud server).

[0129] Furthermore, the image forming apparatus of the present invention may further include a diagnostic unit for diagnosing tissue from the created composite image. The diagnostic unit may include a determination unit for determining the image using an image determination model trained on the composite image data as training data. Examples of image determination models include neural networks and support vector machines. Examples of neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected neural networks.

[0130] The image recognition model can be trained by referencing training data. This training data includes image data of a composite image and at least one piece of information (such as a disease in that area) corresponding to the image data. The image recognition model can be trained by preparing a sufficient amount of the above training data (image data and corresponding area information), training a neural network, and determining the path weights for each image data. Examples of algorithms for training the image recognition model include backpropagation and ID3.

[0131] Note that the image classification model does not have to be a machine learning model. For example, the image classification model may be a regression model in which the image data is the dependent variable and information regarding the appropriateness of the image data is the independent variable.

[0132] In this invention, the sharpness of an image with sufficient image density detected by light of a first wavelength is supplemented by an image with high resolution but insufficient image density detected by light of a second wavelength, thereby obtaining an image that is sufficiently bright and sufficiently sharp. This invention utilizes light in a specific wavelength range in the infrared spectrum. This invention can be applied when a specific wavelength range exists in which the above-described characteristics of image brightness and sharpness can be obtained.

[0133] 〔summary〕 As is clear from the above description, an image forming apparatus according to a first aspect of the present invention includes: an excitation light source for irradiating an object to be observed with excitation light; an imaging unit that separates the light from the object to be observed irradiated with the excitation light into a first infrared light including wavelengths in the short-wave infrared region and a second infrared light including wavelengths in a wavelength region shorter than the short-wave infrared region, and receives each of the lights; and an image processing unit that generates a composite image synthesized from a first image showing the boundary of the specific region corresponding to the first infrared light received by the imaging unit and a second image including the specific region having an image density corresponding to the second infrared light. This first aspect can provide a new technique for acquiring an image that combines the characteristics of both a high-brightness image and a high-resolution image.

[0134] In a second aspect of the present invention, the image forming apparatus may include, in the first aspect described above, an infrared image synthesis unit that generates a first image showing the contour of the image of the specific region by binarizing the amount of the first infrared light received by the imaging unit, and synthesizes the first image and the second image.

[0135] In a third aspect of the present invention, the image forming apparatus may, in the first or second aspect described above, include a separation unit that separates the first infrared light component and the second infrared light component from the light from the object being observed to which the excitation light is irradiated.

[0136] In the fourth aspect of the present invention, the image forming apparatus, in the third aspect described above, may further separate visible light wavelengths from the light from the object being observed to which the excitation light is irradiated.

[0137] An image forming apparatus according to a fifth aspect of the present invention, in the fourth aspect described above, further comprises a visible light image processing unit which generates a visible light image using light of the visible light wavelength received by the imaging unit, and the composite image is generated by adding the visible light image to an image synthesized from the first image and the second image at a specific ratio.

[0138] In the sixth aspect of the present invention, the image forming apparatus may, in the fourth or fifth aspect described above, include an optical system that corrects the focus shift of light from visible light to short-wave infrared from the observation target to which the excitation light is irradiated.

[0139] In the seventh aspect of the present invention, the image forming apparatus may, in any of the first to sixth aspects described above, include a filter in the imaging unit that cuts out the excitation light from the light from the object being observed that is being irradiated with the excitation light.

[0140] In the eighth aspect of the present invention, the image forming apparatus may, in any of the first to seventh aspects described above, generate a composite image in which the luminance signal of the region outside the specific region in the first image is corrected to zero.

[0141] As described above, the present invention relates to an observation device capable of acquiring a composite image of a short-wave near-infrared light image and a short-wave infrared light image from an object containing a fluorescent substance. According to the above-described embodiment of the present invention, it is possible to form an image that includes high contrast from the first image and image density from the second image, making it possible to more precisely identify the target region in biological tissue.

[0142] According to the present invention, the results of tests utilizing fluorescence in vivo can be clearly displayed. Therefore, the present invention is expected to contribute to achieving the Sustainable Development Goals (SDGs) related to ensuring healthy living and promoting well-being. [Explanation of Symbols]

[0143] 1-8 Image forming apparatus 10, 1010 light source 20 Imaging Department 21. Rigid insertion section 22 Imaging Unit 30, 80 Image Processing Unit 31 Visible light image processing unit 32 Fluorescence imaging processing unit 33 Image Correction Unit 34 Image Synthesis Unit 40 monitors 81 RGB Synthesis Processing Unit 211, 2112 First optical system 221, 2212 Second optical system 222 Excitation light cut filter 223, 2233~2236 Dichroic prism 224, 2244 First Imaging Unit 225, 2252 Second imaging unit 226, 2262~2264 Third imaging unit 301 NIR-SWIR filter 331 Binarization Processing Unit 332 Mask Image Generation Unit 333-level tone correction processing unit 334 Color Processing Unit 401 VIS-NIR filter 701 SWIR-SWIRフィルタ 801 VIS-NIR-SWIRフィルタ 1011 Red Light Source 1012 Green Light Source 1013 Blue Light Source 1014 Near-infrared light source I0 Original Portrait I20, I21, I22, I23, I24 domains

Claims

1. An excitation light source for irradiating the object to be observed with excitation light to excite the fluorescent substance contained in the object to be observed, An imaging unit separates the light from the observation target being irradiated with the excitation light into a first infrared light containing fluorescence from the fluorescent substance at wavelengths in the short-wave infrared region and a second infrared light containing fluorescence from the fluorescent substance at wavelengths in a wavelength region shorter than the short-wave infrared region, and receives each of the lights. An image processing unit that generates a composite image from a first image showing the boundary of a specific region containing the fluorescent substance corresponding to the first infrared light received by the imaging unit, and a second image including the specific region having an image density corresponding to the second infrared light received by the imaging unit, An image forming apparatus having

2. The image forming apparatus according to claim 1, wherein the image processing unit comprises an infrared image synthesis unit that generates a first image showing the contour of the image of the specific region by binarizing the amount of the first infrared light received by the imaging unit, and synthesizes the first image and the second image.

3. The image forming apparatus according to claim 1, wherein the imaging unit includes a separation unit that separates the first infrared light component and the second infrared light component from the light from the object being observed to which the excitation light is irradiated.

4. The image forming apparatus according to claim 3, wherein the separation unit further separates visible light wavelengths from the light from the object being observed that is being irradiated with the excitation light.

5. The image processing unit further comprises a visible light image processing unit that generates a visible light image using light of the visible light wavelength received by the imaging unit. The image forming apparatus according to claim 4, wherein the visible light image is added to the image synthesized from the first image and the second image at a specific ratio to generate the composite image.

6. The image forming apparatus according to claim 5, wherein the imaging unit includes an optical system for correcting the focus shift of light in the region from visible light to short-wave infrared from the object being observed to which the excitation light is irradiated.

7. The image forming apparatus according to claim 1, wherein the imaging unit includes a filter that cuts out the excitation light from the light from the object being observed that is being irradiated with the excitation light.

8. The image forming apparatus according to claim 1, wherein the image processing unit generates a composite image in which the luminance signal of the region outside the specific region in the first image is corrected to zero.

Images