Endoscope device and image processing method of endoscope device

Abstract

To solve the problem with a conventional endoscope device that it is difficult to achieve both acquisition of a bright and high-quality image and reduction of a burden on a patient since the endoscope device has conflicting characteristics that a thick diameter is better for acquiring a bright and high-quality image while a thin diameter is desirable for reducing a burden on a patient.SOLUTION: An endoscope device outputs an image obtained by applying visual emphasis (VEViD) algorithm processing by virtual diffraction and coherent detection utilizing functions in low light quantity emphasis and color emphasis to image information on an observation visual field in a living body acquired by an endoscope.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] Endoscopic examinations are widely used as a screening and diagnostic method for early detection of stomach cancer. From the perspective of improving the accuracy of screening and diagnosis, attempts are continually being made to improve the image quality obtained using endoscopes. A schematic diagram of the flexible endoscope used for stomach examinations and the shape of its tip are shown in Figures 1 and 2, respectively. (Non-Patent Document 1)

[0002] In the flexible endoscope system, an LED light source for illumination is arranged inside the controller 1, and the illumination light emitted from the LED light source is guided through the flexible cable 4 and the light guide fiber provided inside the flexible cable 5, and illuminates the image capturing section via the illumination lens 7 arranged at the tip. An objective lens 6 for capturing images is disposed at the tip of the flexible endoscope. The imaging element for acquiring image data may be disposed in the imaging section of the objective lens 6 at the tip of the flexible endoscope, or an image guide fiber may be disposed in the imaging section of the objective lens 6 and the imaging element may be disposed in the endoscope operation unit 3 or the controller 1. The captured image is processed in the controller and displayed on the display 2.

[0003] If a suspicious area of ​​cancer is discovered during a flexible endoscope examination, a tissue sample is taken using forceps. For this reason, the flexible cable 5 that enters the body is provided with a hole called a forceps port through which the forceps for tissue sampling can be inserted. In addition, there are also air and water supply holes for cleaning the area to be examined, blowing off blood, or rinsing it away.

[0004] To make a correct diagnosis during a flexible endoscope examination, it is desirable to obtain clear, high-quality images. To obtain clear, high-quality endoscopic images, it is desirable to use illumination with a sufficient amount of light and high-performance lenses and image sensors. Obtaining a sufficient amount of illumination light requires the use of thick light guide fibers, and high-performance lenses and image sensors require the use of large-diameter objective lenses and large image sensors, which increase the diameter of the flexible endoscope. On the other hand, flexible endoscopes are also expected to reduce the burden on patients. To achieve this, it is desirable for the endoscope to have a small diameter. Therefore, it is difficult to achieve both high-quality endoscopic images and reduced burden on patients.

[0005] To obtain bright images under dim lighting conditions, endoscopes also use a sequential imaging method (Non-Patent Document 2), in which a monochrome image sensor is used to capture images of the three primary colors of red, green, and blue that make up a color image in sequence while switching the wavelength of the illumination, and then a color image is constructed. When using this method, the red, green, and blue component images are taken at different times, which means that if the photographer moves the tip of the endoscope, the colors of the images tend to shift. This means that the photographer must not move the endoscope when taking still images, which places a burden on the photographer.

[0006] On the other hand, there are also non-flexible endoscopes (rigid endoscopes), such as arthroscopes used to observe inside joints. Figure 3 shows a system using a rigid endoscope, Figure 4 shows a schematic diagram of the configuration of a rigid endoscope, and Figure 5 shows a cross-sectional structural diagram of the endoscope barrel, the part inserted into the body, at the position indicated by the broken line 30. The rigid endoscope system 102 shown in FIG. 3 is equipped with a camera 14. Image data captured by the camera 14 is sent via electrical wiring 15 to the controller 11, where it is converted into an image and displayed on the display 12. Illumination light emitted from an LED light source in the illumination unit 13 is guided via a light guide 18 to a light guide connection port 17, illuminating the observation field. As shown in the schematic diagram of the rigid endoscope configuration in FIG. 4, the endoscope barrel 16 of a rigid endoscope often uses a relay lens system using multiple lenses 19, which causes less image degradation than image guide fibers, for image transmission. A camera connection lens 20 allows the camera 14 to capture a focused image of the observation field. As shown in FIG. 5, a cross-sectional view of the endoscope barrel 16, multiple lenses 19 are arranged in the center of the endoscope barrel 16, and light guide wires 21 connected to the light guide connection port 17 are arranged around the periphery.

[0007] Even in rigid endoscope systems, it is desirable to obtain clear, high-quality images for accurate diagnosis. To obtain clear, high-quality endoscopic images, it is desirable to use a sufficient amount of illumination light and high-performance lenses and image sensors. Obtaining a sufficient amount of illumination light requires the use of thick light guide fibers, i.e., a light guide with a large number of light guide filaments. High-performance lenses and image sensors require the use of large-diameter lenses 19, which increases the diameter of the rigid endoscope. Rigid endoscopes also need to have a small diameter so that they can be inserted into the knee joint with minimal strain on the patient, for example. Therefore, it is difficult to achieve high image quality, ease of use for doctors, and reduced strain on the patient at the same time with rigid endoscopes.

[0008] In order to confirm the difficulty of achieving both high image quality and reduced patient burden in endoscopes, an image acquisition experiment was conducted using the optical system shown in Figure 6. The experimental optical system 92 is an optical system that uses a rigid endoscope having the endoscope barrel 16 shown in Figure 4, and captures an image of a sample 23 with a color optical camera 14 by illuminating white LED light through a light guide connection port 17. The sample 23 is mounted on a rotary motor 24, and the optical system rotates the sample 23 at a constant speed within the camera's observation field of view. Figure 7 shows a black and white image of a color image captured with an exposure time of 180 milliseconds, and Figure 8 shows a black and white image of a color image captured with an exposure time of 40 milliseconds. In a normal indoor lighting environment, a sufficiently bright image can be captured with an exposure time of 40 milliseconds, but with the optical system shown in Figure 6, a sufficiently bright image could not be captured with an exposure time of 40 milliseconds. In addition, it can be seen in the image shown in Figure 7 that the image is blurred due to the rotation of motor 24.

[0009] The images shown in Figures 7 and 8 are images with 600 pixels in both the vertical and horizontal directions, and the results of plotting the luminance profile at the 400th pixel position from the left are shown in Figures 9 and 10. Figure 9 shows the luminance of each color in the color image captured with an exposure time of 180 milliseconds as shown in Figure 7, and Figure 10 shows the luminance of each color in the color image captured with an exposure time of 40 milliseconds as shown in Figure 8. In the results shown in Figure 9, the luminance profile around pixel position 330 is oblique, indicating that the image is blurred. On the other hand, in the results shown in Figure 10, the luminance profile around pixel position 330 is not so oblique, indicating that the image brightness is insufficient but there is little blurring. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] Yamato Kanda, "Endoscopy: What You Should Know Now," Journal of Biomedical Engineering, Vol. 62, No. 1, pp. 45-55 (2024) [Non-patent document 2] Tajiri, Hisao, "Advances in Electronic Endoscopes and Related New Developments," Journal of the Japanese Society of Gastroenterological Endoscopy, Vol. 50, Suppl 3, pp. 3429-3435 (2008) [Non-patent document 3] Bahram Jalali and Callen MacPhee, "VEViD: Vision Enhancement via Virtual diffraction and coherent Detection", eLight Vol. 2, No. 1, 24 (2022) Summary of the Invention [Problem to be solved by the invention]

[0011] The present invention has been made in consideration of the above points, and provides a method for solving the problem that when acquiring images using an endoscope, if an exposure time is extended in an attempt to acquire an image with sufficient brightness, the image is likely to become blurred. [Means for solving the problem]

[0012] In order to solve such problems, the endoscopic device and image processing method for the endoscopic device of the present invention are a device configuration or image processing method that performs VEViD (Vision Enhancement via Virtual Diffraction and Coherent Detection) processing on images acquired by a camera, thereby obtaining captured images with sufficient brightness even when the exposure time is short. [Effects of the Invention]

[0013] The endoscopic device and image processing method for the endoscopic device of the present invention are a system capable of displaying images with sufficient brightness compared to conventional illumination light intensity and exposure times, so images with little blur can be obtained even when the surgeon moves the endoscopic system, thereby reducing the burden on the surgeon. Furthermore, if the endoscope is used with the same exposure time as conventional, it is possible to reduce the illumination light intensity compared to conventional methods, i.e., to make the diameter of the light guide thinner, which allows the endoscope barrel to be made thinner, thereby reducing the burden on the patient. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram illustrating the configuration of a typical flexible endoscope system. [Figure 2] FIG. 1 is a schematic diagram illustrating the configuration of a tip portion of a typical flexible endoscope. [Figure 3] FIG. 1 is a schematic diagram illustrating the configuration of a typical rigid endoscope system. [Figure 4] FIG. 1 is a schematic diagram illustrating the configuration of a typical rigid endoscope. [Figure 5] FIG. 1 is a schematic diagram of a cross section of a typical rigid endoscope barrel. [Figure 6] FIG. 1 is a schematic diagram illustrating the configuration of an experimental optical system for characterization. [Figure 7] This is an image obtained during a characteristic determination experiment. [Figure 8] This is an image obtained during a characteristic determination experiment. [Figure 9] FIG. 10 is a luminance distribution diagram of an image acquired in a characteristic determination experiment. [Figure 10] FIG. 10 is a luminance distribution diagram of an image acquired in a characteristic determination experiment. [Figure 11] FIG. 2 is a flowchart showing an image processing method for the endoscope apparatus of the present invention. [Figure 12] 10A and 10B are diagrams showing the image processing results obtained by the image processing method for the endoscope apparatus of the present invention. [Figure 13] 10 is a luminance distribution diagram of the image processing result obtained by the image processing method for the endoscope device of the present invention. [Figure 14] 1 is a diagram illustrating a schematic configuration example of an endoscope apparatus according to the present invention. [Figure 15] 1 is a diagram illustrating a schematic configuration example of an endoscope apparatus according to the present invention. [Figure 16] 1 is a diagram illustrating a schematic configuration example of an endoscope apparatus according to the present invention. [Figure 17] 1 is a diagram illustrating a schematic configuration example of an endoscope apparatus according to the present invention. [Figure 18] 1 is a diagram illustrating a schematic configuration example of an endoscope apparatus according to the present invention. [Figure 19] FIG. 1 is a schematic diagram illustrating the configuration of an experimental optical system for characterization. [Figure 20] This is an image obtained during a characteristic determination experiment. [Figure 21] FIG. 10 is a luminance distribution diagram of an image acquired in a characteristic determination experiment. [Figure 22] FIG. 2 is a flowchart showing an image processing method for the endoscope apparatus of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] An image processing flow diagram of the image processing method for the endoscopic device of the present invention is shown in Figure 11. In the image processing of the endoscopic device and image processing method for the endoscopic device of the present invention, VEViD (Vision Enhancement via Virtual Diffraction and Coherent Detection) processing, which is a visual enhancement process using virtual diffraction and coherent detection that utilizes the functions of low-light enhancement and color enhancement described in Non-Patent Document 3, is performed on image data 31 acquired by a camera by a VEViD processing unit 32, and image output data 33 is obtained from the controller.

[0016] Figure 12 shows the results of VEViD processing on the image data with an exposure time of 40 milliseconds shown in Figure 8. The image shown in Figure 8 is a black and white image of a color image for patent application purposes, but since VEViD processing is effective for processing color images, the image captured with an exposure time of 40 milliseconds was subjected to VEViD processing, and the resulting color image was converted to black and white, resulting in the image data shown in Figure 12. From the image shown in FIG. 12, it can be seen that the VEViD processing has a function of converting low-luminance images into image data with sufficient luminance. The image shown in Fig. 12 has 600 pixels in both the vertical and horizontal directions, and the luminance profile at the position 400 pixels from the left is plotted in Fig. 13. In the results shown in Fig. 13, the luminance profile around pixel position 330 changes sharply, indicating that the image is not blurred. Furthermore, in FIG. 13, the brightness values ​​of the bright white parts are approximately 200 to 230, and it can be seen that sufficient brightness data is obtained.

[0017] VEViD processing converts image data expressed in RGB space into hsv color space, then preserves the numerical values ​​of the h and v channels and processes the s channel space. The processing procedure is as follows: A Fourier transform is performed to obtain a complex image. Then, the phase information representing the frequency is multiplied by a complex exponential function. Here, a constant b may be added first to prevent processing instability. Next, an inverse Fourier transform is performed to obtain complex spatial image information. Then, the imaginary part of the pixel value is divided by the real part to obtain a quotient, and an inverse tangent process is performed on the quotient, thereby obtaining a phase value of each pixel value. In endoscopic systems that use a sequential imaging method in which a monochrome image sensor is used to capture images of the three primary colors (red, green, and blue) in sequence while switching the illumination wavelength, and then construct a color image, it is desirable to apply VEViD processing to the constructed color image, as this allows for faithful reproduction of color information.

[0018] 14 shows a configuration example 103 of a rigid endoscope device as an endoscope device of the present invention. In this configuration example 103 of a rigid endoscope device, an image captured by a camera 14 is subjected to VEViD processing in a VEViD processing execution unit 51 in a controller 61. An image 33 obtained by performing the VEViD processing is displayed on a monitor 12 or is saved in a storage area. 15 shows a configuration example 104 of a rigid endoscope device as an endoscope device of the present invention. In this configuration example 104 of a rigid endoscope device, VEViD processing is performed by a VEViD processing execution unit 52 disposed inside a camera 62. An image 33 obtained by performing VEViD processing is sent to a controller 11 and displayed on a monitor 12 or stored in a storage area.

[0019] 16 shows a configuration example 105 of a flexible endoscope device as an endoscope device of the present invention. In this configuration example 105 of a flexible endoscope device, an image captured by a camera 14 is subjected to VEViD processing in a VEViD processing execution unit 51 in a controller 63. An image 33 obtained by performing the VEViD processing is displayed on a monitor 2 or is saved in a storage area. 17 shows a configuration example 106 of a flexible endoscope device, which is also an endoscope device of the present invention. In this configuration example 106 of a flexible endoscope device, VEViD processing is performed by a VEViD processing execution unit 52 disposed inside an endoscope operation unit 63. An image 33 obtained by performing VEViD processing is sent to the controller 1 and displayed on the monitor 2 or saved in a storage area. 18 shows a configuration example 107 of an endoscope tip portion of a flexible endoscope device, which is also an endoscope device of the present invention. In this configuration example 107, VEViD processing is performed by a VEViD processing execution unit 52 located on the circuit board of a camera located at the tip of the flexible endoscope. An image 33 obtained by performing VEViD processing is sent to the controller 1 and displayed on the monitor 2 or saved in a storage area.

[0020] 12 and 13, by performing VEViD processing, even when capturing an image with an exposure time of 40 milliseconds, it is possible to obtain an image with the same brightness as when capturing an image with an exposure time of 180 milliseconds. Therefore, with the endoscopic device and image processing method for an endoscopic device of the present invention, even when the same endoscopic barrel, illumination method, imaging optical system, and imaging camera as conventional devices are used, it is possible to easily shorten the capturing time and reduce the risk of blurring in the still images captured by the surgeon. Furthermore, in the endoscopic device and image processing method for the endoscopic device of the present invention, the amount of illumination light can be reduced if the same endoscopic barrel, imaging optical system, imaging camera, and imaging time are used as in the conventional devices. This allows the area of ​​the light guide to be reduced, making it possible to reduce the diameter of the endoscope, reducing the burden on the patient and enabling access to areas that were previously inaccessible.

[0021] Figure 20 shows a black-and-white image obtained by converting an image captured using the experimental optical system 91 shown in Figure 19 into digital data. The experimental optical system 91 is an optical system in which the motor 24 and sample 23 of the experimental optical system 92 shown in Figure 6 have been removed, and a plain sample 22 has been placed at the capture position. The capture range of the sample 22 is larger than the diameter of the endoscope barrel 16, so in Figure 20 the center of the sample 22 is brighter than the outer periphery. In a flexible endoscope system, illumination is also emitted from the tip of the flexible endoscope, and the observation area is an area wider than the diameter of the flexible endoscope. Therefore, when no external illumination is used, the center of the image is brighter than the outer periphery, even with a flexible endoscope.

[0022] Figure 21 shows the brightness profile in the left and right center areas of the image shown in Figure 20. As can be seen from this figure, the brightness near the center is close to 250, but the outer periphery has a brightness value of around 55 to 80, which is less than one-third of the brightness in the center. If the camera is attached to the endoscope in a fixed position, the brightness distribution of the illumination in the field of view is fixed, so it is possible to correct the decrease in brightness in the peripheral areas of the captured image due to the lighting brightness distribution by image processing.

[0023] Fig. 22 shows an image processing flow diagram of the image processing method of the endoscopic device of the present invention when both illumination distribution correction and VEViD processing are applied to image 31 acquired by a camera. In image processing flow 42 shown in Fig. 22, image 31 acquired by the camera is subjected to illumination distribution correction processing by illumination distribution correction unit 34, and then VEViD processing is performed by VEViD processing unit 32 to obtain image output data 33. When the VEViD processing unit 32 uses both the correction based on illumination distribution and the VEViD processing, the correction based on illumination distribution must be performed first. The endoscope apparatus that performs the image processing flow shown in FIG. 22 is the apparatus shown in FIGS. 14 to 18, similar to the endoscope apparatus that performs the image processing flow shown in FIG.

[0024] In the above description of the present invention, an example was shown in which a color camera was used as the camera and a white LED was used as the LED used for illumination, but the endoscopic device and image processing method for the endoscopic device of the present invention can be applied to various endoscopic observation methods such as the NBI method, fluorescence endoscopy, and magnifying endoscopy shown in Non-Patent Document 2. [Industrial Applicability]

[0025] The endoscope apparatus and the image processing method for the endoscope apparatus of the present invention can be used for the examination and diagnosis of a patient using the endoscope apparatus. [Explanation of symbols]

[0026] 1, 11, 61, 62...Controller, 2, 12...Display, 3, 63...Endoscope operation unit, 4...Flexible cable 1, 5...Flexible cable 2, 6...Objective lens, 7...Illumination lens, 8...Air / water supply nozzle, 9...Suction port / forceps port, 13...Light source device, 14, 64...Camera, 15...Camera cable, 16...Endoscope barrel, 17...Light guide connection port, 18...Light guide, 19...Lens, 20...Connection Lens, 21...light guide fiber, 30...cross-section position, 22, 23...observation sample, 24...rotation motor, 31...camera image input unit, 32...VEViD image processing unit, 33...camera image output unit, 34...illumination distribution correction, 41, 42...image processing flow diagram, 51, 52...VEViD processing unit, 91, 92...characteristic evaluation optical system, 101, 105, 106, 107...flexible endoscope device, 102, 103, 104...rigid endoscope device

Claims

1. a light guide that guides illumination light emitted from a light source disposed outside the living body into the living body for the purpose of illuminating an observation field inside the living body; and an image transmission means for transmitting image information of an observation field inside a living body to an outside of the living body, An endoscope device that outputs an image that has been processed using a visual enhancement by virtual diffraction and coherent detection (VEViD) algorithm, which utilizes its functions in low-light enhancement and color enhancement, based on image information from an observation field of view within a living organism.

2. a light guide that guides illumination light emitted from a light source disposed outside the living body into the living body for the purpose of illuminating an observation field inside the living body; and an image transmission means for transmitting image information of an observation field inside a living body to an outside of the living body, Fourier transform processing for converting the acquired image information into complex image information; a process of multiplying the complex image information obtained by the process by a complex function; and processing by inverse Fourier transform processing. An endoscope device that outputs an image that has undergone visual enhancement image processing using virtual diffraction and coherent detection, utilizing its functions in low-light enhancement and color enhancement.

3. In the above image processing, Converts an image from RGB color space representation to hsv color space, preserving the values ​​of the h and v channels. Image processing is performed on the s-channel space. The endoscope device according to claim 2 .

4. a light guide that guides illumination light emitted from a light source disposed outside the living body into the living body for the purpose of illuminating an observation field inside the living body; an image transmission means for transmitting image information of an observation field inside a living body to an outside of the living body, An image processing method for an endoscope device, characterized by processing image information of an observation field of view within a living body using a visual enhancement by virtual diffraction and coherent detection (VEViD) algorithm that utilizes its functions in low-light enhancement and color enhancement.

5. a light guide that guides illumination light emitted from a light source disposed outside the living body into the living body for the purpose of illuminating an observation field inside the living body; an image transmission means for transmitting image information of an observation field inside a living body to an outside of the living body, Fourier transform processing for converting the acquired image information into complex image information; a process of multiplying the complex image information obtained by the process by a complex function; and processing by inverse Fourier transform processing. An image processing method for an endoscope system, characterized by comprising visual enhancement image processing by virtual diffraction and coherent detection utilizing its functions in low-light enhancement and color enhancement.

6. In the above image processing, Converts an image from RGB color space representation to hsv color space, preserving the values ​​of the h and v channels. Image processing is performed on the s-channel space. The image processing method for an endoscope apparatus according to claim 5.

Images