Endoscopic system and its operating method

Abstract

Provided are an endoscope system and an operation method therefor that enable highly-reliable detection of a region of interest.　An endoscope system (10) for illuminating a subject and imaging light from the subject acquires an inspection image on the basis of an image signal obtained from imaging by an endoscope (12), divides the inspection image into a plurality of regions as an input image, inputs an input image (102) divided into the plurality of regions to a first classifier (110), outputs region evaluation values (112) corresponding to the plurality of regions, inputs, to a second classifier (120), an input image (121) in which the plurality of regions are assigned with the region evaluation values, and outputs a lesion evaluation value (122).

Description

Technical Field

[0001] The present invention relates to an endoscope system that performs image analysis and an operating method thereof.

Background Art

[0002] In the current medical field, medical image processing systems that use medical images, such as an endoscope system including a light source device, an endoscope, and a processor device, are widespread. In recent years, by extracting a region of interest that may be a lesion from a medical image and performing image analysis on the extracted region of interest, diagnostic information regarding the pathological condition has been obtained.

[0003] In order to more accurately perform image analysis on a region of interest, a technique of dividing an image and evaluating the region of interest with respect to the divided image is known (Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a medical image used for detecting a region of interest, in addition to the region of interest such as a lesion, structures or artifacts that reduce the analysis accuracy, such as dark portions, blurs, residues, and specular reflections, other than the region of interest, may be reflected. The presence of such structures or artifacts hinders the detection of the region of interest and is one of the factors that reduce the detection accuracy of the region of interest.

[0006] If evaluation values ​​are calculated for images with low evaluation suitability that contain factors that reduce detection accuracy, the evaluation results will be unreliable, even if the area of ​​interest is included. Therefore, it is necessary to improve the detection accuracy of the area of ​​interest by identifying the structures that reduce detection accuracy and then detecting the area of ​​interest from the image.

[0007] The present invention aims to provide an endoscopic system and a method for operating the same that can reliably detect a region of interest. [Means for solving the problem]

[0008] The endoscope system of the present invention is an endoscope system that illuminates a subject and images the light from the subject, and comprises an endoscope and an image control processor. The image control processor acquires an examination image based on the image signal captured by the endoscope, divides the examination image into multiple regions as an input image, inputs the input images divided into multiple regions to a first classifier to output region evaluation values ​​for the multiple regions, and inputs the input image with region evaluation values ​​attached to the multiple regions to a second classifier to output lesion evaluation values.

[0009] The image control processor preferably inputs an input image consisting of extracted regions selected from multiple regions based on region evaluation values ​​to a second classifier and outputs lesion evaluation values.

[0010] The region evaluation value is preferably a numerical value that quantifies the presence or absence of evaluation inhibiting factors that are unsuitable for outputting lesion evaluation values ​​on the input image and that reduce the accuracy of the output of lesion evaluation values, as well as the effect that evaluation inhibiting factors have on the output of lesion evaluation values.

[0011] The image control processor preferably extracts multiple regions whose region evaluation value is greater than or equal to a threshold. Multiple regions whose region evaluation value is less than the threshold preferably contain one of the following: puddles, blood pools, liquid pools, bubbles, distortion, blur, reflection, or capping.

[0012] The system is equipped with a light source processor, which controls the emission of a first illumination light and a second illumination light having different emission spectra, and when it automatically switches between a first illumination period in which the first illumination light is emitted and a second illumination period in which the second illumination light is emitted, it is preferable that the first illumination light is emitted in a first emission pattern and the second illumination light is emitted in a second emission pattern, and the image control processor uses either a first illumination light image based on the first illumination light or a second illumination light image based on the second illumination light as the input image.

[0013] The image control processor preferably uses the second illumination light image as its input image. The first illumination light is preferably white light, and the second illumination light is preferably light with a peak wavelength of 410 nm ± 10.

[0014] Preferably, the system includes a display, and the image control processor displays the lesion evaluation value, output based on the second illumination light image acquired in the frame immediately preceding the acquisition of the first illumination light image, superimposed on the first illumination light image as a numerical value or color on the display.

[0015] The image control processor preferably, when the display comprises two different displays, displays the first illumination light image on the first screen of the first display and displays the lesion evaluation value output based on the second illumination light image acquired in the frame immediately preceding the acquisition of the first illumination light image on the second screen of the second display; or, when the display comprises only one specific display, displays the first illumination light image on the first screen of the specific display and displays the lesion evaluation value on the second screen of the specific display.

[0016] The first light emission pattern is preferably one of the following: a first A light emission pattern in which the number of frames in each first illumination period is the same, and a first B light emission pattern in which the number of frames in each first illumination period is different.

[0017] The second emission pattern is preferably one of the following: pattern 2A, in which the number of frames in each second illumination period is the same and the emission spectrum of the second illumination light is the same in each second illumination period; pattern 2B, in which the number of frames in each second illumination period is the same and the emission spectrum of the second illumination light is different in each second illumination period; pattern 2C, in which the number of frames in each second illumination period is different and the emission spectrum of the second illumination light is the same in each second illumination period; and pattern 2D, in which the number of frames in each second illumination period is different and the emission spectrum of the second illumination light is different in each second illumination period.

[0018] Preferably, the image control processor calculates an image evaluation value for one input image using lesion evaluation values ​​for multiple regions output based on one input image, calculates a site evaluation value for multiple input images using lesion evaluation values ​​for multiple regions output based on multiple input images, and calculates an overall evaluation value using the image evaluation values ​​and / or site evaluation values ​​attached to all input images for which lesion evaluation values ​​were output.

[0019] The image control processor preferably determines the shape, size, and / or number of divisions for the input image based on the magnification at which the examination image was acquired. The lesion evaluation value is preferably output based on the evaluation index for ulcerative colitis.

[0020] The operating method of the endoscope system of the present invention includes an endoscope that illuminates a subject and images light from the subject, and an image control processor. The image control processor includes steps of obtaining an inspection image based on the image signal captured by the endoscope, dividing the inspection image as an input image into a plurality of regions, inputting the input image divided into the plurality of regions into a first classifier, and outputting a region evaluation value for the plurality of regions, and inputting the input image with the region evaluation value assigned to the plurality of regions into a second classifier and outputting a lesion evaluation value.

[0021] The endoscope system of the present invention is an endoscope system that illuminates a subject and images light from the subject, and includes an endoscope and an image control processor. The image control processor obtains an inspection image based on the image signal captured by the endoscope, inputs the inspection image as an input image into a third classifier, outputs a removal image in which a specific region of the input image is removed, divides the removal image into a plurality of regions, inputs the removal image divided into the plurality of regions into a second classifier, and outputs a lesion evaluation value.

Effect of the Invention

[0022] According to the present invention, it is possible to provide an endoscope system and an operating method thereof that can detect a highly reliable attention region.

Brief Description of the Drawings

[0023] [Figure 1] It is an explanatory diagram of the configuration of the endoscope system. [Figure 2] It is a block diagram showing the functions of the endoscope system. [Figure 3] It is a graph showing the spectra of purple light V, blue light B, green light G, and red light R. [Figure 4] It is an explanatory diagram showing the first A emission pattern or the second A pattern in the enhanced observation mode. [Figure 5] It is an explanatory diagram showing the first B emission pattern in the enhanced observation mode. [Figure 6] It is an explanatory diagram showing the second B pattern in the enhanced observation mode. [Figure 7] This is an explanatory diagram showing the 2C pattern in enhanced observation mode. [Figure 8] This is an explanatory diagram showing the 2D pattern in enhanced observation mode. [Figure 9] This graph shows the spectral transmittance of each color filter in the image sensor. [Figure 10] This is an explanatory diagram showing the first imaging period and the second imaging period. [Figure 11] This graph shows the spectrum of the second illumination light. [Figure 12] This is an explanatory diagram illustrating the function of the image selection unit when the first illumination light image is the input image. [Figure 13] This is an explanatory diagram illustrating the function of the image selection unit when the second illumination light image is the input image. [Figure 14] This image diagram shows an example of how the image splitting unit has split an input image. [Figure 15] This image shows an example of a medical examination image containing a liquid puddle. [Figure 16] This image shows an example of an examination image containing bubbles. [Figure 17] This image shows an example of an inspection image containing reflections. [Figure 18A] This image shows an example where the edge of a cap is included in the inspection image. [Figure 18B] This image shows an example where the evaluation inhibition target is the entire cap, including the edge of the cap. [Figure 19] This is a block diagram illustrating the function of the first classifier. [Figure 20] This is an explanatory diagram illustrating an example of calculating region evaluation values ​​for each region of a divided input image. [Figure 21] This block diagram illustrates the function of the second classifier when input images with region evaluation values ​​attached to divided regions are input to the second classifier. [Figure 22] This is an explanatory diagram showing how to extract an extraction region. [Figure 23]This is an explanatory diagram showing an example of how lesion evaluation values ​​are calculated. [Figure 24] This is an explanatory diagram illustrating the function of the third classifier. [Figure 25] This is an explanatory diagram illustrating the function of the second classifier when a removal image is input. [Figure 26] This is an explanatory diagram illustrating an example of generating a superimposed image by superimposing lesion evaluation values ​​output from the examination image of the frame immediately preceding the display image. [Figure 27] This image diagram shows an example of displaying multiple superimposed images arranged in chronological order. [Figure 28] This image diagram shows examples of how the first and second screens are displayed. [Figure 29] This image diagram shows an example of displaying a display image and an input image with lesion evaluation values ​​output for each region on a display screen. [Figure 30] This image diagram shows an example of displaying the display image and the lesion evaluation value output for the entire image on a display screen. [Figure 31] This is an explanatory diagram showing an example of how to calculate image evaluation values. [Figure 32] This is an explanatory diagram showing an example of how to calculate body part evaluation values. [Figure 33] This is an explanatory diagram showing an example of how to calculate the overall evaluation value. [Figure 34] This is an explanatory diagram showing how to divide the regions of a foreground image and a background image. [Modes for carrying out the invention]

[0024] As shown in Figure 1, the endoscope system 10 comprises an endoscope 12, a light source device 14, a processor device 15, a computer 16, a display 17, and a user interface 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 15. The endoscope 12 has an insertion section 12a that is inserted into the body of the object to be observed, an operating section 12b provided at the base end of the insertion section 12a, and a bending section 12c and a tip section 12d provided at the tip end of the insertion section 12a. The bending section 12c bends by operating the angle knob 12e of the operating section 12b. The tip section 12d is directed in a desired direction by the bending movement of the bending section 12c.

[0025] The endoscope 12 is equipped with an optical system for forming an image of the subject and an optical system for illuminating the subject with illumination light. The control unit 12b is equipped with an angle knob 12e, an observation mode switch 12f, an image analysis mode switch 12g, a still image acquisition instruction switch 12h, and a zoom control unit 12i. The observation mode switch 12f is used to switch the observation mode. The still image acquisition instruction switch 12h is used to instruct the acquisition of a still image of the observed subject. The zoom control unit 12i is used to operate the zoom lens 42.

[0026] The light source device 14 generates illumination light. The display 17 outputs and displays images of the object being observed and information associated with those images. The user interface 19 has a keyboard, mouse, touchpad, microphone, etc., and has the function of accepting input operations such as function settings. The processor device 15 controls the system of the endoscope system 10 and performs image processing on image signals transmitted from the endoscope 12.

[0027] The endoscope system 10 has three observation modes: a first illumination observation mode, a second illumination observation mode, and an image analysis mode. When the observation mode switch 12f is pressed, the mode is switched via the image processing switch 54.

[0028] In the first illumination observation mode, the observation target is illuminated with normal light such as white light (first illumination light) and image is captured, displaying a first illumination light image with natural colors on the display 17. In the second illumination observation mode, the observation target is illuminated with special light (second illumination light) with a different wavelength band than normal light and image is captured, displaying a second illumination light image that emphasizes specific structures on the display 17. In the image analysis mode, the first illumination light and the second illumination light, which have different emission spectra, are switched and emitted. The image analysis mode analyzes the first illumination light image and / or the second illumination light image, and outputs and displays evaluation values ​​for areas of interest that may be lesions from the examination image.

[0029] In Figure 2, the light source device 14 comprises a light source unit 20 and a light source processor 21 that controls the light source unit 20. The light source unit 20 has, for example, multiple semiconductor light sources, and by turning each of them on or off, and controlling the amount of light emitted from each semiconductor light source when they are on, it emits illumination light to illuminate the object of observation. The light source unit 20 has four color LEDs: V-LED (Violet Light Emitting Diode) 20a, B-LED (Blue Light Emitting Diode) 20b, G-LED (Green Light Emitting Diode) 20c, and R-LED (Red Light Emitting Diode) 20d.

[0030] As shown in Figure 3, V-LED20a generates violet light V with a central wavelength of 405±10nm and a wavelength range of 380~420nm. B-LED20b generates blue light B with a central wavelength of 450±10nm and a wavelength range of 420~500nm. G-LED20c generates green light G with a wavelength range of 480~600nm. R-LED20d generates red light R with a central wavelength of 620~630nm and a wavelength range of 600~650nm.

[0031] The light source processor 21 controls V-LED20a, B-LED20b, G-LED20c, and R-LED20d. By independently controlling each of the LEDs 20a to 20d, the light source processor 21 can independently emit purple light V, blue light B, green light G, or red light R with varying light intensities. Furthermore, in the first illumination observation mode, the light source processor 21 controls each of the LEDs 20a to 20d to emit white light with a light intensity ratio of Vc:Bc:Gc:Rc between purple light V, blue light B, green light G, and red light R. Note that Vc, Bc, Gc, and Rc > 0.

[0032] Furthermore, the light source processor 21 controls each LED 20a to 20d to emit special light with a light intensity ratio of Vs:Bs:Gs:Rs between the special light and the short-wavelength narrow-band light V (purple light V), B (blue light B), G (green light G), and R (red light R). The light intensity ratio Vs:Bs:Gs:Rs is different from the light intensity ratio Vc:Bc:Gc:Rc used in the first illumination observation mode and is determined appropriately according to the purpose of observation.

[0033] Furthermore, when the light source processor 21 automatically switches between the first illumination light and the second illumination light during image analysis mode, it emits the first illumination light in a first emission pattern and the second illumination light in a second emission pattern. Specifically, the first emission pattern is preferably one of the following: as shown in Figure 4, the first A emission pattern in which the number of frames in the first illumination period is the same for each first illumination period, or as shown in Figure 5, the first B emission pattern in which the number of frames in the first illumination period is different for each first illumination period. In the figures, "time" represents the direction of time progression.

[0034] The second emission pattern is preferably one of the following: Pattern 2A, as shown in Figure 4, in which the number of frames in each second illumination period is the same and the emission spectrum of the second illumination light is the same in each second illumination period; Pattern 2B, as shown in Figure 6, in which the number of frames in each second illumination period is the same and the emission spectrum of the second illumination light is different in each second illumination period; Pattern 2C, as shown in Figure 7, in which the number of frames in each second illumination period is different and the emission spectrum of the second illumination light is the same in each second illumination period; and Pattern 2D, as shown in Figure 8, in which the number of frames in each second illumination period is different and the emission spectrum of the second illumination light is different in each second illumination period. The emission spectrum of the first illumination light may be the same or different in each first illumination period.

[0035] Here, it is preferable that the first illumination period be longer than the second illumination period, and that the first illumination period be two frames or more. For example, in Figure 4, when the first light emission pattern is the first A pattern and the second light emission pattern is the second A pattern (number of frames in the second illumination period: same, emission spectrum of the second illumination light: same), the first illumination period is two frames and the second illumination period is one frame. Since the first illumination light is used to generate the display image to be shown on the display 17, it is preferable that a bright image can be obtained by illuminating the object of observation with the first illumination light.

[0036] The first illumination light is preferably white light. White light includes so-called pseudo-white light, which is substantially equivalent to white light when imaging a subject using the endoscope 12, and is obtained by mixing purple light V, blue light B, green light G, or red light R as shown in Figure 3.

[0037] The details of the first and second light emission patterns, which are the switching patterns between the first and second illumination periods, will be described later, as they are determined based on the imaging control of the imaging sensor 43 by the imaging processor 44. A frame is a unit of time in the imaging sensor 43 that includes at least the period from a specific timing until the completion of signal readout.

[0038] In this specification, the light intensity ratio includes the case where the ratio of at least one semiconductor light source is 0 (zero). Therefore, it includes the case where one or more of the semiconductor light sources are not lit. For example, even when only one semiconductor light source is lit and the other three are not lit, such as when the light intensity ratio between violet light V, blue light B, green light G, and red light R is 1:0:0:0, a light intensity ratio is still considered to exist.

[0039] The light emitted from each LED 20a to 20d (see Figure 2) is incident on the light guide 23 via an optical path coupling section 22, which is composed of mirrors, lenses, etc. The light guide 23 propagates the light from the optical path coupling section 22 to the tip 12d of the endoscope 12.

[0040] The tip 12d of the endoscope 12 is equipped with an illumination optical system 30a and an imaging optical system 30b. The illumination optical system 30a has an illumination lens 31, and illumination light propagated by the light guide 23 is irradiated onto the object of observation via the illumination lens 31. The imaging optical system 30b has an objective lens 41 and an imaging sensor 43. Light from the object of observation due to the illumination light is incident on the imaging sensor 43 via the objective lens 41 and the zoom lens 42. As a result, an image of the object of observation is formed on the imaging sensor 43. The zoom lens 42 is a lens for magnifying the object of observation, and can be moved between the telephoto end and the wide-angle end by operating the zoom control unit 12i.

[0041] The image sensor 43 is a primary color sensor and has three types of pixels: B pixels (blue pixels) with a blue color filter, G pixels (green pixels) with a green color filter, and R pixels (red pixels) with a red color filter. As shown in Figure 9, the blue color filter BF mainly captures light in the blue band, specifically 3 The blue color filter (BF) transmits light in the 80-560nm wavelength range. The transmittance of the blue color filter (GF) peaks around 460-470nm. The green color filter (GF) primarily transmits light in the green wavelength range, specifically light in the 460-620nm wavelength range. The red color filter (RF) primarily transmits light in the red wavelength range, specifically light in the 580-760nm wavelength range.

[0042] Furthermore, the imaging sensor 43 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging processor 44 controls the imaging sensor 43. Specifically, the imaging processor 44 reads out signals from the imaging sensor 43, causing an image signal to be output from the imaging sensor 43. In the first illumination observation mode and image analysis mode, with white light illuminating the imaging sensor 43, the imaging processor 44 reads out signals, causing a Bc image signal to be output from the B pixels, a Gc image signal from the G pixels, and an Rc image signal from the R pixels of the imaging sensor 43. In the second illumination observation mode and image analysis mode, with special light illuminating the imaging sensor 43, the imaging processor 44 reads out signals, causing a Bs image signal to be output from the B pixels, a Gs image signal from the G pixels, and an Rs image signal from the R pixels of the imaging sensor 43.

[0043] In image analysis mode, as shown in Figure 10, the imaging processor 44 outputs a first image signal from the imaging sensor 43 by reading out the signal while the imaging sensor 43 is exposed to the first illumination light during the first illumination period. The period during which the first image signal is output is called the first imaging period. The first image signal includes the Bc image signal output from the B pixels, the Gc image signal output from the G pixels, and the Rc image signal output from the R pixels. Furthermore, during the second illumination period, the imaging processor 44 outputs a second image signal from the imaging sensor 43 by reading out the signal while the imaging sensor 43 is exposed to the second illumination light. The period during which the second image signal is output is called the second imaging period. The second image signal includes the Bs image signal output from the B pixels, the Gs image signal output from the G pixels, and the Rs image signal output from the R pixels.

[0044] The CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 45 (see Figure 2) performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the imaging sensor 43. The image signal that has passed through the CDS / AGC circuit 45 is converted into a digital image signal by the A / D (Analog / Digital) converter 46. The digital image signal after A / D conversion is input to the processor device 15.

[0045] In the processor device 15, the central control unit 55, which is composed of an image control processor, operates the program in the program memory, thereby realizing the functions of the image acquisition unit 50, DSP (Digital Signal Processor) 52, noise reduction unit 53, image processing switching unit 54, inspection image acquisition unit 60, image selection unit 90, and display control unit 200.

[0046] The image acquisition unit 50 acquires a color image input from the endoscope 12. The color image includes the blue signal (B image signal), green signal (G image signal), and red signal (R image signal) output from the B, G, and R pixels of the imaging sensor 43. The acquired color image is transmitted to the DSP 52. The DSP 52 performs various signal processing on the received color image, such as defect correction processing, offset processing, gain correction processing, matrix processing, gamma conversion processing, demosaicing processing, and YC conversion processing.

[0047] The noise reduction unit 53 applies noise reduction processing, such as the moving average method or the median filter method, to the color image that has undergone demosaicing or other processing by the DSP 52. The color image with reduced noise is input to the image processing switching unit 54.

[0048] The image processing switching unit 54 switches the destination of the image signal from the noise reduction unit 53 depending on the set mode. Specifically, when set to the first illumination observation mode, the image signal from the noise reduction unit 53 is input to the first illumination light image generation unit 70 of the inspection image acquisition unit 60. When set to the second illumination observation mode, the image signal from the noise reduction unit 53 is input to the second illumination light image generation unit 80. When set to image analysis mode, the image signal from the noise reduction unit 53 is input to both the first illumination light image generation unit 70 and the second illumination light image generation unit 80.

[0049] The following describes the image processing for the first illumination light image, the image processing for the second illumination light image, and the enhanced image display control processing performed on the image signal transmitted from the noise reduction unit 53 to the inspection image acquisition unit 60 via the image processing switching unit 54.

[0050] In the first illumination observation mode, the first illumination light image generation unit 70 performs image processing for the first illumination light image on the input Rc image signal, Gc image signal, and Bc image signal for one frame. Image processing for the first illumination light image includes color conversion processing such as 3x3 matrix processing, grayscale conversion processing, and 3D LUT (Look Up Table) processing, as well as color enhancement processing such as color enhancement and spatial frequency enhancement. The Rc image signal, Gc image signal, and Bc image signal that have undergone image processing for the first illumination light image are transmitted to the display control unit 200 as the first illumination light image.

[0051] In the second illumination observation mode, the second illumination light image generation unit 80 performs image processing for the second illumination light image on the Rs image signal, Gs image signal, and Bs image signal for one frame of input. The image processing for the second illumination light image is performed on the Bs image signal, Gs image signal, and Rs image signal obtained by emitting the second illumination light with the second illumination light emission spectrum S. Luminous The second illumination light emitted in spectrum S is preferably a light that emits violet light V (with a peak wavelength of, for example, 410 nm ± 10), as shown in Figure 11.

[0052] The image processing for the second illumination light image involves assigning the Bs image signal to the B, G, and R channels for display, and adjusting the color tone and gradation balance. The image processing for the second illumination light image includes color conversion processing such as 3x3 matrix processing, gradation conversion processing, and 3D LUT (Look Up Table) processing, as well as color enhancement processing such as color enhancement and spatial frequency enhancement. The image processing for the second illumination light image yields a second illumination light image in which blood vessels or structures at a specific depth are enhanced. The Rs image signal, Gs image signal, and Bs image signal, which have undergone the image processing for the second illumination light image, are transmitted to the display control unit 200 as the second illumination light image. The display control unit 200 displays the second illumination light image on the display 17.

[0053] In image analysis mode, the first illumination light image generation unit 70 applies the above-described image processing for the first illumination light image to the Rc image signal, Gc image signal, and Bc image signal for one frame to obtain the first illumination light image. Furthermore, the second illumination light image generation unit 80 applies the above-described image processing for the second illumination light image to the input Rs image signal, Gs image signal, and Bs image signal for one frame to obtain the second illumination light image. In this specification, claims, drawings, and abstract, the term "inspection image" simply refers to the first illumination light image and / or the second illumination light image.

[0054] In image analysis mode, the first illumination light image is normally transmitted to the display control unit 200 as a display image and displayed on the display 17. The second illumination light image is used to calculate lesion evaluation values, which will be described later. Alternatively, the system can be configured to transmit the second illumination light image to the display control unit 200 and display it on the display 17.

[0055] The image analysis mode is described below. In the image analysis mode, the examination image is divided into multiple regions, and then the lesion is analyzed. In the image analysis mode of this embodiment, there are two methods for outputting a lesion evaluation value for the final lesion. One is to send the examination image to the image division unit 100 to divide the image, evaluate the quality of the examination image for each divided region, and then evaluate the lesion in each divided region. The other is to input the examination image to the third classifier 130, output an image from which parts unsuitable for lesion evaluation have been removed in advance, divide that image, and then evaluate the lesion for each divided region.

[0056] In image analysis mode, the inspection images generated by the inspection image acquisition unit 60 are transmitted to the image selection unit 90. The image selection unit 90 selects input images from the inspection images to be input to the image splitting unit 100 or the third classifier 130.

[0057] In image analysis mode, as shown in Figure 10, a first illumination light image is obtained by outputting a first image signal during the first illumination period, and a second illumination light image is obtained by outputting a second image signal during the second illumination period. The image selection unit 90 selects the first illumination light image and / or the second illumination light image as the input image. The user can arbitrarily set which of the first illumination light image and / or the second illumination light image is used as the input image. The input image selected by the image selection unit 90 is transmitted to the image input unit 92 of the computer 16.

[0058] It is preferable to select the second illumination image as the input image. This is because the second illumination image emphasizes blood vessels, making it easier to recognize lesions than the first illumination image, thus improving the accuracy of the final lesion evaluation value.

[0059] If the first illumination light image is selected as the input image, for example, as shown in Figure 12, the first illumination light image Ws is selected as the input image by the image selection unit 90 from among the inspection images obtained in image analysis mode. The selected input image is input to the image division unit 100 or the third classifier 130. In the specific example shown in Figure 12, the first illumination light images of all frames obtained during the first illumination period are selected as input images, but it is also possible to select which frames to be selected as input images within the first illumination period. For example, the first illumination light image may be selected as the input image every other frame during the first illumination period, or the first illumination light image of the frame immediately before entering the second illumination period may be selected as the input image.

[0060] If the second illumination light image is selected as the input image, for example, as shown in Figure 13, the second illumination light images SPs are selected as input images by the image selection unit 90 from among the inspection images obtained in image analysis mode. The selected input images are input to the image segmentation unit 100 or the third classifier 130. In the specific example shown in Figure 13, the second illumination light images of all frames obtained during the second illumination period are selected as input images, but it is also possible to select which frames to be selected as input images during the second illumination period. For example, the second illumination light image may be selected as the input image during the second illumination period once or every few times.

[0061] Computer 16 includes an image input unit 92, an image segmentation unit 100, a first classifier 110, a second classifier 120, a third classifier 130, an image evaluation value calculation unit 140, a body part evaluation value calculation unit 150, and an overall evaluation value calculation unit 160 (see Figure 2). In computer 16, a control unit (not shown) composed of an image control processor operates a program in the program memory, thereby realizing the functions of the image input unit 92, image segmentation unit 100, first classifier 110, second classifier 120, third classifier 130, image evaluation value calculation unit 140, body part evaluation value calculation unit 150, and overall evaluation value calculation unit 160. Note that computer 16 and / or light source Processor 22 may be included in the processor device 15.

[0062] The first classifier 110, the second classifier 120, and the third classifier 130 are classifiers generated using machine learning and / or image processing. Deep learning is preferred for machine learning, for example, a multilayer convolutional neural network is preferred. In addition to deep learning, machine learning includes decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, neural network-based learning, generative adversarial networks, and the like.

[0063] The following describes the case when an input image is transmitted to the image division unit 100. The image division unit 100 divides the input image into multiple regions. The input image is divided into multiple grid-like regions, for example, as shown in Figure 14. In the specific example in Figure 14, the input image is divided into 16 regions, from region A to region P. Note that the shape of the division is not limited to a grid; any shape that can subdivide the input image is acceptable, such as a polygon shape like a hexagon or a shape consisting of curves. Furthermore, the multiple regions may be cut into random shapes with different shapes and sizes depending on the location.

[0064] The image segmentation unit 100 preferably divides the input image into segments of a size that allows for the determination of potential lesion regions or evaluation-inhibiting objects that are not regions of interest. Evaluation-inhibiting objects refer to structures or artifacts that are unsuitable for outputting lesion evaluation values, which are values ​​that evaluate the region of interest in the input image, and that cause a decrease in evaluation accuracy. For example, this includes specific pools 101 such as puddles of water or blood covering the object of observation, or pools of excess drug solution, as shown in Figure 15, and bubbles 107 as shown in Figure 16. It also includes distortion (distortion due to the objective lens used to image the object of observation), blurring of the image, and reflections 103 such as halation, as shown in Figure 17. Furthermore, a cap (hood) is attached to the tip 12d of the endoscope 12, and the edge 104 of the cap is placed on the input image. a If the cap is visible in the image, the evaluation-inhibiting object will include the edge 104a of the cap, as shown in Figure 18A, or the entire cap 104b including the edge 104a, as shown in Figure 18B.

[0065] As shown in Figure 19, the input image 102, which has been divided into multiple regions by the image division unit 100, is input to the first classifier 110. The first classifier 110 outputs region evaluation values ​​112 for each of the multiple regions in the input image. The region evaluation value 112 is an evaluation value for the quality of the divided input image 102 itself. That is, in order to improve the accuracy of the output of the lesion evaluation value described later, it is a numerical value that quantifies whether or not the divided input image 102 contains an evaluation inhibiting object, and if so, to what extent it affects the output of the lesion evaluation value.

[0066] In the example in Figure 20, region A of the divided input image 102 has an output region evaluation value of 0.5. Similarly, the other regions have the following output values: region B has a region evaluation value of 0.9, region C has a region evaluation value of 0.05, region D has a region evaluation value of 0.5, region E has a region evaluation value of 0.8, region F has a region evaluation value of 0.95, region G has a region evaluation value of 0.95, region H has a region evaluation value of 0.8, region I has a region evaluation value of 0.8, region J has a region evaluation value of 0.95, region K has a region evaluation value of 0.95, region L has a region evaluation value of 0.8, region M has a region evaluation value of 0.5, region N has a region evaluation value of 0.9, region O has a region evaluation value of 0.9, and region P has a region evaluation value of 0.5.

[0067] If the region does not contain any evaluation-inhibiting objects and the region of interest can be appropriately determined, the first classifier 110 calculates a high region evaluation value. On the other hand, if the region contains evaluation-inhibiting objects, the first classifier 110 calculates a low region evaluation value. In other words, the region evaluation value is a value that fluctuates depending on what kind of evaluation-inhibiting objects are included in the input image 102 and to what extent.

[0068] In the specific example shown in Figure 20, regions B (region evaluation value 0.9), E (region evaluation value 0.8), F (region evaluation value 0.95), G (region evaluation value 0.95), H (region evaluation value 0.8), I (region evaluation value 0.8), J (region evaluation value 0.95), K (region evaluation value 0.95), L (region evaluation value 0.8), N (region evaluation value 0.9), and O (region evaluation value 0.9) are normal mucosa, and are regions where the region of interest can be appropriately determined. On the other hand, region C (region evaluation value 0.05) has a reflection of 103. Regions A, D, M, and P are at the edge of the endoscopic field of view, and therefore distortion occurs. Therefore, regions A, C, D, M, and P, which contain evaluation inhibiting targets, have lower region evaluation values ​​than regions B, E, F, G, H, I, J, K, L, N, and O, which do not contain evaluation inhibiting targets.

[0069] As shown in Figure 21, the input image 121, on which region evaluation values ​​are assigned to multiple regions, is input to the second classifier 120. The second classifier 120 outputs lesion evaluation values ​​122 for the input image. The second classifier 120 may output lesion evaluation values ​​for each individual region of the divided input image, or it may output lesion evaluation values ​​for the entire input image.

[0070] The lesion evaluation value is an evaluation value that represents the degree of lesion in the input image. Preferably, the second classifier 120 is trained on examination images that have been given diagnostic results from a specialist in endoscopic image diagnosis. Furthermore, it is preferable that the second classifier 120 is trained on examination images that have been given diagnostic results from both endoscopic image diagnosis and pathological examination (biopsy).

[0071] The lesion evaluation value is preferably an evaluation value based on a known scoring method. The lesion evaluation value is preferably output based on an evaluation index for ulcerative colitis. For example, a score from UCEIS (Ulcerative colitis endoscopic index of severity) may be used.

[0072] UCEIS is characterized by its evaluation of ulcerative colitis activity in endoscopic images by separately evaluating vascular visibility, bleeding, erosion, and ulcers at the site with the strongest findings, as shown in Table 1 (Yasuo Suzuki, Ikuhito Hirai, et al. Collection of Disease Activity Assessment Indicators for Inflammatory Bowel Disease, Second Edition. Grant-in-Aid for Scientific Research from the Ministry of Health, Labour and Welfare, Policy Research Project on Intractable Diseases, etc. "Survey and Research on Intractable Inflammatory Bowel Disorders (Suzuki Group)". March 2020; modified from 16). For example, a total UCEIS score of 2 to 4 is evaluated as mild, 5 to 6 as moderate, and 7 to 8 as severe.

[0073] [Table 1]

[0074] Other assessment indicators for ulcerative colitis may also be used for lesion evaluation, such as the severity classification of the Research Group on Intractable Inflammatory Bowel Disease, the Truelove-Witts index, Powell-Tuck index, Seo index, Lichtiger index, Sutherland index (disease activity index), Mayo score, Rachmilewitz index, Pouchitis disease activity index (PDAI), Baron index, modified Baron index, Matts classification, Simple clinical colitis activity index (SCCAI), Pediatric ulcerative colitis activity index (PUCAI), Geboes histopathology score (GHS), etc.

[0075] Lesion evaluation values ​​may be output based on feature quantities. Preferably, the feature quantities are classified by whether the observed object is located in at least one of the superficial, middle, or deep layers. Preferably, the feature quantities are the shape, color, or values ​​obtained from such shapes and colors of the observed object. Examples of feature quantity items include vascular density, vascular shape, number of vascular branches, vascular diameter, vascular length, degree of vascular tortuosity, depth of vascular penetration, glandular shape, glandular opening shape, glandular length, degree of glandular tortuosity, color information, and brightness. Preferably, the feature quantity is a value that is at least one of these, or a combination of two or more of these. However, the feature quantity items are not limited to these and may be added as appropriate depending on the usage situation.

[0076] The lesion evaluation value may be output based on the presence and extent of the area of ​​interest. Areas of interest include, for example, inflamed areas (including areas with changes such as bleeding or atrophy in addition to so-called inflammation), benign tumors, malignant tumors, colonic diverticula, treatment scars (EMR (Endoscopic mucosal resection) scars, ESD (Endoscopic Submucosal Dissection) scars, clipping sites), bleeding points, perforations, vascular dysplasia, cauterization marks due to heating, or marked areas marked with coloring agents such as colorants or fluorescent agents, or areas including biopsy sites where biopsies have been performed. In other words, areas containing lesions, areas potentially containing lesions, areas where some kind of procedure such as biopsy has been performed, treatment instruments such as clips or forceps, or dark areas (behind folds, deep inside the lumen where observation light is difficult to reach), and other areas that require detailed observation regardless of the possibility of lesions.

[0077] Preferably, the input image input to the second classifier 120 is an input image consisting of extracted regions extracted based on region evaluation values ​​assigned to regions divided by the image division unit 100. In this case, of the multiple divided regions, the extracted region 124 (the white area in Figure 22) as shown in Figure 22 is input to the second classifier 120. In Figure 22, the regions 123 that were not selected are shown as areas with diagonal lines.

[0078] If region evaluation values ​​are assigned to each segmented region of the input image, the images to be input to the second classifier 120 may be extracted based on the region evaluation values. For example, if there is a region for which a region evaluation value lower than a certain value compared to adjacent regions exists, that region may not be extracted. With the above configuration, the accuracy of the evaluation of lesions on the image can be improved by pre-extracting regions unsuitable for evaluating lesions on the examination image before performing the evaluation of the lesion.

[0079] Alternatively, a threshold can be set as a criterion for extracting extraction regions, and regions above the threshold can be input to the second classifier 120 as extraction regions. For example, if the threshold for extraction regions is set to 0.6, in the specific example in Figure 22, the extraction region 124 is the region shown by the diagonal lines, and the region 123 that was not selected as extraction region 124 is the white-out region. Regions below the threshold are regions that contain evaluation-inhibiting objects such as puddles, blood pools, liquid pools, bubbles, distortion, blur, reflections, or caps.

[0080] Furthermore, the second classifier 120 may output lesion evaluation values ​​for regions that have been assigned region evaluation values ​​without extracting regions. Using the specific example of lesion evaluation value output in Figure 23 (see also Figure 20), regions B (region evaluation value 0.9), E (region evaluation value 0.8), F (region evaluation value 0.95), G (region evaluation value 0.95), H (region evaluation value 0.8), I (region evaluation value 0.8), J (region evaluation value 0.95), K (region evaluation value 0.95), L (region evaluation value 0.8), N (region evaluation value 0.9), and O (region evaluation value 0.9) are normal mucosa and are regions where the region of interest can be appropriately determined. On the other hand, region C (region evaluation value 0.05) has a reflection of 103. Regions A, D, M, and P are at the edge of the endoscopic field of view and therefore distort occurs.

[0081] In the specific example of lesion evaluation value output in Figure 23, region A is output as a lesion evaluation value of 0.01. Similarly, in the other regions, region B is output as 0.05, region C as 0.01, region D as 0.01, region E as 0.01, region F as 0.02, region G as 0.2, region H as 0.01, region I as 0.01, region J as 0.01, region K as 0.95, region L as 0.01, region M as 0.7, region N as 0.01, region O as 0.01, and region P as 0.01. Regions K and M, which include region 126 of interest, are output as lesion evaluation values ​​higher than the other regions.

[0082] In this case, regions B (region evaluation value 0.9, lesion evaluation value 0.05), E (region evaluation value 0.8, lesion evaluation value 0.01), F (region evaluation value 0.95, lesion evaluation value 0.02), G (region evaluation value 0.95, lesion evaluation value 0.2), H (region evaluation value 0.8, lesion evaluation value 0.01), I (region evaluation value 0.8, lesion evaluation value 0.01), J (region evaluation value 0.95, lesion evaluation value 0.01), K (region evaluation value 0.95, lesion evaluation value 0.95), L (region evaluation value 0.8, lesion evaluation value 0.01), N (region evaluation value 0.9, lesion evaluation value 0.01), and O (region evaluation value 0.9, lesion evaluation value 0.01), which do not contain the evaluation inhibition target, can output lesion evaluation values ​​with high accuracy.

[0083] On the other hand, in region C (region evaluation value 0.05, lesion evaluation value 0.01), the reflection 103 area has high brightness, making the image almost invisible, thus significantly reducing the accuracy of the lesion evaluation value output. Regions A (region evaluation value 0.5, lesion evaluation value 0.01), D (region evaluation value 0.5, lesion evaluation value 0.01), M (region evaluation value 0.5, lesion evaluation value 0.7), and P (region evaluation value 0.5, lesion evaluation value 0.01) show distortion, resulting in a slightly lower accuracy of the lesion evaluation value output. However, as with region M (region evaluation value 0.5, lesion evaluation value 0.7), it is still possible to determine the presence or absence of lesions to some extent. When outputting lesion evaluation values ​​without extracting regions, regions with low region evaluation values, such as regions C, A, D, M, and P, may be marked with a frame 125 or color, as shown in Figure 23. With the above configuration, it is possible to visualize the quality of an image of a certain quality and evaluate lesions while ensuring a certain level of quality.

[0084] The image input to the second classifier 120 may be an image from which evaluation-inhibiting elements have been removed in advance using machine learning. As shown in Figure 24, when the input image 131 is input to the third classifier 130, a removed image 132 from which evaluation-inhibiting elements have been removed is output. The removed image 132 is sent to the image splitting unit 100 and split into multiple regions. As shown in Figure 25, the removed image 132, which has been split into multiple regions, is input to the second classifier 120. The second classifier 120 outputs a lesion evaluation value 122 for the removed image 132. The second classifier 120 may output a lesion evaluation value for each of the multiple regions of the removed image 132, or it may output a lesion evaluation value for the entire frame of the removed image 132. With the above configuration, regardless of the image splitting method, it is possible to perform an evaluation of the lesion after removing parts unsuitable for evaluation in advance, thereby improving the reliability of the evaluation of the lesion.

[0085] It is preferable that the lesion evaluation value is displayed together with the display image. The display image and lesion evaluation value are transmitted to the display control unit 200, and as shown in Figure 26, it is preferable to superimpose the lesion evaluation value 172a output from the inspection image 172 of the frame immediately preceding the display image 171 onto the display image 171 to generate and display the superimposed image 173. The lesion evaluation value shown in the superimposed image may be displayed numerically. Alternatively, as shown in the superimposed image 173 in Figure 26, the color display may be changed according to the lesion evaluation value and displayed as a heat map. In the lesion evaluation value 172a in Figure 26, the density of the diagonal lines indicates the height of the lesion evaluation value (area with high lesion evaluation value 172b, area with moderate lesion evaluation value 172c, area with low lesion evaluation value 172d, and normal area with extremely low lesion evaluation value 172e). In the superimposed image 173 shown in Figure 26, the lesion evaluation value 172a is superimposed on the display image 171. In Figure 26, differences in lesion evaluation values ​​are shown by differences in the intensity of the diagonal lines, but in reality, they are displayed with differences in the intensity of the colors. When displaying lesion evaluation values ​​with color, for example, areas with high lesion evaluation values ​​can be displayed in red, areas with moderate lesion evaluation values ​​in yellow, and areas with low lesion evaluation values ​​in blue. With the above configuration, areas with a high degree of lesion can be easily identified, and the diagnostic burden on the user can be reduced when it is necessary to examine many images.

[0086] When generating superimposed images, it is preferable that the display image is the first illumination image and the input image for outputting lesion evaluation values ​​is the second illumination image. The first illumination image is an endoscopic image illuminated with white light and is an image that is usually familiar to physicians. On the other hand, the second illumination image is an endoscopic image illuminated with special light, with blood vessels emphasized, but it is an image that requires training for accurate interpretation. Therefore, by outputting lesion evaluation values ​​using the second illumination image, which can determine lesions with higher accuracy, and superimposing it onto the first illumination image in the frame immediately following the output of the lesion evaluation values, the user can confirm the lesion evaluation values ​​on the first illumination image.

[0087] Furthermore, as shown in Figure 27, multiple superimposed images may be displayed in chronological order. In this case, to make it possible to identify areas with a relatively high lesion evaluation value along the timeline, a frame 125 may be added to the areas with a high lesion evaluation value. In the specific example in Figure 27, superimposed images a173a, b173b, and c173c are acquired in that order, and the lesion evaluation values ​​of the images are shown (areas with high lesion evaluation values ​​172b, areas with moderate lesion evaluation values ​​172c, areas with low lesion evaluation values ​​172d, and normal areas with extremely low lesion evaluation values ​​172e). Of these, areas with high lesion evaluation values ​​172b and areas with moderate lesion evaluation values ​​172c are marked with a frame 125. Additionally, each area of ​​interest may be numbered or symbolized for display. With the above configuration, it is possible to display multiple images while aligning areas with a high degree of lesion, making it easier to find the same lesion on images obtained in chronological order.

[0088] The display method when the display 17 consists of a first display 17a and a second display 17b, and the first display 17a and the second display 17b are electrically connected to the processor device 15, will be described. As shown in Figure 28, the display image is shown on the first screen 180 of the first display 17a, and the lesion evaluation value is shown on the second screen 182 of the second display 17b. to It is also possible to display the following. As shown in Figure 28, when displaying a display image and lesion evaluation values ​​using the first and second screens, it is preferable that the display image displayed on the first screen is the first illumination light image, and the image outputting the lesion evaluation values ​​displayed on the second screen is the second illumination light image of the frame immediately preceding the display image. Alternatively, the display image may be displayed on the first screen and the superimposed image on the second screen, or the superimposed image may be displayed on the first screen and the image outputting the lesion evaluation values ​​on the second screen. Furthermore, a third screen may be provided to simultaneously display the display image, the image outputting the lesion evaluation values, and the superimposed image.

[0089] A display method will be described when the display 17 has only one specific display 17c. As shown in Figure 29, a display image 191 as the first screen 180 and an input image 192 outputting lesion evaluation values ​​as the second screen 182 may be displayed on the specific display 17c as a single display screen 190. When the display screen 190 is displayed on the specific display 17c, the display image is preferably the first illumination light image, and the image outputting lesion evaluation values ​​is preferably the second illumination light image of the frame immediately preceding the display image. Note that the specific display 17c may display two or more of the display image, the image outputting lesion evaluation values, and the superimposed image. When the specific display 17c displays two or more of the image outputting lesion evaluation values ​​and the superimposed image, the display size may be changed. For example, as shown in Figure 29, when displaying two or more images including the display image, the display image may be displayed at a larger size compared to the other images. The input image 192, which outputs the lesion evaluation values ​​for the specific example in Figure 29, may display the lesion evaluation values ​​numerically or by color, as shown in Figure 28. With the above configuration, it is possible to compare the display image or superimposed image with the image used for image analysis.

[0090] Furthermore, as shown in Figure 30, if lesion evaluation values ​​are output for the entire divided input image, the display image 191 and lesion evaluation values ​​193 may be displayed on a single display screen 190.

[0091] When the second classifier 120 outputs a lesion evaluation value for each region of an input image or removal image that has been divided into multiple regions, the image evaluation value for any single frame of the input image may be calculated using the lesion evaluation values ​​assigned to multiple regions. The input image or removal image with lesion evaluation values ​​assigned to each region is transmitted to the image evaluation value calculation unit 140, where the image evaluation value is calculated. The image evaluation value may be the average or sum of the lesion evaluation values ​​assigned to each region of the input image or removal image, or, as shown in the example of image evaluation value calculation in Figure 31, the highest lesion evaluation value among the lesion evaluation values ​​141 assigned to each region may be used as the image evaluation value 142.

[0092] Furthermore, lesion evaluation values ​​may be calculated for multiple frames of input images using lesion evaluation values ​​attached to input images or removal images that have been divided into multiple regions of multiple frames. Multiple frames of input images or removal images, each with a lesion evaluation value attached to its respective region, are transmitted to the site evaluation value calculation unit 150, where the site evaluation values ​​are calculated. Site evaluation values ​​are obtained for multiple frames of input images or removal images obtained for each part of the large intestine, such as the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, etc. The multiple frames of input images transmitted to the site evaluation value calculation unit 150 may be arbitrarily selected by the user. In other words, the "parts" for which site evaluation values ​​are calculated may be set by any method the user chooses. Alternatively, the computer 16 may automatically recognize anatomical parts such as the rectum, sigmoid colon, descending colon, transverse colon, and ascending colon from the examination images (input images or removal images), and the site evaluation value calculation unit 150 may calculate the site evaluation value corresponding to each site from the lesion evaluation values ​​attached to the input images or removal images of the frames corresponding to each site.

[0093] The site evaluation value is the average or sum of the lesion evaluation values ​​assigned to each region of the input images of multiple frames. Alternatively, it may be the average or sum of the image evaluation values, or the highest image evaluation value among the set sites may be used as the site evaluation value. An example of calculating the site evaluation value shown in Figure 32 will be explained. Images 151a, 151b, and 151c in Figure 32 are images arranged in chronological order in which lesion evaluation values ​​were obtained (when the endoscope 12 is inserted through the anus and the examination is performed, images 151c, 151b, and 151a are obtained in that order). In images 151a, 151b, and 151c in Figure 32, the density of the diagonal lines indicates the height of the lesion evaluation value. The respective lesion evaluation values ​​151 from images 151a, 151b, and 151c obtained in each region of the descending colon are transmitted to the site evaluation value calculation unit 150, and the average is taken to obtain the site evaluation value 152 for one site (descending colon in Figure 32). In the example in Figure 32, the local evaluation value is calculated for the descending colon, so the local evaluation value of 152 may be used as the descending colon evaluation value. Following this, the evaluation values ​​for the rectum, sigmoid colon, transverse colon, ascending colon, etc., may also be calculated.

[0094] Alternatively, the overall evaluation value may be calculated using the lesion evaluation values ​​attached to the input images or removal images, which are divided into multiple regions of all frames acquired during the examination. The input images or removal images, each with a lesion evaluation value attached to its respective region, are transmitted to the overall evaluation value calculation unit 160, where the overall evaluation value is calculated. The overall evaluation value may also be calculated using image evaluation values ​​or site evaluation values.

[0095] An example of calculating the overall evaluation value shown in Figure 33 will be explained. Images 161a, 161b, 161c, 161d, 161e, and 161f in Figure 33 are images arranged in chronological order in which lesion evaluation values ​​were obtained (when the endoscope 12 is inserted through the anus and the examination is performed, images are obtained in the order of 161f, 161e, 161d, 161c, 161b, and 161a). In images 161a to 161f in Figure 33, the density of the diagonal lines indicates the level of the lesion evaluation value. The lesion evaluation values ​​161 calculated from each of the images 161a to 161f are sent to the overall evaluation value calculation unit 160, and the average is taken to obtain the overall evaluation value 162 for one site (descending colon in Figure 32). Alternatively, the transverse colon evaluation value may be calculated from images 161a and 161b, the descending colon evaluation value from images 161c, 161d, and 161e, the sigmoid colon evaluation value from image 161f, and the overall evaluation value 162 may be calculated using the transverse colon evaluation value, the descending colon evaluation value, and the sigmoid colon evaluation value.

[0096] The image division unit 100 preferably determines the shape, size, and / or number of divisions for the input image based on the magnification at which the inspection image was acquired. For example, as shown in Figure 34, the number and size of divisions are set to be larger for close-up images 210 with high observation magnification, and smaller for distant images 212 with low observation magnification, according to the size of the blood vessels 213. When the observation magnification is high, the area occupied by the region of interest and evaluation-inhibiting objects in the input image becomes larger, so it is good to make each divided region larger. On the other hand, when the observation magnification is low, the area occupied by the region of interest and evaluation-inhibiting objects in the input image becomes smaller, so making each divided region smaller can improve evaluation accuracy. It should also be noted that the shape, size, and / or number of divisions may be changed within the same image, such as increasing the number of divisions at the edges of the screen where blurring and distortion are likely to occur, and decreasing the number of divisions in the center of the image where it is easier to focus.

[0097] In this embodiment, the processor unit 15 and the computer 16 are described in an example where they are provided in the endoscope system 10, but the present invention is not limited thereto, and other medical devices may be used. Furthermore, the endoscope 12 may be a rigid or flexible endoscope. Also, part or all of the examination image acquisition unit 60 and / or the central control unit 55 of the endoscope system 10 can be provided in a medical image processing device that communicates with the processor unit 15 and cooperates with the endoscope system 10. For example, it can be provided in a diagnostic support device that acquires images captured by the endoscope 12 directly from the endoscope system 10 or indirectly from a PACS. Furthermore, the image acquisition unit of the endoscope system 10 can be provided in a medical business support device that connects to various examination devices, such as the first examination device, the second examination device, ..., the Nth examination device, including the endoscope system 10, via a network. 50 And / or part or all of the control unit 30 may be provided.

[0098] In this embodiment, the hardware structure of the processing unit that executes various processes, such as the image acquisition unit 50, DSP (Digital Signal Processor) 52, noise reduction unit 53, image processing switching unit 54, inspection image acquisition unit 60, image selection unit 90 and display control unit 200, image input unit 92, image splitting unit 100, first classifier 110, second classifier 120, third classifier 130, image evaluation value calculation unit 140, part evaluation value calculation unit 150 and overall evaluation value calculation unit 160, is the various processors shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units; a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array); and a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to execute various processes.

[0099] A single processing unit may be composed of one of these various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of System-on-a-Chip (SoC) systems. Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0100] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit formed by combining circuit elements such as semiconductor devices. The hardware structure of the memory unit is a storage device such as an HDD (hard disk drive) or SSD (solid state drive). [Explanation of symbols]

[0101] 10 Endoscopy Systems 12 Endoscopes 12a Insertion section 12b Operation section 12c curved section 12d Tip 12e Angle Knob 12f Observation Mode Switch 12g Image Analysis Mode Switch 12h still image acquisition instruction switch 12i Zoom Control Section 14 Light source device 15 Processor Unit 16 Computer 17 displays 17a First display 17b Second display 17c Specific Display 19 User Interface 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 21 Light source processor 22 Optical path coupling section 23 Light Guide 30a illumination optical system 30b Imaging optical system 31 Illumination Lens 41 Objective lens 42 Zoom Lens 43 Imaging Sensor 44 Imaging Processor 45 CDS / AGC circuit 46 A / D converters 50 Image acquisition unit 52 DSP 53 Noise Reduction Section 54 Image processing switching unit 55 Central Control Unit 60. Inspection Image Acquisition Unit 70 First illumination light image generation unit 80 Second illumination light image generation unit 90 Image Selection Section 92 Image Input Section 100 Image division section 101 Tamari-bu 102 divided input images 103 reflection 104a Edge of the cap 104b The entire cap, including the edge of the cap 106 Periphery of the image 107 Bubbles 110 First classifier 112 Domain Evaluation Values 120 Second classifier Input image with 121 region evaluation values 122 Lesion evaluation values 124 Extraction area 125 slots 126 Areas of Interest 130 Third classifier 131 Input image 132 Removed Images 140 Image evaluation value calculation unit 141 Lesion evaluation values ​​for the example in Figure 31 142 Image evaluation value 150-part evaluation value calculation unit 151 Lesion evaluation values ​​for the example in Figure 32 Images 151a, 151b, and 151c in Figure 32 show the output of lesion evaluation values. 152 body part evaluation values 160 Overall Evaluation Value Calculation Unit 161 Lesion evaluation values 161a, 161b, 161c, 161d, 161e, 161f: Input images for the example in Figure 33 162 Overall rating 171 Display Image 172 Inspection image of the frame immediately preceding the display image 172a Lesion evaluation value 172b Areas with high lesion evaluation values 172c: The lesion evaluation value falls within the moderate range. 172d Areas with low lesion evaluation values 172e Normal area with extremely low lesion evaluation value 173 Superimposed images 173a Superimposed image a 173b Superimposed image b 173c Superimposed image c 180 1st screen 182 2nd screen 190 Display screen 191 Display Image 192 Input image with output lesion evaluation values 193 Lesion evaluation values 200 Display Control Unit 210 Close-up image 212 Distant view image 213 Blood vessels

Claims

1. An endoscope system that illuminates a subject and images the light from the subject, It comprises an endoscope and an image control processor, The aforementioned image control processor is: Based on the image signals captured by the endoscope, an examination image is acquired. The aforementioned inspection image is divided into multiple regions as input images, By inputting the input image, which has been divided into the aforementioned multiple regions, to the first classifier, region evaluation values ​​for the aforementioned multiple regions are output. By inputting the input image, on which the region evaluation values ​​are assigned to the multiple regions, into the second classifier, the lesion evaluation value is output. The lesion evaluation value is output for each region where the region evaluation value is output. The aforementioned image control processor is: Based on the magnification at which the inspection image was acquired, the shape, size, and / or number of divisions of the input image are determined. An endoscope system that determines the number of divisions in the input image to be changed depending on the position.

2. The aforementioned image control processor is: The endoscopic system according to claim 1, wherein the input image consisting of extracted regions extracted from the plurality of regions based on the region evaluation value is input to the second classifier, thereby outputting a lesion evaluation value.

3. The endoscope system according to claim 1 or 2, wherein the region evaluation value is a numerical value that represents the presence or absence of an evaluation inhibiting object that is unsuitable for outputting the lesion evaluation value on the input image and reduces the output accuracy of the lesion evaluation value, and the effect that the evaluation inhibiting object has on the output of the lesion evaluation value.

4. The aforementioned image control processor is: The endoscope system according to claim 2, wherein the plurality of regions whose region evaluation value is equal to or greater than a threshold are defined as the extraction regions.

5. The endoscope system according to claim 4, wherein the plurality of regions whose region evaluation value is less than a threshold region includes one of the following: a puddle of water, a pool of blood, a liquid pool, bubbles, distortion, blur, reflection, or cap.

6. Equipped with a light source processor, The aforementioned light source processor is Controlling the emission of first and second illumination lights, which have different emission spectra from each other. When automatically switching between a first illumination period in which the first illumination light is emitted and a second illumination period in which the second illumination light is emitted, the first illumination light is emitted in a first emission pattern, and the second illumination light is emitted in a second emission pattern. The aforementioned image control processor is: The endoscopic system according to any one of claims 1 to 5, wherein the input image is a first illumination light image based on the first illumination light, or a second illumination light image based on the second illumination light.

7. The aforementioned image control processor is: The endoscope system according to claim 6, wherein the second illumination light image is used as the input image.

8. The endoscopic system according to claim 6 or 7, wherein the first illumination light is white light and the second illumination light has a peak wavelength of 410 nm ± 10.

9. Equipped with a display, The aforementioned image control processor is: The endoscopic system according to any one of claims 6 to 8, wherein the lesion evaluation value, output based on the second illumination light image acquired in the frame immediately preceding the acquisition of the first illumination light image, is superimposed on the first illumination light image as a numerical value or color and displayed on the display.

10. The aforementioned image control processor is: When the display comprises a first display and a second display, the first illumination light image is displayed on the first screen of the first display, and the lesion evaluation value output based on the second illumination light image acquired in the frame immediately preceding the acquisition of the first illumination light image is displayed on the second screen of the second display, or The endoscopic system according to claim 9, wherein, when the display has only one specific display, the first illumination light image is displayed on the first screen of the specific display, and the lesion evaluation value is displayed on the second screen of the specific display.

11. The endoscopic system according to any one of claims 6 to 10, wherein the first light emission pattern is one of a first A light emission pattern in which the number of frames in the first illumination period is the same in each of the first illumination periods, and a first B light emission pattern in which the number of frames in the first illumination period is different in each of the first illumination periods.

12. The above second emission pattern is, A second A pattern in which the number of frames in the second illumination period is the same in each of the second illumination periods, and the emission spectrum of the second illumination light is the same in each of the second illumination periods. A second B pattern in which the number of frames in the second illumination period is the same for each of the second illumination periods, and the emission spectrum of the second illumination light is different for each of the second illumination periods. A second C pattern in which the number of frames in the second illumination period is different for each of the second illumination periods, and the emission spectrum of the second illumination light is the same for each of the second illumination periods, The endoscopic system according to any one of claims 6 to 11, wherein the number of frames in the second illumination period is different in each of the second illumination periods, and the emission spectrum of the second illumination light is one of the different second D patterns in each of the second illumination periods.

13. The aforementioned image control processor is: Using the lesion evaluation values ​​for the plurality of regions output based on the input image of any one frame, an image evaluation value for the input image of one frame is calculated. Using the lesion evaluation values ​​for the multiple regions output based on the input images of multiple frames, a site evaluation value for the input images of multiple frames is calculated. The endoscopic system according to any one of claims 6 to 12, which calculates an overall evaluation value using the image evaluation value and / or site evaluation value attached to the input image for all frames on which the lesion evaluation value is output.

14. The endoscopic system according to any one of claims 1 to 13, wherein the lesion evaluation value is output based on an evaluation index for ulcerative colitis.

15. The endoscope system according to any one of claims 1 to 14, wherein the central part of the input image is divided into a smaller number of sections.

16. The system comprises an endoscope that illuminates a subject and captures light from the subject, and an image control processor. The aforementioned image control processor is: The steps include: acquiring an examination image based on the image signal captured by the endoscope; The steps include dividing the aforementioned inspection image into multiple regions as input images, The steps include inputting the input image, which has been divided into the plurality of regions, to the first classifier and outputting region evaluation values ​​for the plurality of regions, The process includes the step of inputting the input image, on which the region evaluation values ​​are assigned to the plurality of regions, into a second classifier and outputting a lesion evaluation value, The lesion evaluation value is output for each region where the region evaluation value is output. Based on the magnification at which the inspection image was acquired, the shape, size, and / or number of divisions of the input image are determined. A method for operating an endoscope system that determines how many divisions to make in the input image, depending on the position.

Images