Endoscope system, medical image processing device, and method for operating same

Abstract

Provided are an endoscope system, a medical image processing device, and a method for operating the same with which it is possible to obtain a diagnostic result from an endoscope image in accordance with the objective of the user.　This medical image processing device acquires an endoscope image captured by irradiating a subject with one or a plurality of types of illuminating light, and uses the endoscope image to output at least one of a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the degree of disorder, and a first evaluation result based on a second evaluation criterion, which is different from the first evaluation criterion. The first evaluation criterion is an evaluation criterion for the degree of disorder based on an image captured by an imaging device different from the endoscope. The image captured by the imaging device different from the endoscope is a pathological image. The second evaluation criterion is an evaluation criterion for the degree of disorder based on the endoscope image.

Description

Endoscope system, medical image processing device, and operation method thereof

[0001] The present invention relates to an endoscope system that outputs multiple diagnostic results from medical images, a medical image processing apparatus, and an operating method thereof.

[0002] Advances in image analysis and artificial intelligence have led to the active development of advanced medical devices for computer-aided diagnosis (CAD). CAD assists physicians in endoscopic examinations, shortening diagnostic time and improving diagnostic accuracy and reproducibility. For example, endoscopic images can be used to determine whether a tumor is benign or malignant, and the degree of inflammation can be scored according to disease evaluation criteria.

[0003] Specifically, a technique is known for calculating the Mayo endoscopic subscore (MES), which is one of the evaluation criteria for ulcerative colitis, from two types of image signals (a first image signal and a second image signal) obtained by illuminating a subject with white light or special light, and displaying a high-score display image based on the second image signal if the MES calculated from the second image signal is greater than the MES calculated from the first image signal, thereby improving diagnostic accuracy (Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2020-65685

[0005] In the case of ulcerative colitis, there are several types of evaluation criteria, such as the Mayo score, which is a standard for evaluating the degree of bleeding, ulcers, erosion, etc. using endoscopic images, and the Geboes histopathology score (GHS), which is a standard for evaluating the degree of infiltration of plasma cells and neutrophils using biopsy specimens of the colonic mucosa. Depending on the disease, users may want to select an evaluation criterion that suits their purpose from several types of evaluation criteria, make a judgment, and obtain evaluation results.

[0006] An object of the present invention is to provide an endoscope system, a medical image processing apparatus, and an operating method thereof that can obtain diagnostic results from an endoscopic image according to the user's purpose.

[0007] The medical image processing device of the present invention is a medical image processing device equipped with a processor, which is configured to acquire an endoscopic image captured by illuminating a subject with one or more types of illumination light, and to use the endoscopic image to output at least one of a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the severity of a disease, and a second evaluation result based on a second evaluation criterion that is different from the first evaluation criterion.

[0008] The first evaluation criterion is preferably an evaluation criterion of the extent of the disease based on an image captured by an imaging device other than an endoscope.

[0009] The images captured by an imaging device other than an endoscope are preferably pathological images of biopsy tissue taken from the subject.

[0010] The processor is configured to use the endoscopic image to output a first evaluation result by a first classifier, and it is preferable that the first classifier outputs the first evaluation result using a pre-stored surface morphology classification, which is a correspondence between the endoscopic image and the degree of disease based on the pathological image.

[0011] The processor is preferably configured to generate a first frequency domain image by extracting a first frequency domain from the endoscopic image, a second frequency domain image by extracting a second frequency domain different from the first frequency domain from the endoscopic image, and a third frequency domain image by extracting a third frequency domain different from the first frequency domain and the second frequency domain, and to output a first evaluation result based on the first evaluation criterion by a first determiner using the first frequency domain image, the second frequency domain image, and the third frequency domain image.

[0012] The processor is configured to use the endoscopic image to output a first evaluation result by a second classifier that is different from the first classifier, and it is preferable that the second classifier has been trained using the training endoscopic image and the degree of disease based on the pathological image.

[0013] The processor is configured to output the second evaluation result by a third classifier different from the first and second classifiers using the endoscopic images, and the third classifier is preferably trained using the training endoscopic images and the disease severity based on the endoscopic images.

[0014] It is preferable that the first evaluation criterion is an evaluation criterion for the severity of the disease based on images captured by an imaging device other than an endoscope that evaluates the severity of the disease in two stages, and that the second evaluation criterion is an evaluation criterion for the severity of the disease based on endoscopic images that evaluate the severity of the disease in two stages.

[0015] It is preferable that the first evaluation criterion is an evaluation criterion for the severity of the disease based on images taken with an imaging device other than an endoscope that evaluates the severity of the disease on a multi-level scale of three or more, and that the second evaluation criterion is an evaluation criterion for the severity of the disease based on endoscopic images that evaluate the severity of the disease on a multi-level scale of three or more.

[0016] Preferably, the disease is ulcerative colitis. Preferably, the first evaluation criterion is Geboes histopathology score, Riley score, Modified Riley score, or Roberts histopathology index, and the second evaluation criterion is Mayo endoscopic subscore or UCEIS score.

[0017] Preferably, the first evaluation criterion includes the morphology of superficial blood vessels and / or the degree of bleeding, and the second evaluation criterion includes the degree of redness of the mucosa, the degree of visibility of deep blood vessels and / or the degree of erosion or ulceration.

[0018] The first evaluation criterion and the second evaluation criterion have a range in which the disease is in remission and a range in which it is not in remission, and the range in which the disease is in remission in the first evaluation criterion is smaller than the range in which the disease is in remission in the second evaluation criterion, and the first non-remission boundary value, which is the lowest value in the first evaluation criterion in which the disease is not in remission, is different from the second non-remission boundary value, which is the lowest value in the second evaluation criterion in which the disease is not in remission, and it is preferable that the first non-remission boundary value be set within the range in which the disease is in remission in the second evaluation criterion.

[0019] It is preferable that the processor is configured to output at least one of a first evaluation result and a second evaluation result for each frame of endoscopic image, and to output and display at least one of a first multi-frame evaluation result calculated using one evaluation result relating to at least two or more frames of endoscopic image, and a second multi-frame evaluation result calculated using a second evaluation result relating to at least two or more frames of endoscopic image.

[0020] The processor is preferably configured to divide the endoscopic image into a plurality of regions and output at least one of a first evaluation result and a second evaluation result for each of the plurality of regions.

[0021] It is preferable that the processor is configured to output area evaluation indices for the multiple areas based on an endoscopic image divided into multiple areas, and to output at least one of a first evaluation result and a second evaluation result for the multiple areas using an endoscopic image consisting of the multiple areas to which area evaluation indices are assigned.

[0022] It is preferable that the processor is configured to output at least one of a first evaluation result and a second evaluation result for each region of the endoscopic image, and to output at least one of a first multi-region evaluation result calculated using the first evaluation results for at least two or more regions, and a second multi-region evaluation result calculated using the second evaluation results for at least two or more regions.

[0023] The processor is preferably configured to output the first multi-region frame evaluation result and the second multi-region frame evaluation result for each frame of the endoscopic image.

[0024] The processor is preferably configured to acquire a first illumination light image captured by illuminating a subject with the first illumination light as an endoscopic image, generate a structure-enhanced image using the first illumination light image, and output a first evaluation result based on a first evaluation criterion using the structure-enhanced image.

[0025] The structure-emphasized image is preferably an image in which the surface structure, color difference, or edge amount of the first illumination light image is emphasized.

[0026] The processor is preferably configured to use the first illumination light image to output a second evaluation result based on a second evaluation criterion.

[0027] The first illumination light is preferably white light. The processor is preferably configured to acquire a plurality of endoscopic images including a first illumination light image captured by illuminating the subject with the first illumination light and a second illumination light image captured by illuminating the subject with second illumination light having a spectrum different from that of the first illumination light, and to output a first evaluation result based on a first evaluation criterion using the second illumination light image.

[0028] Preferably, the processor is configured to automatically switch between a first illumination period in which the object is illuminated with the first illumination light and a second illumination period in which the object is illuminated with at least one type of second illumination light.

[0029] The second illumination light preferably has a narrower wavelength band than the first illumination light, and preferably contains a larger amount of a specific blue or specific purple color than the first illumination light.

[0030] Preferably, the processor is configured to divide the second illumination light image into a plurality of second illumination regions and output the first evaluation results for the plurality of second illumination regions.

[0031] The processor is preferably configured to output the first evaluation result and the second evaluation result, and to issue a notification when a combination of the first evaluation result and the second evaluation result satisfies a specific condition.

[0032] The specific condition for reporting is preferably when the first evaluation result is non-remission, or when the second evaluation result is remission and the first evaluation result is non-remission.

[0033] The processor preferably notifies the user by displaying a warning on a display on which the endoscopic image is displayed.

[0034] The processor preferably causes the endoscopic image, the first evaluation result, and the second evaluation result to be displayed on a display.

[0035] The processor is preferably configured to store the part information of the subject for which the first evaluation result and the second evaluation result are output, the first evaluation result, and the second evaluation result in association with each other.

[0036] An endoscope system of the present invention includes the above-described medical image processing device, a light source device, and an endoscope.

[0037] The operating method of the medical image processing device of the present invention includes the steps of acquiring a medical image photographed by illuminating a subject with one or more types of illumination light, and using the medical image to output at least one of a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the severity of the disease, and a second evaluation result based on a second evaluation criterion that is different from the first evaluation criterion.

[0038] According to the present invention, it is possible to provide an endoscope system, a medical image processing apparatus, and an operating method thereof that can obtain diagnostic results from an endoscopic image according to the user's purpose.

[0039] 1 is an explanatory diagram of the configuration of an endoscope system. A block diagram showing the functions of an endoscope system. A graph showing the spectrum of a first illumination light. A graph showing the spectrum of a second illumination light. A block diagram showing the functions of an image determination unit. An explanatory diagram showing blood vessel forms, intramucosal bleeding, and extramucosal bleeding in forms 2 to 6. An explanatory diagram showing the tissue of the digestive tract. A block diagram showing the functions of a frequency domain image generation unit and a first determiner serving as an image processing unit. An explanatory diagram showing the flow in which the first determiner serving as an image processing unit analyzes a frequency domain image from a second illumination light image. An explanatory diagram showing the flow in which the first determiner serving as an image processing unit outputs a first evaluation result. A block diagram showing the functions of a structure-enhanced image generation unit and a first determiner serving as an image processing unit. An explanatory diagram showing a method for generating first learning data. An explanatory diagram showing a case in which a frequency domain image is input to the first determiner serving as an image processing unit and a first evaluation result is output. An explanatory diagram showing a case in which a structure-enhanced image is input to the first determiner serving as an image processing unit and a first evaluation result is output. An explanatory diagram showing a case in which a second illumination light image is input to the first determiner serving as a trained model and a first evaluation result is output. 1 is an explanatory diagram showing a case where a second illumination light image is input to a second determiner and a first evaluation result is output. FIG. 1 is an explanatory diagram showing a case where a first illumination light image is input to a second determiner and a first evaluation result is output. FIG. 2 is an explanatory diagram showing a case where a first illumination light image is input to a third determiner and a second evaluation result is output. FIG. 3 is an explanatory diagram showing a case where a second illumination light image is input to a third determiner and a second evaluation result is output. FIG. 4 is an explanatory diagram showing a case where a frequency domain image is input to a third determiner and a second evaluation result is output. FIG. 5 is an image diagram showing a display image when an endoscopic image and an evaluation result display field are displayed. FIG. 6 is a block diagram showing the function of a comprehensive evaluation result calculation unit. FIG. 7 is an image diagram showing a display image when an endoscopic image, a region currently being observed, and frame evaluation results are displayed. FIG. 8 is an image diagram showing a display image when an endoscopic image, a schematic diagram, and multi-frame region evaluation results are displayed. FIG. 9 is an image diagram showing a display image when an endoscopic image, a three-dimensional structure diagram, and multi-frame region evaluation results are displayed.1 is an image diagram showing a display image when an endoscopic image, a simple schematic diagram, and a multi-frame site evaluation result are displayed. FIG. 1 is an image diagram showing an example of an endoscopic image divided by an image division unit. FIG. 1 is an image diagram showing an example of an evaluation result output from an endoscopic image divided into a plurality of regions. FIG. 1 is an image diagram showing an example of an endoscopic image including a puddle. FIG. 1 is an image diagram showing an example of an endoscopic image including bubbles. FIG. 1 is an image diagram showing an example of an endoscopic image including a reflection. FIG. 1 is an image diagram showing an example of an endoscopic image including the edge of a cap. FIG. 1 is an image diagram showing an evaluation inhibition region when the edge of a cap is included in an endoscopic image. FIG. 1 is an explanatory diagram showing the output of a region evaluation index. FIG. 1 is an explanatory diagram showing an example of outputting a region evaluation index and evaluation result. FIG. 1 is an image diagram showing a display image when a divided endoscopic image and a multi-region frame evaluation result are displayed. FIG. 1 is an image diagram showing a display image when an endoscopic image, a site currently being observed, and a multi-region site evaluation result are displayed. FIG. 1 is an image diagram showing a display image when an endoscopic image, a schematic diagram, and a multi-region site evaluation result are displayed. FIG. 1 is an image diagram showing a display image when an endoscopic image, a three-dimensional structural diagram, and a multi-region site evaluation result are displayed. 1 is an image diagram showing a display image when an endoscopic image, a simple schematic diagram, and a multi-region site evaluation result are displayed. FIG. 1 is an image diagram showing a display image when an endoscopic image, a site currently being observed, a multi-region frame evaluation result, and a multi-region site evaluation result are displayed. FIG. 1 is an image diagram showing a display image when an endoscopic image, a schematic diagram, a multi-region frame evaluation result, and a multi-region site evaluation result are displayed. FIG. 1 is an image diagram showing a display image when an endoscopic image, a three-dimensional structure diagram, a multi-region frame evaluation result, and a multi-region site evaluation result are displayed. FIG. 1 is an image diagram showing a display image when an endoscopic image, a simple schematic diagram, a multi-region frame evaluation result, and a multi-region site evaluation result are displayed. FIG. 2 is an image diagram showing a display image when an endoscopic image, a simple schematic diagram, a multi-region frame evaluation result, and a multi-region site evaluation result are displayed. FIG. 2 is a flow chart showing the functional flow of the endoscopic system in the first embodiment. FIG. 3 is an explanatory diagram showing a first light emission pattern. FIG. 4 is an explanatory diagram showing a second light emission pattern. FIG. 4 is an image diagram showing a display image when a first illumination light image, a second illumination light image, a schematic diagram, and an evaluation result are displayed.1 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a region currently being observed, a multi-region frame evaluation result, and a multi-region region evaluation result. 2 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a schematic diagram, a multi-region frame evaluation result, and a multi-region region evaluation result. 3 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a three-dimensional structure diagram, a multi-region frame evaluation result, and a multi-region region evaluation result. 4 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, a simple schematic diagram, a multi-region frame evaluation result, and a multi-region region evaluation result. 5 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, and a region currently being observed. 6 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, and a schematic diagram. 7 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, and a three-dimensional structure diagram. 8 is an image diagram showing a display image when displaying a first illumination light image, a second illumination light image, and a three-dimensional structure diagram. FIG. 1 is an image diagram showing a display image when a first illumination light image and a site currently being observed are displayed. FIG. 1 is an image diagram showing a display image when a first illumination light image and a schematic diagram are displayed. FIG. 2 is an image diagram showing a display image when a first illumination light image and a three-dimensional structure diagram are displayed. FIG. 3 is an image diagram showing a display image when a first illumination light image and a simple schematic diagram are displayed. FIG. 4 is an image diagram showing a display image when the entire first illumination light image and second illumination light image are surrounded by a warning display frame. FIG. 5 is an image diagram showing a display image when corresponding first illumination regions and second illumination regions are surrounded by a warning display frame.

[0040] 1, an endoscopic system 10 includes an endoscope 12, a light source device 13, a medical image processing device (processor device) 14, a display 15, and a user interface 16. The endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14.

[0041] The endoscope 12 has an insertion section 12a, an operating section 12b, a bending section 12c, and a tip section 12d. The insertion section 12a is inserted into the body of the subject. The operating section 12b is provided at the base end of the insertion section 12a. The bending section 12c and the tip section 12d are provided on the tip side of the insertion section 12a. The bending section 12c is bent by operating the angle knob 12e of the operating section 12b. The tip section 12d is directed in a desired direction by the bending of the bending section 12c. A forceps channel (not shown) is provided from the insertion section 12a to the tip section 12d for inserting a treatment tool or the like. The treatment tool is inserted into the forceps channel through the forceps port 12j.

[0042] The endoscope 12 is provided with an optical system for forming an image of a subject and an optical system for irradiating the subject with illumination light. The operation unit 12b is provided with an angle knob 12e, a mode selector switch 12f, a still image acquisition instruction switch 12h, and a zoom operation unit 12i. The mode selector switch 12f is used to switch observation modes. The still image acquisition instruction switch 12h is used to instruct acquisition of a still image of the subject. The zoom operation unit 12i is used to operate the zoom lens 42.

[0043] The light source device 13 generates illumination light. The display 15 displays an endoscopic image and a display image. The endoscopic image includes at least one of a first illumination light image and a second illumination light image, which are captured using different illumination lights. The display image is an image generated to display the endoscopic image, the disease evaluation results output from the endoscopic image, and information accompanying the evaluation results. The user interface 16 has a keyboard, mouse, microphone, tablet, touch pen, etc., and accepts input operations such as function settings. The processor device 14 controls the light source device 13 and controls image processing, analysis, and display of image signals transmitted from the endoscope 12.

[0044] 2, the light source device 13 includes a light source unit 20, a light source processor 21 that controls the light source unit 20, and an optical path coupling unit 22. The light source unit 20 has multiple semiconductor light sources, each of which is turned on or off. When the multiple semiconductor light sources are turned on, the light emission intensity of each semiconductor light source is controlled to emit illumination light that illuminates the subject. The light source unit 20 has four color LEDs: a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d. The light source unit 20 and / or the light source processor 21 may be built into the endoscope 12. Alternatively, the light source processor 21 may be built into the processor device 14.

[0045] In the first embodiment, the endoscope system 10 has a first illumination light mode and a second illumination light mode. The first illumination light mode and the second illumination light mode are switched between via the central control unit 50 by operating the mode selector switch 12f. The illumination light includes a first illumination light and a second illumination light having a spectrum different from that of the first illumination light. The first illumination light is normal light used for screening observation by providing brightness to the entire subject, and is preferably broadband light such as white light. The second illumination light is at least one type of special light used to highlight specific structures, such as glands and blood vessels in the mucosa of the digestive tract, which is the subject. The second illumination light preferably has a narrower wavelength band than the first illumination light. Furthermore, the second illumination light preferably contains more of a specific blue or specific purple color than the first illumination light. In particular, the second illumination light preferably has a central wavelength of 410 nm.

[0046] When emitting the first illumination light, as shown in Fig. 3, the V-LED 20a emits violet light V with a central wavelength of 405±10 nm and a wavelength range of 380 to 420 nm. The B-LED 20b emits blue light B with a central wavelength of 450±10 nm and a wavelength range of 420 to 500 nm. The G-LED 20c emits green light G with a wavelength range of 480 to 600 nm. The R-LED 20d emits red light R with a central wavelength of 620 to 630 nm and a wavelength range of 600 to 650 nm. When emitting the second illumination light, for example, as shown in Fig. 4, they emit light with a central wavelength of 410 nm.

[0047] The light source processor 21 independently controls the light intensity of the four colors of violet light V, blue light B, green light G, and red light R, and emits the first illumination light or the second illumination light by changing the light intensity.

[0048] In the first illumination light mode, the subject is illuminated with the first illumination light for each frame, and an image is captured, thereby displaying a first illumination light image with natural coloring on the display 15. In the second illumination light mode, the subject is illuminated with the second illumination light for each frame, and an image is captured, thereby displaying a second illumination light image with a specific structure emphasized on the display 15. Note that a frame refers to a unit of time that includes at least the period from the time of light emission to the time when the image sensor 43 completes reading of the image signal.

[0049] In the first illumination light mode, the second illumination light mode, and the multi-light emission mode of the second embodiment, if the user wishes to capture an endoscopic image as a still image, by operating the still image capture instruction switch 12h, a signal regarding a still image capture instruction is sent to the endoscope 12, the light source device 13, and the processor device 14.

[0050] Light emitted from each of the LEDs 20a to 20d (see FIG. 2) is incident on a light guide 23 via an optical path coupling unit 22 that is composed of a mirror, a lens, etc. The light guide 23 propagates the light from the optical path coupling unit 22 to the tip 12d of the endoscope 12.

[0051] The distal end 12d of the endoscope 12 is provided with an illumination optical system 30a and an imaging optical system 30b. The illumination optical system 30a has an illumination lens 31, and illumination light propagated by the light guide 23 is irradiated onto the subject via the illumination lens 31. On the other hand, when the light source unit 20 is built into the distal end 12d of the endoscope 12, light is emitted toward the subject via the illumination lens of the illumination optical system without passing through a light guide. The imaging optical system 30b has an objective lens 41 and an imaging sensor 43. Light from the subject irradiated with illumination light is incident on the imaging sensor 43 via the objective lens 41 and a zoom lens 42. As a result, an image of the subject is formed on the imaging sensor 43. The zoom lens 42 is a lens for enlarging the subject, and is moved between the telephoto end and the wide-angle end by operating the zoom operation unit 12i.

[0052] The image sensor 43 is a primary color sensor and has three types of pixels: B pixels (blue pixels) with blue color filters, G pixels (green pixels) with green color filters, and R pixels (red pixels) with red color filters.

[0053] The imaging sensor 43 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging processor 44 controls the imaging sensor 43. Specifically, the imaging processor 44 reads out the signal from the imaging sensor 43, thereby outputting an image signal from the imaging sensor 43. The output image signal is transmitted to the medical image acquisition unit 60 of the processor device 14.

[0054] The medical image acquisition unit 60 performs various signal processing on the received color image, such as defect correction processing, offset processing, demosaic processing, matrix processing, white balance adjustment, gamma conversion processing, and YC conversion processing, etc. Next, the medical image acquisition unit 60 performs image processing including color conversion processing such as 3×3 matrix processing, tone conversion processing, and 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement, thereby acquiring a first illumination light image for the first illumination light and a second illumination light image for the second illumination light.

[0055] The processor device 14 includes a central control unit 50, a medical image acquisition unit 60, an image assessment unit 70, a comprehensive evaluation result calculation unit 71, a display control unit 130, an image division unit 160, an area evaluation unit 170, and a notification control unit 200 (see Figure 2).

[0056] In the processor device 14, the central control unit 50, which is composed of a light source processor and an image processing processor, operates the programs in the program memory, thereby realizing the functions of the medical image acquisition unit 60, image assessment unit 70, overall evaluation result calculation unit 71, display control unit 130, image division unit 160, and area evaluation unit 170.

[0057] The first illumination light image or the second illumination light image acquired by the medical image acquisition unit 60 is input to the image assessment unit 70. The image assessment unit 70 outputs at least one of a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the severity of a disease, and a second evaluation result based on a second evaluation criterion different from the first evaluation criterion. The evaluation criterion (hereinafter, a term collectively describing the first evaluation criterion and the second evaluation criterion) is a scale for assessing the severity of a disease, as will be described in detail later. The evaluation result (hereinafter, a term collectively describing the first evaluation result and the second evaluation result) is a result output by the image assessment unit 70, as will be described in detail later. For example, the evaluation result may represent a condition such as "remission," "non-remission," "intramucosal bleeding," or "severe vascular irregularity," or may use an existing scale such as "Mayo 0" or "Geboes 2B," or may use a pre-stored scale such as "morphology 1" or "morphology 2," or may be a value such as "0" or "1." The assessment refers to the image assessment unit 70 analyzing the endoscopic image to output the evaluation result. The output means that the image evaluation unit 70 outputs the first evaluation result and / or the second evaluation result.

[0058] As shown in FIG. 5 , the image determination unit 70 has a first determiner 80, a second determiner 90, and a third determiner 100 that perform determination using an endoscopic image. The first determiner 80 is an image processing unit that stores a surface morphology classification, which will be described later, or a trained model. The second determiner 90 and the third determiner 100 are different trained models. A trained model is a model that has been trained using training data. The image determination unit 70 further has a frequency domain image generation unit 110 and a structure-enhanced image generation unit 120, which will be described later.

[0059] Deep learning is preferably used for machine learning to generate a learning model, for example, a multilayer convolutional neural network. In addition to deep learning, machine learning also includes decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, learning using neural networks, generative adversarial networks, etc.

[0060] The first determiner 80, which serves as a trained model, is trained using the first training data. The second determiner 90 is a trained model in which the training model has trained using the second training data. The first evaluation criterion is used when generating a data set of the first training data and a data set of the second training data. Details of the first training data and the second training data will be described later. The third determiner 100 is a trained model in which the training model has trained using the third training data. The second evaluation criterion is used when generating the third training data. Details of the third training data will be described later.

[0061] The image evaluation unit 70, which has the first classifier 80, the second classifier 90, and the third classifier 100, outputs at least one of the first evaluation result and the second evaluation result based on different first and second evaluation criteria, and thus the results can be output according to the user's purpose, such as a treatment goal.

[0062] Ulcerative colitis has different treatment goals depending on the severity: clinical remission, endoscopic remission, and pathological remission. Clinical remission is a state in which patient symptoms, such as diarrhea and abdominal pain, are mild or absent. Traditionally, clinical remission through prescription of therapeutic drugs has been the treatment goal for ulcerative colitis. However, even after clinical remission, cases of relapse, in which symptoms reappear, have been observed. Therefore, endoscopic examination of patients with relapse has sometimes revealed endoscopic non-remission (e.g., mucosal redness and lack of healing) in the gastrointestinal tract. That is, as shown in Table 1, it has been revealed that relapse occurs when endoscopic non-remission occurs even in clinical remission. Since long-term remission was not achieved in such cases, to avoid short-term remission, appropriate tests are performed to distinguish between "endoscopic non-remission despite clinical remission" and "endoscopic remission," and treatment is carried out with endoscopic remission (mucosal healing) as the treatment goal.

[0063]

[0064] In recent years, biologics used in the treatment of ulcerative colitis have demonstrated high therapeutic efficacy, resulting in a high probability of achieving endoscopic remission. In addition, it has been reported that when pathological remission is confirmed by biopsy tissue, remission lasts longer. In other words, the next treatment goal is to distinguish between "endoscopic remission but no pathological remission" and "pathological remission," as shown in Table 1, and to aim for pathological remission with appropriate therapeutic agents. As treatment goals change over the course of the disease, from clinical remission to endoscopic remission and from endoscopic remission to pathological remission, it is important to be able to examine the current condition at each stage. The above configuration satisfies the needs of physicians and other users who want to obtain results tailored to different treatment goals, such as endoscopic remission and pathological remission, depending on the patient's progress.

[0065] The first evaluation criterion, which is an evaluation criterion for the extent of the disease, is preferably an evaluation criterion for the extent of the disease based on images captured by an imaging device other than the endoscope. With this configuration, an evaluation result different from an evaluation result obtained from only the endoscopic image can be obtained.

[0066] The image captured by the imaging device other than the endoscope is preferably a pathological image of biopsy tissue taken from the subject. A pathological image is an image captured after the biopsy tissue has been prepared for diagnosis by observation under a microscope, such as a pathological tissue specimen stained with hematoxylin and eosin.

[0067] When pathological remission is the treatment goal, steroids, immunosuppressants, and biological agents are used as therapeutic agents. These therapeutic agents are highly effective during the remission induction phase, suppressing the symptoms of ulcerative colitis. However, these therapeutic agents have significant systemic effects on the body, and increasing their dosage increases the risk of side effects. To prevent overadministration of therapeutic agents with significant side effects, it is necessary to switch to a therapeutic agent with relatively few side effects (5-aminosalicylic acid preparations) once sufficient therapeutic effect has been achieved. Since confirming pathological remission serves as a measure of whether the therapeutic agent has been sufficiently effective, the above-described configuration, which distinguishes between pathological non-remission and pathological remission based on evaluation criteria based on pathological images and outputs the evaluation results, allows users to determine when to switch medications. Furthermore, the ability to output evaluation results based on pathological image-based evaluation criteria based solely on endoscopic images eliminates the need for biopsies, significantly reducing the burden on the surgeon and the subject.

[0068] The first evaluation criterion may be a criterion for evaluating the severity of the disease in two stages, or may be a multi-stage evaluation of three or more stages in which remission or non-remission is further divided into subclasses. The disease is preferably ulcerative colitis. When the first evaluation criterion is a two-stage criterion, the severity of the disease is in two stages, remission or non-remission, and the first evaluation result is output as "remission" or "non-remission." In this case, output of the first evaluation result is a relatively simple process, so that the evaluation can be made quickly without placing a burden on the processor. On the other hand, when the first evaluation criterion is a multi-stage criterion of three or more stages, a more detailed evaluation can be made than when the evaluation is in two stages, remission or non-remission.

[0069] It is preferable to use the Geboes histopathology score (GHS) (Table 2) as the first evaluation criterion. Hereinafter, when outputting the GHS as the first evaluation result, for simplicity, a GHS score of "0" will be referred to as "Geboes 0." As shown in Table 1, Geboes 0 to 2A indicate remission (pathological remission), and Geboes 2B to 5 indicate non-remission (pathological non-remission). If the image assessment unit 70 assesses the endoscopic image as "Geboes 2B," it may output "Geboes 2B" or "non-remission" as the first evaluation result. Alternatively, an evaluation criterion for ulcerative colitis based on a pathological image, such as the Riley score, the Modified Riley score, or the Roberts histopathology index, may be used as the first evaluation criterion. With the above configuration, evaluation results can be output based on evaluation criteria used in clinical practice.

[0070]

[0071] The first evaluation criterion may be the morphology of the superficial blood vessels and / or the degree of bleeding. In this case, the first evaluation result may be output, for example, as "superficial blood vessels: honeycomb pattern," "superficial blood vessels: mild irregularity," or "mild bleeding." With the above configuration, the first evaluation result can be output as a finding based on the first evaluation criterion from the endoscopic image.

[0072] The surface morphology classification is a correspondence relationship discovered by the inventors between the pattern of vascular structure seen in endoscopic images obtained by irradiating the subject with illumination light having a central wavelength of 410 nm and the degree of disease (GHS) based on pathological images, as shown in Table 3. In ulcerative colitis, the pattern of vascular structure changes, as shown in Figures 6(A) to 6(E), and there is a correspondence relationship between this pattern of vascular structure and the pathological state.

[0073]

[0074] When ulcerative colitis is in a very mild state of remission and the mucosa is normal and the surface blood vessels are not visible, this is classified as "Type 1" in Table 1. When ulcerative colitis is in remission and there is some abnormality in the mucosa, the pattern of the surface blood vessels is regular (Figure 6(A); Table 3, "Type 2") or there is only a slight disruption in the regularity of the pattern of the surface blood vessels (Figure 6(B); Table 3, "Type 3"). On the other hand, when ulcerative colitis is not in remission and the severity is mild, the surface blood vessels become dense, with some areas of densely packed surface blood vessels present (Figure 6(C), Table 3, "Type 4"). When ulcerative colitis is not in remission and the severity is moderate, bleeding 81 occurs within the mucosa (Figure 6(D), Table 3, "Type 5"). When ulcerative colitis is not in remission and is moderate to severe in severity, extramucosal bleeding 82 occurs (FIG. 6(E), Table 3, "Form 6").

[0075] Here, a "highly dense superficial vascular region" refers to a region where superficial blood vessels gather in a meandering pattern, and in the image, it appears as if several superficial blood vessels surround an intestinal crypt (see Figure 7). "Intramucosal bleeding" refers to bleeding within the mucosa (see Figure 7) and is different from bleeding within the lumen (lumen) of the digestive tract. "Extramucosal bleeding" refers to a small amount of blood in the lumen, blood visible in the lumen or the mucosal epithelium forward of the endoscope tip 12d after the lumen has been cleaned, or blood in the lumen accompanied by oozing on the mucosa.

[0076] The following describes a flow in which the first determiner 80 serving as an image processing unit outputs the first evaluation result. In this case, the first frequency domain image or the second frequency domain image generated by the frequency domain image generation unit 110 (see also FIG. 2 ) shown in FIG. 8 is transmitted to the first determiner 80. As shown in FIG. 8 , the frequency domain image generation unit 110 includes a mask processing unit 112, a frequency component spatial distribution calculation unit 113, and a frequency domain extraction unit 114, and the first determiner 80 includes a blood vessel density calculation unit 115, a luminance calculation unit 116, and a first evaluation result output unit 117.

[0077] The first evaluation result is output according to the flow shown in Fig. 9A . In Fig. 9A , processing progresses from top to bottom. The mask processing unit 112 performs mask processing on the peripheral portion of the second illumination light image 111a, which is an endoscopic image, where distortion is large, to generate a masked second illumination light image 112a by extracting only the central portion. Next, the frequency component spatial distribution calculation unit 113 calculates the frequency component spatial distribution and performs frequency processing to extract the frequency region. The frequency component spatial distribution calculation unit 113 calculates the frequency component spatial distribution by applying a Laplacian filter to the masked second illumination light image 112a.

[0078] The frequency domain extraction unit 114 extracts a first frequency domain 114b (the shaded portion of the first frequency domain 114a in FIG. 9A ) from the masked second illumination light image 112a to generate a first frequency domain image 114a. Specifically, the frequency domain extraction unit 114 extracts the first frequency domain (high frequency domain) 114b having a first frequency (high frequency) by Hessian analysis of the spatial distribution of frequency components.

[0079] Next, the blood vessel density calculation unit 115 performs an averaging process on the first frequency domain image 114a to generate a density distribution image 115a by calculating the density distribution of the superficial blood vessels. In Fig. 9A, a region 115b with a high density distribution of the superficial blood vessels, a region 115c with a medium density distribution of the superficial blood vessels, and a region 115d with a low density distribution of the superficial blood vessels are shown.

[0080] Finally, the first evaluation result output unit 117 performs threshold processing using a density distribution threshold, and divides the density distribution of the superficial blood vessels into an area 117a where the density distribution of the superficial blood vessels is equal to or greater than the density distribution threshold, and an area 117b where the density distribution of the superficial blood vessels is less than the density distribution threshold. The area 117a where the density distribution of the superficial blood vessels is equal to or greater than the density distribution threshold corresponds to a "high-density superficial blood vessel area."

[0081] The frequency domain extraction unit 114 further extracts a second frequency domain 114d (low-frequency domain; in FIG. 9A , the shaded area of ​​the second frequency domain image 114c) having a second frequency (low frequency) lower than the first frequency from the masked second illumination light image 112a, thereby generating a second frequency domain image 114c. Specifically, based on the spatial distribution of frequency components, if a frequency statistic (e.g., maximum value, minimum value, or standard deviation) of nine pixels in the masked second illumination light image 112a, including a specific pixel, is equal to or less than a certain value, the specific pixel is determined to belong to the second frequency. This detection of the specific pixel is performed for all pixels, thereby extracting the second frequency domain 114d and generating the second frequency domain image 114c.

[0082] Next, the luminance calculation unit 116 calculates the luminance of the second frequency domain image 114c to generate a second frequency luminance distribution image 116a. In Fig. 9A, a region 116b with low luminance values ​​and a region 116c with high luminance values ​​are shown.

[0083] Finally, the first evaluation result output unit 117 performs threshold processing using the first brightness threshold to separate the region 117c where the brightness is equal to or less than the first brightness threshold and the region 117d where the brightness is greater than the first brightness threshold. The region 117c where the brightness is equal to or less than the first brightness threshold corresponds to "extramucosal bleeding."

[0084] The frequency domain extraction unit 114 also extracts a third frequency domain 114f (mid-frequency domain; the shaded area of ​​the third frequency domain image 114e in FIG. 9A ) having a frequency lower than the first frequency and higher than the second frequency (mid-frequency), thereby generating a third frequency domain image 114e. Specifically, the masked second illumination light image 112a is extracted as a third frequency domain 114f, excluding the first frequency domain 114b and the second frequency domain 114d, thereby generating the third frequency domain image 114e. Next, the luminance calculation unit 116 calculates the luminance of the third frequency domain image 114e to generate a third frequency luminance distribution image 116d, which is divided into a high-luminance region 116e and a low-luminance region 116f.

[0085] Next, the first evaluation result output unit 117 performs threshold processing using the second luminance threshold to separate the region 117e where the luminance is equal to or less than the second luminance threshold and the region 117f where the luminance is greater than the first luminance threshold. The region 117e where the luminance is equal to or less than the second luminance threshold corresponds to "intramucosal bleeding."

[0086] 9B , the number of pixels of “bleeding” is calculated by adding the number of pixels in the region 117c where the brightness is equal to or less than the first brightness threshold, i.e., “extramucosal bleeding,” and the number of pixels in the region 117e where the brightness is equal to or less than the second brightness threshold, i.e., “intramucosal bleeding.” If the number of pixels of “bleeding” is equal to or greater than the bleeding threshold, the second illumination light image 111a corresponds to “Type 5” or “Type 6,” and the first evaluation result is output as “pathological non-remission.” Furthermore, if the number of pixels of “bleeding” is less than the bleeding threshold, the number of pixels in the region 117a where the density distribution of superficial blood vessels is equal to or greater than the density distribution threshold, i.e., the “high-density superficial blood vessel region,” is calculated. If the number of pixels in the “high-density superficial blood vessel region” is equal to or greater than the superficial blood vessel density threshold, the second illumination light image 111a corresponds to “Type 4,” and the first evaluation result is output as “pathological non-remission.” Furthermore, if the number of pixels in "bleeding" is less than the bleeding threshold and the number of pixels in "high-density superficial vascular area" is less than the superficial vascular density threshold, the second illumination light image 111a corresponds to either "form 1," "form 2," or "form 3," and pathological remission is output as the first evaluation result.

[0087] The first determiner 80 serving as the image processing unit preferably outputs the first evaluation result using the first illumination light image. In this case, as shown in FIG. 10 , the structure-enhanced image generated by the structure-enhanced image generator 120 from the first illumination light image is transmitted to the first determiner 80.

[0088] The structure-enhanced image generation unit 120 generates a structure-enhanced image in which superficial blood vessels and superficial bleeding, such as bleeding within the mucosa and bleeding outside the mucosa, are emphasized in the first illumination light image, which is an endoscopic image for training, by utilizing differences in frequency characteristics associated with the magnitude of the pixel value difference (amplitude value) of the bilateral filter. Specifically, as in the texture image generation method disclosed in Japanese Patent Application Laid-Open No. 2012-135345, a first filter coefficient is calculated based on a filter coefficient generation parameter obtained from the absolute value of the difference between a pixel of interest extracted from the first illumination light image and a surrounding pixel region, and a second filter coefficient is calculated using a noise amount estimation model obtained by calculating the average value and standard deviation of pixel values ​​in a predetermined region within a gray chart of the first illumination light image as a filter coefficient generation parameter, and a structure-enhanced image is generated that reflects the difference between the values ​​obtained using the first filter coefficient and the second filter coefficient for pixel values ​​of superficial blood vessels and superficial bleeding, such as bleeding within the mucosa and bleeding outside the mucosa. In the structure-weighted image, surface blood vessels and surface hemorrhages are emphasized, so the first evaluation result can be output based on the structure-weighted image and the surface morphology classification using the same principle as in FIG.

[0089] The "remission" output by the first determiner 80 as the image processing unit as the first evaluation result includes "form 1," "form 2," and "form 3," while the "non-remission" includes "form 4," "form 5," and "form 6" (see Table 3).

[0090] The first determiner 80 as an image processing unit outputs the first evaluation result using the surface morphology classification, so that the pathological evaluation result can be obtained from the first illumination light image and the second illumination light image using the evaluation criteria based on the pathological image, without taking a biopsy tissue. Furthermore, the surface morphology classification pre-stored in the first determiner 80 clearly shows the relationship between the endoscopic image and the pathological image, so that the evaluation result is highly interpretable.

[0091] The first determiner 80 serving as a trained model preferably outputs a first evaluation result when the first illumination light image or the second illumination light image is input. In this case, the first determiner 80 has been trained using first training data.

[0092] A method for generating the first training data will now be described. As shown in Fig. 11 , the association unit 118 serving as the first training data generator preferably generates first training data 118c as a data set of image data in which a surface morphology classification is assigned to a first illumination light image 118a or a second illumination light image 118b, which are training endoscopic images. The first illumination light image 118a or the second illumination light image 118b serving as training endoscopic images may be endoscopic images acquired in advance by an endoscope other than the endoscopic system 10, or may be endoscopic images acquired by an endoscope other than the endoscopic system 10.

[0093] The association unit 118 can associate the surface morphology classification with the second illumination light image, which was used to generate the first frequency domain image, the second frequency domain image, and the third frequency domain image, or with the first illumination light image, which was used to generate the structure-enhanced image, to create first learning data. The association unit 118 may be included in the processor device 14 or in a device other than the endoscope system 10. As described above, the surface morphology classification can be used to assign the degree of disease based on the pathology image to the learning endoscopic image to generate first learning data, thereby configuring the first determiner 80.

[0094] The second determiner 90, which is a trained model, preferably outputs a first evaluation result when the first illumination light image or the second illumination light image is input. The second learning data used for training of the second determiner 90 will be described. The second learning data is preferably a data set of image data in which a learning endoscopic image is accompanied by a first evaluation result obtained from a pathology image. A second learning data generator that generates the second learning data may be provided in the processor device 14 or in a device other than the endoscope system 10. The second determiner 90 preferably learns using the second learning data, then determines the endoscopic image and outputs the first evaluation result.

[0095] The first evaluation result assigned to the training endoscopic image used in the second training data is an evaluation result diagnosed by a physician based on a pathology image obtained from a biopsy tissue collected at the same position as the position at which the training endoscopic image was captured. The diagnosis of the pathology image may be performed by a pathology image diagnosing device. In this case, the training endoscopic image may be either a first illumination light image or a second illumination light image.

[0096] With the above configuration, pathological evaluation can be obtained from endoscopic images without taking a biopsy, just as in the case where the first classifier 80 is used. Furthermore, because the second learning data set is generated based on established pathology, the reliability of the data set is high. Furthermore, because the first evaluation result is output using artificial intelligence, the first evaluation result is output based on multidimensional features including not only human-understandable findings such as vascular morphology, pixel values, and brightness contained in the endoscopic image, but also other findings, so high accuracy of the evaluation result can be expected.

[0097] The image determination unit 70 outputs the first evaluation result based on the first evaluation criterion in the following ways (1) to (5): (1) As shown in FIG. 12 , the frequency-domain image generation unit 110 generates a first frequency-domain image, a second frequency-domain image, and a third frequency-domain image (referred to as "frequency-domain image 110a" in FIG. 12 ) from the second illumination light image 60a, inputs the first frequency-domain image, the second frequency-domain image, and the third frequency-domain image to a first determiner 80 serving as an image processing unit, and outputs a first evaluation result 80a. (2) As shown in FIG. 13 , the structure-emphasized image generation unit 120 generates a structure-emphasized image 120a from the first illumination light image 60b, inputs the structure-emphasized image 120a to a first determiner 80 serving as an image processing unit, and outputs a first evaluation result 80a. (3) As shown in Fig. 14, the second illumination light image 60a is input to a first determiner 80 as a trained model, and the first determiner 80 outputs a first evaluation result 80a. (4) As shown in Fig. 15, the second illumination light image 60a is input to a second determiner 90, and the second determiner 90, which is a trained model, outputs a first evaluation result 90a. (5) As shown in Fig. 16, the first illumination light image 60b is input to the second determiner 90, and the second determiner 90, which is a trained model, outputs a first evaluation result 90a. In this manner, the first illumination light image or the second illumination light image captured by the endoscope 12 can be used to output a first evaluation result based on the first evaluation criterion.

[0098] The second evaluation criterion is preferably an evaluation criterion for the severity of the disease based on endoscopic images. The second evaluation criterion may be a criterion for evaluating the severity of the disease in two stages, or may be a multi-stage evaluation of three or more stages in which remission or non-remission is further subdivided into subclasses. The disease is preferably ulcerative colitis. When the second evaluation criterion is two-stage, the severity of the disease is two-stage, remission or non-remission, and the second evaluation result is output as "remission" or "non-remission." In this case, output of the second evaluation result is a relatively simple process, so that the evaluation can be made quickly without placing a burden on the processor. When the second evaluation criterion is three or more stages, a more detailed evaluation can be made than when the second evaluation criterion is two-stage, remission or non-remission.

[0099] The Mayo endoscopic subscore (MES) (Table 4) is preferably used as the second evaluation criterion. Hereinafter, for simplicity, when outputting the second evaluation result as an MES, an MES of "0" will be referred to as "Mayo 0." As shown in Table 1, Mayo 0-1 indicates remission (endoscopic remission), and Mayo 2-3 indicates non-remission (endoscopic non-remission). When the image assessment unit 70 assesses the endoscopic image as "Mayo 2," it may output "Mayo 2" or "non-remission" as the second evaluation result. Alternatively, an evaluation criterion for ulcerative colitis, such as the Ulcerative Colitis Endoscopic Index of Severity (UCEIS) score, may be used as the second evaluation criterion. The above configuration allows evaluation results to be output based on evaluation criteria used in clinical practice.

[0100]

[0101] The second evaluation criteria may be the degree of mucosal redness, the degree of vascular visibility of deep blood vessels, and / or the degree of erosion or ulceration. In this case, the second evaluation result may be output as, for example, "redness: mild," "vascular visibility: normal," or "bleeding: moderate." With the above configuration, the second evaluation result can be output as a finding based on the second evaluation criteria from the endoscopic image.

[0102] As shown in Table 1, the first and second evaluation criteria have ranges for disease remission and non-remission, and the range for disease remission according to the first evaluation criteria is smaller than the range for disease remission according to the second evaluation criteria. In other words, the criteria for pathological remission are stricter than the criteria for endoscopic remission. The first non-remission boundary value, which is the lowest value for disease non-remission according to the first evaluation criteria, is different from the second non-remission boundary value, which is the lowest value for disease non-remission according to the second evaluation criteria. The first non-remission boundary value is preferably set within the range for disease remission according to the second evaluation criteria. For example, as shown in Table 1, the lowest value for ulcerative colitis non-remission according to the first evaluation criteria is "Geboes 2B." On the other hand, the lowest value for ulcerative colitis non-remission according to the second evaluation criteria is "Mayo 2." When applied to the second evaluation standard, MES, "Geboes 2B" falls within the "Mayo 1" range, which is generally considered endoscopic remission. Because the ranges for determining non-remission of disease differ between the two evaluation standards, when evaluation results are output based on two different evaluation standards, a range that is considered remission by one evaluation standard may be considered non-remission by the other. This discrepancy can prevent "hidden non-remission" from being overlooked in one evaluation standard.

[0103] The third learning data used for learning by the third determiner 100, which receives an endoscopic image as an input and outputs a second evaluation result, will now be described. The third learning data is preferably a data set of image data in which a learning endoscopic image is accompanied by a second evaluation result obtained from the endoscopic image. A third learning data generator that generates the third learning data may be provided in the processor device 14 or in a device other than the endoscope system 10. The third determiner 100 preferably learns using the third learning data, then determines the endoscopic image and outputs the second evaluation result.

[0104] The second evaluation result assigned to the training endoscopic image used in the third learning data is an evaluation result obtained by a doctor diagnosing the training endoscopic image. The diagnosis of the training endoscopic image may be performed by a system other than the third determiner 100 that diagnoses endoscopic images. In this case, the training endoscopic image may be a first illumination light image or a second illumination light image. That is, the third determiner 100 can output the second evaluation result based on the second evaluation criteria from both the first illumination light image and the second illumination light image. Because the second evaluation result is output using artificial intelligence, the second evaluation result is output based on multidimensional features including not only human-understandable findings such as vascular morphology, pixel values, and brightness contained in the endoscopic image, but also other findings, and therefore high accuracy of the evaluation result can be expected.

[0105] When the second illumination light image is input to the third determiner 100 and the second evaluation result is output, the training endoscopic image may be the second illumination light image acquired at a position close to the position where the first illumination light image was captured.

[0106] Furthermore, when the second evaluation result is output using the second illumination light image, the first frequency domain image, the second frequency domain image, and the third frequency domain image may be input to the third determiner 100, which may then output the second evaluation result. In this case, the third learning data is preferably a data set of image data in which the second evaluation result is added to the first frequency domain image, the second frequency domain image, and the third frequency domain image.

[0107] It is preferable that the image assessment unit 70 inputs an endoscopic image and outputs both a first evaluation result and a second evaluation result. By outputting both the first evaluation result and the second evaluation result, the endoscopic image can be evaluated from multiple perspectives.

[0108] As described above, the image determination unit 70 outputs the second evaluation result based on the second evaluation criterion in the following methods (1) to (3). (1) As shown in FIG. 17 , the first illumination light image 60b is input to the third determiner 100, which then outputs the second evaluation result 101. (2) As shown in FIG. 18 , the second illumination light image 60a is input to the third determiner 100, which then outputs the second evaluation result 101. (3) As shown in FIG. 19 , the frequency domain image generation unit 110 generates a first frequency domain image, a second frequency domain image, and a third frequency domain image (denoted as "frequency domain image 110a" in FIG. 12 ) from the second illumination light image 60a, inputs the first frequency domain image, the second frequency domain image, and the third frequency domain image to the third determiner 100, which then outputs the second evaluation result 101.

[0109] The first evaluation result and / or the second evaluation result output by the image assessment unit 70 is transmitted to the display control unit 130. As shown in Fig. 20 , the display control unit 130 preferably generates a display image 131 that displays an endoscopic image 132 currently being observed and an evaluation result display field 133 showing the first evaluation result and / or the second evaluation result, and displays the image on the display 15.

[0110] The image assessment unit 70 outputs at least one of a first evaluation result and a second evaluation result for one frame of the endoscopic image. Hereinafter, the first evaluation result output for one frame of the endoscopic image will be referred to as the first frame evaluation result, and the second evaluation result output for one frame of the endoscopic image will be referred to as the second frame evaluation result.

[0111] Furthermore, in order to obtain a comprehensive evaluation result for multiple frames acquired over a certain section or a certain time period, it is preferable to use the first frame evaluation result or the second frame evaluation result as a unit score and output the comprehensive evaluation result as a comprehensive score based on the unit scores. The comprehensive evaluation result is output by the comprehensive evaluation result calculation unit 71. The first determiner 80, the second determiner 90, or the third determiner 100 of the image determination unit 70 outputs the first frame evaluation result or the second frame evaluation result, which is a unit score, and transmits it to the comprehensive evaluation result calculation unit 71 (see FIG. 2 ).

[0112] The overall evaluation result calculation unit 71 has a multiple frame evaluation result calculation unit 72 and a multiple region evaluation result calculation unit 73, which will be described later, shown in Fig. 21. The multiple frame evaluation result calculation unit 72 uses the first frame evaluation result or the second frame evaluation result as a unit score, and outputs the first multiple frame evaluation result or the second multiple frame evaluation result as an overall score based on the unit score.

[0113] The multi-frame evaluation result calculation unit 72 preferably outputs at least one of a first multi-frame evaluation result calculated using the first evaluation results for at least two or more frames of endoscopic images and a second multi-frame evaluation result calculated using the second evaluation results for at least two or more frames of endoscopic images. The first multi-frame evaluation result and / or the second multi-frame evaluation result is transmitted to the display control unit 130.

[0114] Specifically, when the first multiple-frame evaluation result is output in two stages, remission or non-remission, the multiple-frame evaluation result calculation unit 72 outputs a percentage indicating the proportion of pathological remission and a percentage indicating the proportion of pathological non-remission. Alternatively, the multiple-frame evaluation result calculation unit 72 may output a pathological state with a higher proportion of either pathological remission or pathological non-remission, or may output a pathological state with a proportion equal to or greater than a certain value between pathological remission and pathological non-remission. When the first multiple-frame evaluation result is output in multiple stages, such as GHS, the multiple-frame evaluation result calculation unit 72 outputs a percentage indicating the proportion of Geboes0, a percentage indicating the proportion of Geboes1, a percentage indicating the proportion of Geboes2A, a percentage indicating the proportion of Geboes2B, a percentage indicating the proportion of Geboes3, a percentage indicating the proportion of Geboes4, and a percentage indicating the proportion of Geboes5. Also, the GHS with the highest ratio among Geboes0 to Geboes5 may be output, or the GHS with a ratio equal to or greater than a certain value among Geboes0 to Geboes5 may be output.

[0115] When the second multi-frame evaluation results are output in two stages, remission or non-remission, a percentage indicating the rate of endoscopic remission and a percentage indicating the rate of endoscopic non-remission are output. Alternatively, the endoscopic state with a higher rate of endoscopic remission or endoscopic non-remission may be output, or the endoscopic state with a rate of endoscopic remission or endoscopic non-remission above a certain value may be output. When the first multi-frame site evaluation results are output in multiple stages, such as MES, a percentage indicating the rate of Mayo 0, a percentage indicating the rate of Mayo 1, a percentage indicating the rate of Mayo 2, and a percentage indicating the rate of Mayo 3 may be output. Alternatively, the MES with the highest rate among Mayo 0 to Mayo 3 may be output, or the MES with a rate above a certain value among Mayo 0 to Mayo 3 may be output.

[0116] It is preferable that the multi-frame evaluation result is output using the first frame evaluation result or the second frame evaluation result relating to at least two or more frames of endoscopic images for any section within the body or for any time period during which multiple endoscopic images are acquired. With the above configuration, it is possible to obtain a comprehensive evaluation result that spatiotemporally integrates the evaluation results.

[0117] 21 , the multi-frame evaluation result calculation unit 72 has a multi-frame region evaluation result calculation unit 140. The multi-frame region evaluation result calculation unit 140 uses the first frame evaluation result or the second frame evaluation result as a unit score, and outputs the first multi-frame region evaluation result or the second multi-frame region evaluation result as a total score based on the unit score.

[0118] The multi-frame region evaluation result calculation unit 140 preferably outputs at least one of a first multi-frame region evaluation result calculated using a first frame evaluation result related to at least two or more frames of endoscopic images, and a second multi-frame region evaluation result calculated using a second frame evaluation result related to at least two or more frames of endoscopic images. The first multi-frame region evaluation result and / or the second multi-frame region evaluation result are transmitted to the display control unit 130 and displayed on a display image 131 as shown in FIGS. 22 to 25 .

[0119] The first and / or second multi-frame site evaluation results are preferably output for each anatomical site of the gastrointestinal tract (e.g., esophagus, stomach, duodenum, jejunum, ileum, cecum, ascending colon, transverse colon, descending colon, sigmoid colon, rectum, etc.). The first and / or second multi-frame site evaluation results may be output by majority vote of the number of frames showing remission and non-remission in the first-frame evaluation results for each anatomical site, or by majority vote of the number of frames showing remission and non-remission in the second-frame evaluation results. The multi-frame site evaluation results may be output as A% for "remission" and B% for "non-remission" based on the proportion of the number of frames showing remission and non-remission in the first-frame evaluation results obtained for each anatomical site, and C% for "remission" and D% for "non-remission" based on the proportion of the number of frames showing remission and non-remission in the second-frame evaluation results. Furthermore, the multi-frame region evaluation results may be output as percentages calculated in a multi-level evaluation based on the proportion of each GHS in the first frame evaluation results obtained for each anatomical region, such as E% for "Geboes0" and F% for "Geboes1." The multi-frame region evaluation results may also be statistical quantities such as averages, maximum values, and minimum values. It is preferable that the multi-frame region evaluation results be output for any section, such as dividing the descending colon into three sections: proximal, middle, and distal. It is also preferable that the multi-frame region evaluation results be output at any timing. For example, the user may input the timings when imaging of the descending colon begins and ends, and the multi-frame region evaluation results may be output at the end timing using the evaluation results obtained from the start to end of imaging.

[0120] A specific example of a method for displaying multi-frame region evaluation results will be described. When the first multi-frame region evaluation result and the second multi-frame region evaluation result are output, as shown in Fig. 22, a currently observed endoscopic image 150 and a multi-frame region evaluation result display field 151 are displayed in the display image 131. In the specific example shown in Fig. 22, the multi-frame region evaluation result display field 151 displays, for the region currently being observed (descending colon), the first multi-frame region evaluation result of "remission" as A%, the first multi-frame region evaluation result of "non-remission" as B%, the second multi-frame region evaluation result of "remission" as C%, and the second multi-frame region evaluation result of "non-remission" as D%, with bar graphs corresponding to the multi-frame region evaluation results displayed nearby.

[0121] 23, a schematic diagram 152 may be displayed, and the region currently being observed may be highlighted with a frame 153. The contents of the multi-frame region evaluation result display field 151 are the same as the specific example shown in FIG.

[0122] As shown in Fig. 24, a three-dimensional structural diagram 154 of the large intestine may be displayed, reflecting regions 156 where the first multi-frame region evaluation result or the second multi-frame region evaluation result indicates "remission" and regions 157 where the first multi-frame region evaluation result indicates "non-remission." The contents of the multi-frame region evaluation result display field 151 are the same as the specific example shown in Fig. 22. The three-dimensional structural diagram 154 may be stored in advance, or may be generated from an endoscopic image using a method such as Structure from Motion.

[0123] As shown in Figure 25, a simplified schematic diagram 158 is shown in which the position of the large intestine is represented by a bar, and the first or second multi-frame region evaluation result may reflect regions 156 where the first or second multi-frame region evaluation result is "remission" and regions 157 where the first or second multi-frame region evaluation result is "non-remission." The contents of the multi-frame region evaluation result display field 151 are the same as the specific example shown in Figure 22. R indicates the rectum, SC indicates the sigmoid colon, DC indicates the descending colon, TC indicates the transverse colon, AC indicates the ascending colon, and C indicates the cecum.

[0124] In the case of diseases such as ulcerative colitis, in which the lesions are not spot-like but spread over a surface, the visibility of the evaluation results can be improved by displaying evaluation results that evaluate a relatively wide area, as in the above configuration.

[0125] It is preferable that an endoscopic image divided into a plurality of regions is input to the image assessment unit 70, and at least one of a first evaluation result and a second evaluation result is output for each divided region. In this case, the medical image acquisition unit 60 transmits the endoscopic image to the image segmentation unit 160 (see FIG. 2 ), the image segmentation unit 160 segments the endoscopic image into a plurality of regions, the endoscopic image divided into a plurality of regions is input to the image assessment unit 70, and the image assessment unit 70 outputs at least one of a first evaluation result and a second evaluation result. Hereinafter, the first evaluation result output for one region will be referred to as the first region evaluation result, and the second evaluation result output for one region will be referred to as the second region evaluation result.

[0126] The image dividing unit 160 divides the endoscopic image into at least two or more regions. For example, the endoscopic image 60c is divided into a plurality of lattice-shaped regions as shown in FIG. 26. In the specific example of FIG. 26, the endoscopic image 60c is divided into 16 regions, region A to region P. The shape of the division is not limited to a lattice shape, and may be any shape that can subdivide the endoscopic image, such as a polygonal shape such as a hexagon, or a curved shape. Furthermore, the plurality of regions may be cut out into random shapes that vary in shape and size depending on the location.

[0127] As shown in Fig. 27, the image assessment unit 70 outputs a first evaluation result and / or a second evaluation result for multiple regions. In the specific example shown in Fig. 27, the first evaluation result for regions A to F, H, and L to P is output as "remission," and the first evaluation result for regions G and I to K is output as "non-remission." This configuration allows for detailed evaluation of lesions and the like present in a portion of an endoscopic image. In particular, when the treatment goal is pathological remission, pathological results may differ from region to region even within the same image. Therefore, it may be preferable to output the first evaluation result spotwise for each small divided region rather than for each frame.

[0128] Furthermore, it is preferable that the image assessment unit 70 assesses the quality of the image itself for each of the multiple regions prior to evaluating the lesion, and then outputs at least one of the first and second evaluation results. In this case, the medical image acquisition unit 60 first transmits the endoscopic image to the image segmentation unit 160, which then segments the endoscopic image into multiple regions. Next, the endoscopic image segmented into the multiple regions is input to the region evaluation unit 170, which then outputs region evaluation indices for the multiple segments.

[0129] The region evaluation index is an evaluation value for the quality of the divided endoscopic image itself. In other words, the higher the region evaluation index, the more the quality of the original endoscopic image is ensured, and the higher the reliability of the first evaluation result and the second evaluation result. The endoscopic image, in which region evaluation indices are assigned to the multiple divided regions, is input to the image assessment unit 70, which then outputs at least one of the first evaluation result and the second evaluation result for each divided region.

[0130] In this case, it is preferable that the image segmentation unit 160 segments the image into portions of a size that allows for the determination of lesions and non-lesion evaluation inhibitors. Evaluation inhibitors refer to structures or artifacts that are not suitable for outputting the first or second evaluation results and that may reduce the accuracy of the evaluation. Examples of evaluation inhibitors include specific liquid pools 161, such as puddles of water or blood covering the object of observation, or pools of excess drug solution, as shown in FIG. 28 , and bubbles 162, as shown in FIG. 29 . Also included are distortions (distortions caused by the objective lens used to capture the image of the object of observation) occurring in the peripheral portion 163 of the image, blurring of the image, and reflections 164, such as halation, as shown in FIG. 30 . Furthermore, if a cap (hood) is attached to the tip 12d of the endoscope 12 and the edge 165a of the cap is reflected in the endoscopic image, the evaluation obstruction targets include the edge 165a of the cap, as shown in Figure 31, or the edge 165a of the cap and the outside 165b of the cap, as shown in Figure 32.

[0131] If the region does not contain an evaluation obstruction object and the endoscopic image can be appropriately judged, the region evaluation unit 170 calculates a high region evaluation index. On the other hand, if the region contains an evaluation obstruction object, the region evaluation unit 170 calculates a low region evaluation index.

[0132] In the specific example of the output of region evaluation indices shown in Figure 33, regions A to P are each assigned a region evaluation index. Of these, region B (region evaluation index 0.9), region E (region evaluation index 0.8), region F (region evaluation index 0.95), region G (region evaluation index 0.95), region H (region evaluation index 0.8), region I (region evaluation index 0.8), region J (region evaluation index 0.95), region K (region evaluation index 0.95), region L (region evaluation index 0.8), region N (region evaluation index 0.9), and region O (region evaluation index 0.9) are regions in which the first evaluation result or the second evaluation result can be appropriately output. On the other hand, regions A, D, M, and P (region evaluation index 0.5) are located at the edge of the endoscope's field of view, so distortion occurs, and region C (region evaluation index 0.05) has reflection 164, so the accuracy of the output of the first evaluation result or the second evaluation result is reduced. For this reason, areas A, C, D, M and P, which contain evaluation inhibition targets, are output with lower area evaluation indices than areas B, E, F, G, H, I, J, K, L, N and O, which do not contain evaluation inhibition targets.

[0133] The region evaluation unit 170 may generate an extracted image by extracting only regions equal to or greater than a region evaluation threshold from an endoscopic image divided into multiple regions, and input the extracted image to the image determination unit 70. With the above configuration, it is possible to exclude regions of low quality in advance and output the first evaluation result and / or the second evaluation result only for regions where a certain level of quality is guaranteed.

[0134] Furthermore, when an endoscopic image in which area evaluation indices are assigned to the divided areas is input to the image assessment unit 70, and the first evaluation result and / or the second evaluation result is output to generate the display image 131, areas in which the area evaluation index is less than the area evaluation threshold may be highlighted with a frame 166 to draw attention, as shown in Figure 34.

[0135] The image assessment unit 70 outputs at least one of a first evaluation result and a second evaluation result for each divided region of an endoscopic image frame. Furthermore, to obtain an overall evaluation result for the divided regions, it is preferable to use the first region evaluation result or the second region evaluation result output for each divided region as a unit score and output a multi-region evaluation result, which is an overall evaluation result, as an overall score based on the unit scores. The multi-region evaluation result is output by a multi-region evaluation result calculation unit 73 of the overall evaluation result calculation unit 71 shown in FIG. 21 . The first determiner 80, the second determiner 90, or the third determiner 100 of the image assessment unit 70 outputs the first region evaluation result or the second region evaluation result, which is a unit score, and transmits it to the multi-region evaluation result calculation unit 73 of the overall evaluation result calculation unit 71 (see FIG. 2 ).

[0136] The multiple region evaluation result calculation unit 73 preferably outputs at least one of a first multiple region evaluation result calculated using the first region evaluation results for at least two or more divided regions and a second multiple region evaluation result calculated using the second evaluation results for at least two or more divided regions. The first multiple region evaluation result and / or the second multiple region evaluation result is transmitted to the display control unit 130.

[0137] Specifically, when the first multiple region evaluation result is output in two stages, remission or non-remission, the multiple region evaluation result calculation unit 73 outputs a percentage indicating the proportion of pathological remission and a percentage indicating the proportion of pathological non-remission. Alternatively, the multiple region evaluation result calculation unit 73 may output a pathological state with a higher proportion of pathological remission or pathological non-remission, or may output a pathological state with a proportion equal to or greater than a certain value between pathological remission and pathological non-remission. When the first multiple frame evaluation result is output in multiple stages, such as GHS, the multiple region evaluation result calculation unit 73 outputs a percentage indicating the proportion of Geboes0, a percentage indicating the proportion of Geboes1, a percentage indicating the proportion of Geboes2A, a percentage indicating the proportion of Geboes2B, a percentage indicating the proportion of Geboes3, a percentage indicating the proportion of Geboes4, and a percentage indicating the proportion of Geboes5. Also, the GHS with the highest ratio among Geboes0 to Geboes5 may be output, or the GHS with a ratio equal to or greater than a certain value among Geboes0 to Geboes5 may be output.

[0138] When the second multi-region evaluation results are output in two stages, remission or non-remission, a percentage indicating the rate of endoscopic remission and a percentage indicating the rate of endoscopic non-remission are output. Alternatively, the endoscopic state with a higher rate of endoscopic remission or endoscopic non-remission may be output, or the endoscopic state with a rate of endoscopic remission or endoscopic non-remission above a certain value may be output. When the first multi-frame site evaluation results are output in multiple stages, such as MES, a percentage indicating the rate of Mayo 0, a percentage indicating the rate of Mayo 1, a percentage indicating the rate of Mayo 2, and a percentage indicating the rate of Mayo 3 may be output. Alternatively, the MES with the highest rate among Mayo 0 to Mayo 3 may be output, or the MES with a rate above a certain value among Mayo 0 to Mayo 3 may be output.

[0139] The multiple-region evaluation result is preferably output using the first region evaluation result or the second region evaluation result relating to at least two or more regions for an arbitrary internal body section, an arbitrary time period in which a plurality of endoscopic images are acquired, an arbitrary number of regions within one frame of endoscopic image, and a medium region combining arbitrary regions within one frame of endoscopic image. With the above configuration, it is possible to obtain an overall evaluation result that spatiotemporally combines the evaluation results using unit scores finer than one frame.

[0140] 21 , the multiple region evaluation result calculation unit 73 has a multiple region frame evaluation result calculation unit 180. The multiple region frame evaluation result calculation unit 180 uses the first region evaluation result or the second region evaluation result as a unit score, and outputs at least one of a first multiple region frame evaluation result calculated using the first region evaluation result and a second multiple region frame evaluation result calculated using the second region evaluation result as a total score for each frame of endoscopic image based on the unit score.

[0141] The first multi-region frame evaluation result and / or the second multi-region frame evaluation result may be output for each frame of endoscopic image. Alternatively, the first multi-region frame evaluation result may be output by majority vote based on the number of remission and non-remission regions in the first region evaluation result for each frame of endoscopic image, or by majority vote based on the number of remission and non-remission regions in the second region evaluation result. The multi-region frame evaluation result may be output as A% for "remission" and B% for "non-remission" based on the proportion of remission and non-remission regions in the first region evaluation result obtained for each frame of endoscopic image, or as C% for "remission" and D% for "non-remission" based on the proportion of remission and non-remission regions in the second region evaluation result obtained for each frame of endoscopic image. Alternatively, the multi-region frame evaluation result may be output as a percentage calculated using a multi-level evaluation, such as E% for "Geboes0" and F% for "Geboes1," based on the proportion of each GHS in the first region evaluation result obtained for each frame of endoscopic image. The multi-region frame evaluation results may be statistics such as averages, maximums and minimums.

[0142] The first and / or second multiple-region frame evaluation results are transmitted to the display control unit 130 and displayed on the display image 131. In this case, for example, as shown in FIG. 35 , the display image 131 displays the currently observed endoscopic image 150 and a multiple-region frame evaluation result display field 181. The currently observed endoscopic image 150 may be displayed with grid lines 182 indicating how the regions are divided. In the specific example shown in FIG. 35 , the multiple-region frame evaluation result display field 181 displays, for the currently observed endoscopic image 150, the first multiple-region evaluation result for "remission" as A%, the first multiple-region evaluation result for "non-remission" as B%, the second multiple-region evaluation result for "remission" as C%, and the second multiple-region evaluation result for "non-remission" as D%, with bar graphs corresponding to the multiple-region evaluation results displayed nearby. With the above configuration, an overall evaluation result that combines the evaluation results for the regions into which one endoscopic image frame is divided can be visually confirmed.

[0143] In order to obtain an evaluation result that integrates the divided regions by anatomical region, the image assessment unit 70 preferably uses the first region evaluation result or the second region evaluation result as a unit score and outputs a multi-region evaluation result as an integrated score for each anatomical region based on the unit score. In this case, the multi-region region evaluation result calculation unit 190 shown in FIG. 21 preferably outputs at least one of a first multi-region region evaluation result calculated using the first region evaluation result and a second multi-region region evaluation result calculated using the second region evaluation result.

[0144] The first multi-region site evaluation result and / or the second multi-region site evaluation result are preferably output for each anatomical site of the gastrointestinal tract. The first region evaluation result and / or the second multi-region site evaluation result may be output by majority vote based on the number of remission and non-remission regions in the first region evaluation result at the anatomical site, or by majority vote based on the number of remission and non-remission regions in the second region evaluation result at the anatomical site. The multi-region evaluation result may be output as A% for "remission" and B% for "non-remission" based on the proportion of remission and non-remission regions in the first region evaluation result obtained for each anatomical site, or as C% for "remission" and D% for "non-remission" based on the proportion of remission and non-remission regions in the second region evaluation result obtained for each anatomical site. Alternatively, the multi-region site evaluation result may be output as a percentage calculated using a multi-level evaluation, such as E% for "Geboes0" and F% for "Geboes1," based on the proportion of each GHS in the first region evaluation result obtained for each anatomical site. The multiple region site evaluation results may be statistics such as averages, maximum values, and minimum values. It is also preferable that the multiple region site evaluation results be output at any timing. For example, the user may input the timing when imaging of the descending colon begins and ends, and the multiple region site evaluation results may be output at the end timing using the evaluation results obtained from the start to the end of imaging.

[0145] The first and / or second multiple-region site evaluation results are transmitted to the display control unit 130 and displayed on the display image 131. Specifically, as shown in FIGS. 36 to 39 , the display image 131 displays the currently observed endoscopic image 150 and a multiple-region site evaluation result display field 191. The multiple-region site evaluation result display field 191 displays the first multiple-region site evaluation result for "remission" as A%, the first multiple-region site evaluation result for "non-remission" as B%, the second multiple-region site evaluation result for "remission" as C%, the second multiple-region site evaluation result for "non-remission" as D%, and a bar graph corresponding to the multiple-region site evaluation results is displayed nearby. This configuration allows the user to visually confirm an overall evaluation result that further integrates the evaluation results for the divided regions by anatomical site.

[0146] Similar to the specific examples of displaying multi-frame site evaluation results in Figures 22 to 25, Figure 36 displays the site currently being observed, and the multi-region site evaluation results for that site are displayed in a multi-region site evaluation result display field 191. Figure 37 shows a schematic diagram 152, with the currently observed site highlighted in a frame 153, and the multi-region site evaluation results for that site are displayed in a multi-region site evaluation result display field 191. Figure 38 displays a three-dimensional structural diagram 154 of the large intestine, and reflects the first multi-region site evaluation result or the second part multi-region site evaluation result, reflecting the site 156 of "remission" and the site 157 of "non-remission," respectively. Furthermore, the multi-region site display field 191 may display the multi-region site evaluation results for any site or all sites previously imaged. 39 shows a simplified schematic diagram 158 in which the position of the large intestine is represented by a bar, and the "remission" region 156 and the "non-remission" region 157 of the first multiple region site evaluation result or the second multiple region site evaluation result are reflected and displayed. The multiple region site evaluation result display field 191 may display the multiple region site evaluation results for any region or for all regions that have been imaged so far.

[0147] Alternatively, the multi-region frame evaluation results and the multi-region site evaluation results may be output and displayed on the display image. A specific example of this case will be described with reference to FIGS. 40 to 43. The display image 131 displays the currently observed endoscopic image 150, a multi-region frame evaluation result display field 181, and a multi-region site evaluation result display field 191. In the specific example shown in FIGS. 40 to 43, the multi-region frame evaluation result display field 181 displays, for the currently observed frame, the first multi-region frame evaluation result for "remission" as A%, the first multi-region frame evaluation result for "non-remission" as B%, the second multi-region frame evaluation result for "remission" as C%, and the second multi-region frame evaluation result for "non-remission" as D%, with bar graphs corresponding to the multi-region frame evaluation results displayed nearby. Furthermore, the multi-area site evaluation result display field 191 shows that the first multi-area site evaluation result of "remission" for the site currently being observed (descending colon) is E%, the first multi-area site evaluation result of "non-remission" is F%, the second multi-area site evaluation result of "remission" is G%, and the second multi-area site evaluation result of "non-remission" is H%, and bar graphs corresponding to the multi-area site evaluation results are displayed nearby.

[0148] In FIG. 40 , the currently observed region (descending colon) is displayed, and the multi-region region evaluation results for that region are displayed in a multi-region region evaluation result display field 191. In FIG. 41 , a schematic diagram 152 is shown, with the currently observed region highlighted by a frame 153, and the multi-region region evaluation results for that region are displayed in a multi-region region evaluation result display field 191. In FIG. 42 , a three-dimensional structural diagram 154 of the large intestine is displayed, reflecting the region 156 of “remission” and the region 157 of “non-remission” in the first or second multi-region region evaluation results. The multi-region region evaluation result display field 191 may also display the multi-region region evaluation results for any region or all regions imaged to date. In FIG. 43 , a simplified schematic diagram 158 is shown, in which the position of the large intestine is represented by a bar, reflecting the region 156 of “remission” and the region 157 of “non-remission” in the first or second multi-region region evaluation results. The multi-region site evaluation results for any site or all sites that have been imaged so far may be displayed in the multi-region site evaluation result display field 191. With the above configuration, it is possible to visually confirm both the results of integrating the evaluation results for the divided regions for each frame and the results of integrating the results for each anatomical site.

[0149] The notification control unit 200 (see FIG. 2) preferably issues a notification in accordance with the first evaluation result and the second evaluation result output by the image assessment unit 70. The notification is preferably issued by displaying a warning on the display image 131 when specific conditions, which will be described later, are satisfied. The warning is preferably issued by providing a warning display frame 201 on the endoscopic image 150 currently being observed, as shown in FIGS. 44 and 45. The display format of the warning is not limited to this, and a warning mark or a warning message may also be used. Furthermore, the notification is not limited to a warning display, and may be issued by sound or light.

[0150] The specific conditions for issuing a notification will be described below. Specific examples of the specific conditions are shown in (1)(A) to (4)(L) below, but are not limited to these.

[0151] (1) When only the first evaluation result is output for each frame of endoscopic image, (A) when the first evaluation criterion is two-stage and the first evaluation result is non-remission, or (B) when the first evaluation criterion is multi-stage and GHS, the first evaluation result is Geboes 2B or higher ("Geboes 2B", "Geboes 3", "Geboes 4", "Geboes 5").

[0152] (2) When the first evaluation result and the second evaluation result are output for each frame, (C) when the first evaluation criterion is two-stage, the first evaluation result is non-remission, (D) when the first evaluation criterion is multi-stage and GHS, the first evaluation result is Geboes 2B or more, (E) when the first evaluation criterion and the second evaluation criterion are two-stage, the second evaluation result is remission and the first evaluation result is non-remission, or (F) when the first evaluation criterion is multi-stage and GHS, and the second evaluation criterion is multi-stage and MES, the second evaluation result is Mayo 1 or less ("Mayo 0" or "Mayo 1") and the first evaluation result is Geboes 2B or more. Note that with regard to (E) and (F), even when one of the first evaluation criterion and the second evaluation criterion is multi-stage and the other is two-stage, if the first evaluation result corresponds to "non-remission" and the second evaluation result corresponds to "remission", it may be considered that the "specific condition" is met.

[0153] (3) When an endoscopic image is divided into multiple regions and the first and second evaluation results for the regions are output, (G) when the first evaluation criterion is two-stage, there is an area in one frame of the endoscopic image where the first evaluation result is non-remission; (H) when the first evaluation criterion is multi-stage and GHS, there is an area in one frame of the endoscopic image where the first evaluation result is Geboes 2B or more; (I) when the first evaluation criterion and the second evaluation criterion are two-stage, there is an area in one frame of the endoscopic image where the second evaluation result is remission and the first evaluation result is non-remission; or (J) when the first evaluation criterion is multi-stage and GHS, and the second evaluation criterion is multi-stage and MES, there is an area in one frame of the endoscopic image where the second evaluation result is Mayo 1 or less and the first evaluation result is Geboes 2B or more. Regarding (I) and (J), even if one of the first and second evaluation criteria is multi-stage and the other is two-stage, the "specific condition" may be satisfied if the first evaluation result corresponds to "non-remission" and the second evaluation result corresponds to "remission."

[0154] (4) When the first multiple-area frame evaluation result and the second multiple-area frame evaluation result are output, (K) the first multiple-area frame evaluation result satisfies the first area overall notification condition. Here, the first area overall notification condition indicates that, when the first multiple-area frame evaluation result expresses (a) non-remission or Geboes 2B or more, or (b) non-remission or Geboes 2B or more as a percentage, the percentage is equal to or greater than the first area overall notification threshold, or (c) non-remission or Geboes 2B or more by majority vote. The first area overall notification condition is not limited to this and may be set arbitrarily. Furthermore, (4) when the first multiple-area frame evaluation result and the second multiple-area frame evaluation result are output, (L) the first multiple-area frame evaluation result satisfies the first area overall notification condition and the second multiple-area frame evaluation result satisfies the second area overall notification condition. Here, the second region comprehensive notification condition indicates that when the second multiple region frame evaluation result is expressed as a percentage of (a) remission or Mayo 1 or less, or (b) remission or Mayo 1 or less, the percentage is equal to or greater than the second region comprehensive notification threshold, or (c) remission or Mayo 1 or less is more prevalent by majority vote. The second region comprehensive notification condition is not limited to this and may be set arbitrarily.

[0155] Specific examples of specific conditions have been given above, but in summary, a "specific condition" is when the first evaluation result (or the first multiple-area frame evaluation result) is non-remission, or when the second evaluation result (or the second multiple-area frame evaluation result) is remission and the first evaluation result (or the first multiple-area frame evaluation result) is non-remission.

[0156] Regarding the warning display, if specific conditions (1), (2), and (4) are met, it is determined whether the "specific conditions" are met on a frame-by-frame basis, and a notification is given by surrounding the entire endoscopic image 150 currently being observed with a warning frame 201, as shown in FIG. 44 . On the other hand, if specific condition (3) is met, it is determined whether the "specific conditions" are met on a region-by-region basis, and a notification is given by surrounding a single region (the area indicated by the warning frame 201) with the warning frame 201, as shown in FIG. 45 . Furthermore, a schematic diagram 152 may be displayed in the display image 131 to indicate the position currently being observed. With the above configuration, when pathological remission is the treatment goal, it is possible to prevent the overlooking of a "pathological non-remission despite endoscopic remission" condition during endoscopic examination.

[0157] A series of steps in the first embodiment will be described with reference to FIG. 46 . First, the medical image acquisition unit 60 acquires an endoscopic image (S101). Next, the image assessment unit 70 outputs at least one of the first and second evaluation results (referred to as the “evaluation result” in FIG. 46 ) (S102). Next, the overall evaluation result calculation unit 71 outputs the first and / or second multi-frame region evaluation results, the first and / or second multi-region frame evaluation results, and / or the first and / or second multi-region region evaluation results (referred to as the “overall evaluation result” in FIG. 46 ) (S103). Next, the display control unit 130 generates a display image 131 (S104). If a specific condition is met (S105), the notification control unit sends an instruction to the display control unit 130 to issue a notification (S106), and the display image 131 is displayed on the display 15 (S107).

[0158] In the first embodiment, at least one of the first evaluation result and the second evaluation result can be output from one type of endoscopic image. In this case, the type of evaluation result to be output can be selected depending on the user's purpose. Furthermore, in the first embodiment, both the first evaluation result and the second evaluation result can be output from multiple types of endoscopic images. In this case, multiple types of endoscopic images are evaluated using multiple types of evaluation criteria, so the endoscopic images obtained in the examination can be evaluated from multiple perspectives. This makes it possible to prevent "hidden non-remission" from being overlooked based on one of the evaluation criteria.

[0159] Furthermore, in the first embodiment, the first evaluation result and the second evaluation result as unit scores, and an overall evaluation result that is an overall score obtained by combining the unit scores, can be output. In summary, when obtaining evaluation results for each frame, (1) the frame evaluation result can be output as the unit score, and (2) the multi-frame evaluation result, which is an overall evaluation result for an arbitrary section or period, can be output as the overall score. As the multi-frame evaluation result, (3) the multi-frame evaluation result, which is an overall evaluation result for each anatomical region for which a specific section is set, can be output. On the other hand, when obtaining evaluation results for each region into which an endoscopic image is divided, (4) the region evaluation result can be output as the unit score, and (5) the multi-region evaluation result, which is an overall evaluation result for an arbitrary section, period, or intermediate region, can be output as the overall score. Specifically, as the multi-region evaluation result, (6) the multi-region frame evaluation result, which is an overall evaluation result for each frame, and (7) the multi-region frame region evaluation result, which is an overall evaluation result for each anatomical region, can be output. By outputting the overall evaluation result, evaluation results for a certain range within the body can be obtained. In ulcerative colitis, a disease in which the lesions spread over a surface rather than in spots, showing evaluation results over a wide area makes it possible to recognize the extent of the disease in three dimensions, making it easier to monitor the progress.

[0160] [Second Embodiment] In the second embodiment, the first illumination light mode and the second illumination light mode in the first embodiment are replaced with a mono-emission mode, and in addition to the mono-emission mode, a multi-emission mode is provided in which the illumination light can be automatically switched.

[0161] The mono-flash mode and the multi-flash mode can be switched via the central control unit 50 by operating the mode selector switch 12f. The mono-flash mode is a mode in which illumination light of the same spectrum (first illumination light or second illumination light) is continuously emitted. The multi-flash mode is a mode in which multiple illumination lights of different spectrums (first illumination light and second illumination light) are emitted while being switched according to a specific pattern to illuminate the subject. Note that in the mono-flash mode, it is preferable that the first illumination light and the second illumination light be switchable.

[0162] In the multi-flash mode, control is performed to change the light intensities of the purple light V, blue light B, green light G, and red light R for each frame F according to a specific light emission pattern. Examples of light emission patterns are given below. For example, in the first light emission pattern, as shown in Figure 47, two frames of the first illumination light L1 are emitted during a first illumination period Pe1 in which the subject is illuminated with the first illumination light L1, and one frame of the second illumination light L2 is emitted during a second illumination period Pe2 in which the subject is illuminated with the second illumination light L2. This pattern is repeated.

[0163] 48 , in the second light emission pattern, the first illumination light L1 is emitted for one frame F in the first illumination period Pe1, and the second illumination light L2 is emitted for four frames in the second illumination period Pe2, and this pattern is repeated. At this time, during the second illumination period Pe2, second illumination first-spectrum light L2SP, second illumination second-spectrum light L2SQ, second illumination third-spectrum light L2SR, and second illumination fourth-spectrum light L2SS, each having a different emission spectrum, are automatically switched to be emitted for each frame F. The light emission pattern is not limited to this and can be set arbitrarily.

[0164] In the multi-flash mode, by automatically switching the illumination light, it is possible to acquire images of the subject illuminated with different illumination lights at approximately the same position in the digestive tract. In the multi-flash mode, the light source processor 21 adjusts the light intensity of each light source according to the set illumination pattern to perform imaging. Specific examples of the first illumination light and the second illumination light are the same as those in the first embodiment, so description thereof will be omitted. It is preferable that there is at least one type of second illumination light.

[0165] In the multi-flash mode, it is preferable to display the first illumination light image, which is familiar to users such as doctors, on the display 15 as the endoscopic image currently being observed. It is also preferable to use the second illumination light image, which is unfamiliar to users but suitable for analyzing specific structures such as blood vessels, for image analysis and determining notification conditions, and to display it only when necessary. With the above configuration, it is possible to perform image analysis of specific structures while displaying the minimum necessary images and information on the screen observed by the doctor, making observation easier.

[0166] As shown in Figure 49, it is preferable that the display image 231 has a main section 210 and a sub-section 211, and that the first illumination light image 212 currently being observed is displayed in the main section 210, and the second illumination light image 213, a schematic diagram 252, and text information 215 indicating the first evaluation result and the second evaluation result are displayed in the sub-section.

[0167] In the second embodiment, the medical image acquisition unit 60 acquires a first illumination light image and a second illumination light image. The medical image acquisition unit 60 inputs the first illumination light image and the second illumination light image to the image determination unit 70. Here, it is preferable that the second illumination light image is input to a first determiner 80 or a second determiner 90, which outputs a first evaluation result based on a first evaluation criterion, and the first illumination light image is input to a third determiner 100, which outputs a second evaluation result based on a second evaluation criterion. The first determiner 80, the second determiner 90, and the third determiner 100 are common to the first embodiment, and therefore description thereof will be omitted.

[0168] The medical image acquisition unit 60 preferably inputs the first illumination light image and the second illumination light image to the image segmentation unit 160, which segments each of the first illumination light image and the second illumination light image into a plurality of regions. Each of the plurality of regions obtained by segmenting the first illumination light image is referred to as a first illumination segmentation, and each of the plurality of regions obtained by segmenting the second illumination light image is referred to as a second illumination segmentation. The second illumination light image input to the image assessment unit 70 as the second illumination segmentation is preferably input to a first determiner 80 or a second determiner 90 as an image processing unit or a trained model, which outputs a first evaluation result based on a first evaluation criterion. Furthermore, the first illumination light image input to the image assessment unit 70 as the first illumination segmentation is preferably input to a third determiner 100, which outputs a second evaluation result based on a second evaluation criterion.

[0169] It is also preferable to input the first illumination light image as the first illumination division and the second illumination light image as the second illumination division to the area evaluation unit 170, and then assign area evaluation indices to each of the first illumination division and the second illumination division before inputting them to the image determination unit 70.

[0170] In the multi-flash mode, it is preferable to input the second illumination light image to the image judgment unit 70 to output the first evaluation result, and input the first illumination light image to the image judgment unit 70 to output the second evaluation result, thereby outputting the first evaluation result and the second evaluation result, and to generate a display image 131 on which the first evaluation result and the second evaluation result are displayed.

[0171] It is preferable that the first evaluation result output using the second illumination light image of one frame or the second illumination light image as the second illumination division is transmitted to the overall evaluation result calculation unit 71, which outputs a first multiple-frame evaluation result, a first multiple-frame site evaluation result, a first multiple-region evaluation result, a first multiple-region frame evaluation result, and / or a first multiple-region site evaluation result. It is also preferable that the second evaluation result output using the first illumination light image of one frame or the first illumination light image as the first illumination division is transmitted to the overall evaluation result calculation unit 71, which outputs a second multiple-frame evaluation result, a second multiple-frame site evaluation result, a second multiple-region evaluation result, a second multiple-region frame evaluation result, and / or a second multiple-region site evaluation result. The output of the multiple-frame evaluation result, the multiple-frame site evaluation result, the multiple-region evaluation result, the multiple-region frame evaluation result, and the multiple-region site evaluation result is the same as in the first embodiment, and therefore will not be described here.

[0172] In this case, it is preferable to display the multiple-area frame evaluation results and the multiple-area part evaluation results in the display image 231. Specific examples will be described with reference to Figures 50 to 53. The first illumination light image 212 currently being observed is displayed in the main section 210 of the display image 231. The second illumination light image 213 currently being observed, a multiple-area frame evaluation result display field 281, and a multiple-area part evaluation result display field 291 are displayed in the sub-section 211.

[0173] 50 to 53 , the second multiple-region frame evaluation result for “remission” for the currently observed first illumination-light image 212 is displayed as W%, the second multiple-region frame evaluation result for “non-remission” is displayed as X%, the first multiple-region frame evaluation result for “remission” for the currently observed second illumination-light image 213 is displayed as Y%, and the first multiple-region frame evaluation result for “non-remission” is displayed as Z%, with bar graphs corresponding to the multiple-region frame evaluation results displayed nearby. Furthermore, the multiple-region site evaluation result display field 291 displays the second multiple-region site evaluation result for “remission” for the currently observed site (descending colon) as P%, the second multiple-region site evaluation result for “non-remission” as Q%, the first multiple-region site evaluation result for “remission” as R%, and the first multiple-region site evaluation result for “non-remission” as S%, with bar graphs corresponding to the multiple-region site evaluation result display field displayed nearby.

[0174] In Fig. 50, the region currently being observed (descending colon) is displayed, and the multi-region region evaluation results for that region are displayed in a multi-region region evaluation result display field 291. In Fig. 51, a schematic diagram 252 is shown, the region currently being observed is highlighted with a frame 253, and the multi-region region evaluation results for that region are displayed in a multi-region region evaluation result display field 291. In Fig. 52, a three-dimensional structural diagram 254 of the large intestine is displayed, and the first multi-region region evaluation result or the second multi-region region evaluation result is displayed, reflecting the region 256 of "remission" and the region 257 of "non-remission," respectively. In addition, the multi-region region evaluation result display field 291 may display the multi-region region evaluation results for any region or all regions that have been imaged so far. 53 shows a simplified schematic diagram 258 in which the position of the large intestine is represented by a bar, and the "remission" region 256 and the "non-remission" region 257 of the first multiple region site evaluation result or the second multiple region site evaluation result are reflected and displayed. Furthermore, the multiple region site evaluation result display field 291 may display the multiple region site evaluation results for any region or for all regions that have been imaged so far.

[0175] 54 to 57, the display image 231 does not necessarily have to display the multiple-region frame evaluation result display field 281 and the multiple-region site evaluation result display field 291. In each of FIGS. 54 to 57, a first illumination light image 212 currently being observed and a second illumination light image 213 currently being observed are displayed, and in each of FIGS. 54 and 55, a schematic diagram 252 in which the currently observed site is highlighted with a frame 253 is displayed. In each of FIGS. 54 and 55, a three-dimensional structural diagram 254 of the large intestine is displayed, reflecting the sites 256 and 257 of "remission" and "non-remission" in the first or second multiple-region site evaluation results. In each of FIGS. 57 and 58, a simplified schematic diagram 258 in which the positions of the large intestine are represented in the form of bars is displayed, reflecting the sites 256 and 257 of "remission" and "non-remission" in the first or second multiple-region site evaluation results. With the above configuration, the visibility of the first evaluation result, the second evaluation result, and the overall evaluation result can be improved.

[0176] 58 to 61, the display image 231 does not need to display the multiple region frame evaluation result display field 281, the multiple region part evaluation result display field 291, and the second illumination light image 213 currently being observed. Fig. 58 shows a display form in which the second illumination light image 214 currently being observed is not displayed in Fig. 54, Fig. 59 shows a display form in Fig. 55, Fig. 60 shows a display form in Fig. 56, and Fig. 61 shows a display form in Fig. 57.

[0177] As in the first embodiment, it is preferable that the notification control unit 200 issues a notification in accordance with the first evaluation result and the second evaluation result output by the image determination unit 70. The specific conditions for issuing a notification are the same as those in the first embodiment, and therefore will not be described here.

[0178] In the second embodiment, when the specific conditions (1), (2), and (4) described in the first embodiment are satisfied, it is finally determined whether the first illumination light image and the second illumination light image on a frame-by-frame basis satisfy the "specific conditions." Therefore, when the specific conditions (1), (2), and (4) are satisfied, a warning is issued by surrounding the entire first illumination light image 212 and the entire second illumination light image 213 with a warning display frame 301, as shown in FIG.

[0179] Furthermore, in the second embodiment, when the specific condition (3) described in the first embodiment is satisfied, it is finally determined whether the "specific condition" is satisfied for each first illumination area or each second illumination area. Therefore, when the specific condition (3) is satisfied, a warning is issued by surrounding the corresponding first illumination area of ​​the first illumination light image 212 and the second illumination area of ​​the second illumination light image 213 with a warning display frame 301, as shown in FIG. 63 . The corresponding first illumination area and second illumination area refer to positionally corresponding portions of the first illumination area of ​​the first illumination light image and the second illumination area of ​​the second illumination light image in chronologically consecutive or nearby frames. With the above configuration, when pathological remission is the treatment goal, it is possible to prevent a condition of "endoscopic remission but pathological non-remission" from being overlooked when using the multi-light emission mode during endoscopic examination.

[0180] The first evaluation result output using the second illumination light image and the second evaluation result output using the first illumination light image are preferably stored in a memory (not shown) in association with each other. Furthermore, the second illumination light image to which the first evaluation result is attached and the first illumination light image to which the second evaluation result is attached are preferably stored in association with site information of the anatomical site at which the first illumination light image or the second illumination light image is captured.

[0181] In the second embodiment, evaluation results based on endoscopic images can be output from white light images that doctors are accustomed to seeing, and evaluation results based on pathological images can be output from special light images that can analyze specific structures such as blood vessels, which are information that cannot be grasped from white light images. In this case, while observing the white light images that doctors are accustomed to seeing, doctors can recognize "hidden non-remission" that would be overlooked in white light images with the assistance of the medical image processing device.

[0182] In the present embodiment, the processor device 14 is connected to the endoscopic system 10. However, the present invention is not limited to this example, and other medical devices, such as an ultrasound imaging device or a radiography device, may also be used. The endoscope 12 may be a rigid or flexible endoscope. Furthermore, some or all of the central control unit 50, medical image acquisition unit 60, image assessment unit 70, comprehensive evaluation result calculation unit 71, display control unit 130, image segmentation unit 160, region evaluation unit 170, and notification control unit 200 of the endoscopic system 10 may be provided in an image processing device that communicates with the processor device 14 and cooperates with the endoscopic system 10. For example, they may be provided in a diagnosis support device that acquires images captured by the endoscope 12 directly from the endoscopic system 10 or indirectly from a PACS. Furthermore, a medical business support device that is connected via a network to various inspection devices, including the endoscopic system 10, such as the first inspection device, second inspection device, ..., Nth inspection device, can be provided with some or all of the central control unit 50, medical image acquisition unit 60, image assessment unit 70, overall evaluation result calculation unit 71, display control unit 130, image division unit 160, area evaluation unit 170, and notification control unit 200 of the endoscopic system 10.

[0183] In this embodiment, the hardware structure of processing units that perform various processes, such as the central control unit 50, medical image acquisition unit 60, image assessment unit 70, comprehensive evaluation result calculation unit 71, display control unit 130, image division unit 160, area evaluation unit 170, and notification control unit 200, is made up of various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to function as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically for performing various processes.

[0184] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include: a first configuration, as typified by client or server computers, in which a single processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units; and a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip. In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0185] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the storage unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD).

[0186] 10 Endoscope system 12 Endoscope 12a Insertion section 12b Operation section 12c Bending section 12d Tip section 12e Angle knob 12f Observation mode changeover switch 12h Still image acquisition instruction switch 12i Zoom operation section 12j Forceps port 13 Light source device 14 Processor device 15 Display 16 User interface 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 21 Light source processor 22 Optical path coupling section 23 Light guide 30a Illumination optical system 30b Imaging optical system 31 Illumination lens 41 Objective lens 42 Zoom lens 43 Imaging sensor 44 Imaging processor 50 Central control section 60 Medical image acquisition section 60a, 111a, 118b, 213 Second illumination light image 60b, 118a, 212 First illumination light image 60c Endoscopic image 70 Image determination unit 71 Overall evaluation result calculation unit 72 Multi-frame evaluation result calculation unit 73 Multi-region evaluation result calculation unit 80 First determiner 80a, 90a First evaluation result 81 Intramucosal bleeding 82 Extramucosal bleeding 90 Second determiner 100 Third determiner 101 Second evaluation result 110 Frequency domain image generation unit 110a Frequency domain image 112 Mask processing unit 112a Masked second illumination light image 113 Frequency component spatial distribution calculation unit 114 Frequency domain extraction unit 114a First frequency domain image 114b First frequency domain 114c Second frequency domain image 114d Second frequency domain 114e Third frequency domain image 114f Third frequency domain 115 Blood vessel density calculation unit 115a Density distribution image 115b, 115c, 115d, 117a, 117b Region on density distribution image 116 Luminance calculation unit 116a Second frequency luminance distribution image 116b, 116c, 117c, 117d Region on second frequency luminance distribution image 116d Third frequency luminance distribution image 116e, 116f, 117e, 117f Region on third frequency luminance distribution image 117 First evaluation result output unit 118 Correspondence unit 118c First learning data 120 Structure-emphasized image generation unit 120a Structure-emphasized image 130 Display control unit131, 231 Display image 132, 150 Endoscopic image under observation 133 Evaluation result display field 140 Multi-frame region evaluation result calculation unit 151 Multi-frame region evaluation result display field 152, 252 Schematic diagram 153, 253 Frame 154, 254 Three-dimensional structure diagram 156, 157, 256, 257 Overall evaluation result 158, 258 Simplified schematic diagram 160 Image division unit 161 Puddle portion 162 Bubbles 163 Image peripheral portion 164 Reflection 165a Cap edge 165b Entire cap area 170 Region evaluation unit 180 Multi-region frame evaluation result calculation unit 181, 281 Multi-region frame evaluation result display field 182 Grid lines 190 Multi-region region evaluation result calculation unit 191, 291 Multiple region site evaluation result display field 200 Notification control unit 201, 301 Warning display frame 210 Main section 211 Sub section 215 Character information

Claims

1. 1. A medical imaging device comprising a processor, The processor: Obtaining an endoscopic image captured by illuminating a subject with one or more types of illumination light; a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the degree of a disease, and a second evaluation result based on a second evaluation criterion different from the first evaluation criterion, using the endoscopic image; The processor: acquiring a plurality of endoscopic images including a first illumination light image captured by illuminating the subject with a first illumination light and a second illumination light image captured by illuminating the subject with a second illumination light having a spectrum different from that of the first illumination light; a first evaluation result based on the first evaluation criterion is output using the second illumination light image; The processor: A medical image processing device configured to automatically switch between a first illumination period in which the subject is illuminated with the first illumination light and a second illumination period in which the subject is illuminated with at least one type of the second illumination light.

2. The medical image processing apparatus according to claim 1 , wherein the first evaluation criterion is an evaluation criterion of the extent of a disease based on an image captured by an imaging device other than an endoscope.

3. The medical image processing apparatus according to claim 2 , wherein the image captured by the imaging device other than the endoscope is a pathological image of a biopsy tissue taken from the subject.

4. The processor: a first determiner is configured to output the first evaluation result using the endoscopic image; The medical image processing device according to claim 3 , wherein the first classifier outputs the first evaluation result using a pre-stored surface morphology classification that is a correspondence relationship between the endoscopic image and the degree of disease based on the pathological image.

5. The processor: generating a first frequency domain image by extracting a first frequency domain from the endoscopic image, a second frequency domain image by extracting a second frequency domain different from the first frequency domain from the endoscopic image, and a third frequency domain image by extracting a third frequency domain different from the first frequency domain and the second frequency domain; The medical image processing apparatus according to claim 4 , configured to output the first evaluation result based on the first evaluation criterion by the first determiner using the first frequency domain image, the second frequency domain image, and the third frequency domain image.

6. 1. A medical imaging device comprising a processor, The processor: Obtaining an endoscopic image captured by illuminating a subject with one or more types of illumination light; a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the degree of a disease, and a second evaluation result based on a second evaluation criterion different from the first evaluation criterion, using the endoscopic image; The processor: a second determiner is configured to output the first evaluation result using the endoscopic image; The second classifier is a medical image processing device that is trained using a training endoscopic image and the degree of disease based on a pathological image of biopsy tissue taken from the subject at the same position where the training endoscopic image was taken.

7. The medical image processing apparatus according to claim 1 , wherein the second evaluation criterion is an evaluation criterion of the severity of a disease based on the endoscopic image.

8. The processor: a third determiner different from the second determiner is configured to output the second evaluation result using the endoscopic image; The medical image processing apparatus according to claim 6 , wherein the third classifier is trained using the training endoscopic images and the degree of disease based on the endoscopic images.

9. the first evaluation criterion is an evaluation criterion for evaluating the extent of the disease based on an image captured by an imaging device other than an endoscope, which evaluates the extent of the disease in two stages; The medical image processing apparatus according to claim 1 , wherein the second evaluation criterion is an evaluation criterion for evaluating the extent of the disease based on an endoscopic image, which evaluates the extent of the disease in two stages.

10. the first evaluation criterion is an evaluation criterion of the degree of the disease based on an image captured by an imaging device other than an endoscope, which evaluates the degree of the disease in a multi-stage of three or more stages, The medical image processing apparatus according to claim 1 , wherein the second evaluation criterion is an evaluation criterion of the extent of the disease based on an endoscopic image, which evaluates the extent of the disease in three or more stages.

11. The medical image processing apparatus according to claim 1 , wherein the disease is ulcerative colitis.

12. the first evaluation criterion is Geboes histopathology score, Riley score, Modified Riley score, or Roberts histopathology index; The medical image processing apparatus according to claim 1 , wherein the second evaluation criterion is a Mayo endoscopic subscore or a UCEIS score.

13. the first evaluation criteria include the morphology of superficial blood vessels and / or the degree of bleeding; The medical image processing apparatus according to claim 1 , wherein the second evaluation criteria include a degree of redness of the mucous membrane, a degree of visibility of deep blood vessels, and / or a degree of erosion or ulcer.

14. the first evaluation criterion and the second evaluation criterion have a range in which the disease is in remission and a range in which the disease is not in remission, the range in which the disease is in remission according to the first evaluation criterion is smaller than the range in which the disease is in remission according to the second evaluation criterion; A first non-remission boundary value, which is the lowest value of the first evaluation criterion that indicates that the disease is not in remission, is different from a second non-remission boundary value, which is the lowest value of the second evaluation criterion that indicates that the disease is not in remission, The medical image processing apparatus according to claim 1 , wherein the first non-remission boundary value is set to a range in which the disease is in remission according to the second evaluation criterion.

15. The processor: outputting at least one of the first evaluation result and the second evaluation result for each frame of the endoscopic image; 7. A medical image processing device according to claim 1, configured to output and display at least one of a first multi-frame evaluation result calculated using the first evaluation result relating to at least two or more frames of the endoscopic image, and a second multi-frame evaluation result calculated using the second evaluation result relating to at least two or more frames of the endoscopic image.

16. The processor: Dividing the endoscopic image into a plurality of regions; The medical image processing apparatus according to claim 1 , wherein the apparatus is configured to output at least one of the first evaluation result and the second evaluation result for a plurality of the regions.

17. 1. A medical imaging device comprising a processor, The processor: Obtaining an endoscopic image captured by illuminating a subject with one or more types of illumination light; a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the degree of a disease, and a second evaluation result based on a second evaluation criterion different from the first evaluation criterion, using the endoscopic image; The processor: outputting region evaluation indices for the plurality of regions based on the endoscopic image divided into the plurality of regions; a computer configured to output at least one of the first evaluation result and the second evaluation result for the plurality of regions using the endoscopic image consisting of the plurality of regions to which the region evaluation indexes are assigned, A medical image processing device in which the region evaluation index is determined based on the degree to which an evaluation obstruction target is included.

18. The processor: outputting at least one of the first evaluation result and the second evaluation result for each of the regions of the endoscopic image; 17. A medical image processing device according to claim 16, configured to output at least one of a first multi-region evaluation result calculated using the first evaluation results for at least two or more of the regions, and a second multi-region evaluation result calculated using the second evaluation results for at least two or more of the regions.

19. The processor: The medical image processing apparatus according to claim 18 , configured to output at least one of the first multiple region evaluation result and the second multiple region evaluation result for each frame of the endoscopic image.

20. The processor: acquiring a first illumination light image captured by illuminating the subject with first illumination light as the endoscopic image; generating a structure-enhanced image using the first illumination light image; The medical image processing apparatus according to claim 1 , wherein the apparatus is configured to output the first evaluation result based on the first evaluation criterion using the structure-weighted image.

21. The medical image processing apparatus according to claim 20 , wherein the structure-emphasized image is an image in which the surface structure, color difference, or edge amount of the first illumination light image is emphasized.

22. The processor: The medical image processing apparatus according to claim 20 , configured to output the second evaluation result based on the second evaluation criterion using the first illumination light image.

23. The medical image processing apparatus according to claim 1 , wherein the first illumination light is white light.

24. The medical image processing apparatus according to claim 1 , wherein the second illumination light has a narrower wavelength band than the first illumination light.

25. The medical image processing apparatus according to claim 1 , wherein the second illumination light contains a larger amount of a specific blue or specific purple color than the first illumination light.

26. The processor: Dividing the second illumination light image into a plurality of second illumination regions; The medical image processing apparatus according to claim 1 , configured to output the first evaluation results for the plurality of second illumination regions.

27. The processor: The medical image processing apparatus according to claim 1 , configured to output the first evaluation result and the second evaluation result.

28. The processor: The medical image processing apparatus according to claim 27, configured to issue a notification when a combination of the first evaluation result and the second evaluation result satisfies a specific condition.

29. The specific condition for performing the notification is: The medical image processing apparatus according to claim 28 , wherein the first evaluation result is non-remission, or the second evaluation result is remission and the first evaluation result is non-remission.

30. The processor: The medical image processing apparatus according to claim 28, wherein the notification is performed by displaying a warning on a display on which the endoscopic image is displayed.

31. The processor:

29. The medical image processing apparatus according to claim 28, wherein the endoscopic image, the first evaluation result, and the second evaluation result are displayed on a display.

32. The processor:

29. The medical image processing apparatus according to claim 28, configured to store, in association with each other, site information of the subject from which the first evaluation result and the second evaluation result are output, the first evaluation result, and the second evaluation result.

33. A medical image processing apparatus according to any one of claims 1 to 6; a light source device that emits illumination light; and an endoscope that captures the endoscopic image.

34. acquiring an endoscopic image by illuminating a subject with one or more types of illumination light; and outputting, using the endoscopic image, at least one of a first evaluation result based on a first evaluation criterion, which is an evaluation criterion for the degree of a disease, and a second evaluation result based on a second evaluation criterion different from the first evaluation criterion, a processor that performs the steps of acquiring an endoscopic image and outputting at least one of the first evaluation result and the second evaluation result, acquiring a plurality of endoscopic images including a first illumination light image captured by illuminating the subject with a first illumination light and a second illumination light image captured by illuminating the subject with a second illumination light having a spectrum different from that of the first illumination light; a first evaluation result based on the first evaluation criterion is output using the second illumination light image; The processor: A method for operating a medical image processing device configured to automatically switch between a first illumination period in which the subject is illuminated with the first illumination light and a second illumination period in which the subject is illuminated with at least one type of second illumination light.